Failing to build rasa, "attribute 'type' missing"

[alpmestan]

Hello,

I'm trying to write a Nix derivation for rasa, and after trying pypi2nix and friends which ended up with undecipherable failures, I gave a shot at poetry2nix. For the record, I'm very new to Python/Nix packaging, being a lot more used to the Haskell/C/C++ corners of nixpkgs.

When I try to do the obvious thing with poetry2nix, by writing suitable nix files directly in a checkout of the rasa repo, and fixing a naming problem with one of the deps (typed-ast <=> typed_ast), I end up with a slightly confusing error.

error: while evaluating the attribute 'drvPath' at /nix/store/1mgwylas2q898px4sgwlvc8jrd9ilakl-nixos-20.03.1619.ab3adfe1c76/nixos/lib/customisation.nix:158:7:
while evaluating the attribute 'nativeBuildInputs' of the derivation 'rasa-1.11.0a1' at /nix/store/1mgwylas2q898px4sgwlvc8jrd9ilakl-nixos-20.03.1619.ab3adfe1c76/nixos/pkgs/development/interpreters/python/mk-python-derivation.nix:105:3:
while evaluating the attribute 'out.outPath' at /nix/store/1mgwylas2q898px4sgwlvc8jrd9ilakl-nixos-20.03.1619.ab3adfe1c76/nixos/lib/customisation.nix:151:13:
while evaluating the attribute 'buildInputs' of the derivation 'python3.7-rasabaster-0.7.26' at /nix/store/1mgwylas2q898px4sgwlvc8jrd9ilakl-nixos-20.03.1619.ab3adfe1c76/nixos/pkgs/development/interpreters/python/mk-python-derivation.nix:105:3:
while evaluating 'chooseDevOutputs' at /nix/store/1mgwylas2q898px4sgwlvc8jrd9ilakl-nixos-20.03.1619.ab3adfe1c76/nixos/lib/attrsets.nix:474:22, called from undefined position:
while evaluating 'optional' at /nix/store/1mgwylas2q898px4sgwlvc8jrd9ilakl-nixos-20.03.1619.ab3adfe1c76/nixos/lib/lists.nix:254:20, called from /nix/store/1mgwylas2q898px4sgwlvc8jrd9ilakl-nixos-20.03.1619.ab3adfe1c76/nixos/pkgs/development/tools/poetry2nix/poetry2nix/mk-poetry-dep.nix:116:14:
attribute 'type' missing, at /nix/store/1mgwylas2q898px4sgwlvc8jrd9ilakl-nixos-20.03.1619.ab3adfe1c76/nixos/pkgs/development/tools/poetry2nix/poetry2nix/mk-poetry-dep.nix:47:29

You can find my nix code along with the full trace in this gist: https://gist.github.com/alpmestan/f3d86a8ede2296049ddf4bb380e56b01 - I just placed rasa.nix and default.nix at the root of the rasa repo and am running nix-build from there. I tried using both the poetry2nix from my channel (20.03) as well as the git one, and both fail with the attribute type missing error.

Am I doing something wrong, or just hitting an edge case in poetry2nix?

For reference, the poetry.lock file that this is running against is: https://github.com/RasaHQ/rasa/blob/master/poetry.lock

[NobbZ]

I have similar behaviour of poetry2nix with stuff in my office that doesn't use pypi.org, but some internal pypi-like source.

For "Rasa" it seems to be the same, at least it has an additional "source" given in the pyproject.toml:

https://github.com/RasaHQ/rasa/blob/master/pyproject.toml#L24-L26

I can find one reference of that source also in the lock file:

https://github.com/RasaHQ/rasa/blob/master/poetry.lock#L2166-L2179

[NobbZ]

Today I found out about the source attribute for a dependency.

So I changed in my pyproject.toml (stripped diff to relevant parts:

diff --git a/pyproject.toml b/pyproject.toml
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -10,7 +10,7 @@

 [tool.poetry.dependencies]
 python = "^3.6"
-example-dep = "^0.1.0"
+example-dep = { version = "^0.1.0", source = "examplesource" }

This makes the "attribute 'type' missing" disappear, instead a new error occurs:

building '/nix/store/qchwsl55fdin4hnhzvd1figyjz45pa40-example-dep-0.1.0.tar.gz.drv'...

trying https://files.pythonhosted.org/packages/source/n/example-dep/example-dep-0.1.0.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (22) The requested URL returned error: 404
error: cannot download example-dep-0.1.0.tar.gz from any mirror
builder for '/nix/store/qchwsl55fdin4hnhzvd1figyjz45pa40-example-dep-0.1.0.tar.gz.drv' failed with exit code 1

An appropriate source in the pyproject.toml exists:

[[tool.poetry.source]]
name = "examplesource"
url = "http://example.com/repository/internal/simple"

Which is also appropriately listed in the poetry.lock:

[[package]]
category = "main"
description = ""
name = "example-dep"
optional = false
python-versions = ">=3.6,<4.0"
version = "0.1.0"

[…]

[package.source]
reference = "examplesource"
type = "legacy"
url = "http://example.com/repository/internal/simple"

The inability to use our internal python repository without manually overlaying internal dependencies in the expression is a big show stopper for nix in my company as it means for us "copy" parts of the lock file to the nix expression each time we bump the version of an internal dependency or update the lock file in general.

[Mic92]

I implemented type = url but it looks quite similar to type = legacy: https://github.com/nix-community/poetry2nix/pull/205

[NobbZ]

Maybe, don't have something using type = url at hands.

Though I checked our internal repository for the actual package location.

The url field refers to http://example.com/repository/internal/simple, while the tarball location is at http://example.com/repository/internal/packages/example-dep/0.1.0/example-dep-0.1.0.tar.gz.

So constructing the final URL from the content of the url field involves not only appending some path, but also going "upwards" in the folders.

As I am not sure how the poetry2nix tooling is structured, I'm not sure if I could tackle that problem in my spare time.

Also I am wondering if it could help to improve poetry2nix in this regard if fetchFromPypi had an option to provide the "pypi repository URL".

[Mic92]

You can have a look at the diff to see what parts need to be change to add a new type. Basically you need a new class to fetch checksums: https://github.com/nix-community/poetry2nix/pull/205/files#diff-6055b4e04c72459017e1d5f7d211d9f61b301bca4e0a24ffff69927fe8ad8151R26

And than add an additional if statement for the src for the source: https://github.com/nix-community/poetry2nix/pull/205/files#diff-e6ff7bae6f9e3999431388a8c7688c1c0c3985527683382409a415bb12e313fdR165

[NobbZ]

I'll try to get some time during the weekend, thanks for the hints!