search

nix-community / poetry2nix

Convert poetry projects to nix automagically [maintainer=@adisbladis,@cpcloud]
MIT License
792 stars 407 forks source link

Error with `pyarrow` #1724

Open akaihola opened 2 weeks ago

akaihola commented 2 weeks ago

Describe the issue

If pyarrow is in [tool.poetry.dependencies], nix build fails with error: value is a set while a string was expected.

Additional context

To reproduce as simply as possibly, I added pyarrow = "^16.1.0" to pyproject.toml in poetry2nix/templates/app/ and ran nix build.

Output from `nix build --show-trace`: ```nix error: … while evaluating the attribute 'drvPath' at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/lib/customisation.nix:365:7: 364| in commonAttrs // { 365| drvPath = assert condition; drv.drvPath; | ^ 366| outPath = assert condition; drv.outPath; … while calling the 'derivationStrict' builtin at /builtin/derivation.nix:9:12: (source not available) … while evaluating derivation 'python3.11-app-0.1.0' whose name attribute is located at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/pkgs/stdenv/generic/make-derivation.nix:331:7 … while evaluating attribute 'propagatedBuildInputs' of derivation 'python3.11-app-0.1.0' at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/pkgs/stdenv/generic/make-derivation.nix:385:7: 384| depsHostHostPropagated = elemAt (elemAt propagatedDependencies 1) 0; 385| propagatedBuildInputs = elemAt (elemAt propagatedDependencies 1) 1; | ^ 386| depsTargetTargetPropagated = elemAt (elemAt propagatedDependencies 2) 0; … while calling anonymous lambda at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/pkgs/stdenv/generic/make-derivation.nix:321:13: 320| (map (drv: getDev drv.__spliced.hostHost or drv) (checkDependencyList "depsHostHostPropagated" depsHostHostPropagated)) 321| (map (drv: getDev drv.__spliced.hostTarget or drv) (checkDependencyList "propagatedBuildInputs" propagatedBuildInputs)) | ^ 322| ] … from call site at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/pkgs/stdenv/generic/make-derivation.nix:321:18: 320| (map (drv: getDev drv.__spliced.hostHost or drv) (checkDependencyList "depsHostHostPropagated" depsHostHostPropagated)) 321| (map (drv: getDev drv.__spliced.hostTarget or drv) (checkDependencyList "propagatedBuildInputs" propagatedBuildInputs)) | ^ 322| ] … while calling 'getOutput' at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/lib/attrsets.nix:1796:23: 1795| */ 1796| getOutput = output: pkg: | ^ 1797| if ! pkg ? outputSpecified || ! pkg.outputSpecified … while calling anonymous lambda at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/lib/lists.nix:334:29: 333| */ 334| imap1 = f: list: genList (n: f (n + 1) (elemAt list n)) (length list); | ^ 335| … from call site at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/lib/lists.nix:334:32: 333| */ 334| imap1 = f: list: genList (n: f (n + 1) (elemAt list n)) (length list); | ^ 335| … while calling anonymous lambda at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/pkgs/stdenv/generic/make-derivation.nix:276:15: 275| imap1 276| (index: dep: | ^ 277| if isDerivation dep || dep == null || builtins.isString dep || builtins.isPath dep then dep … from call site at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/pkgs/stdenv/generic/make-derivation.nix:277:12: 276| (index: dep: 277| if isDerivation dep || dep == null || builtins.isString dep || builtins.isPath dep then dep | ^ 278| else if isList dep then checkDependencyList' ([index] ++ positions) name dep … while calling 'isDerivation' at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/lib/attrsets.nix:1283:5: 1282| isDerivation = 1283| value: value.type or null == "derivation"; | ^ 1284| … while calling 'checkDrv' at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/pkgs/development/interpreters/python/mk-python-derivation.nix:206:26: 205| 206| checkDrv = attrName: drv: | ^ 207| if (isPythonModule drv) && (isMismatchedPython drv) then throwMismatch attrName drv … from call site at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/pkgs/development/interpreters/python/mk-python-derivation.nix:207:11: 206| checkDrv = attrName: drv: 207| if (isPythonModule drv) && (isMismatchedPython drv) then throwMismatch attrName drv | ^ 208| else drv; … while calling 'isPythonModule' at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/pkgs/development/interpreters/python/mk-python-derivation.nix:42:20: 41| 42| isPythonModule = drv: | ^ 43| # all pythonModules have the pythonModule attribute … while calling anonymous lambda at /nix/store/papjgkg8lwdad6qrqys31hx2rq87vd1y-source/default.nix:50:13: 49| ( 50| dep: | ^ 51| let … while calling 'func' at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/pkgs/development/interpreters/python/passthrufun.nix:26:20: 25| valid = value: pythonPackages.hasPythonModule value || providesSetupHook value || lib.elem value exceptions; 26| func = name: value: | ^ 27| if lib.isDerivation value then … from call site at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/pkgs/development/interpreters/python/passthrufun.nix:27:12: 26| func = name: value: 27| if lib.isDerivation value then | ^ 28| lib.extendDerivation (valid value || throw "${name} should use `buildPythonPackage` or `toPythonModule` if it is to be part of the Python packages set.") {} value … while calling 'isDerivation' at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/lib/attrsets.nix:1283:5: 1282| isDerivation = 1283| value: value.type or null == "derivation"; | ^ 1284| … from call site at /nix/store/papjgkg8lwdad6qrqys31hx2rq87vd1y-source/overrides/default.nix:2230:17: 2229| 2230| pyarrow = prev.pyarrow.overridePythonAttrs ( | ^ 2231| old: lib.optionalAttrs ((!old.src.isWheel or false) && lib.versionAtLeast old.version "0.16.0") … while calling 'overridePythonAttrs' at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/pkgs/development/interpreters/python/python-packages-base.nix:21:35: 20| passthru = (previousAttrs.passthru or { }) // { 21| overridePythonAttrs = newArgs: makeOverridablePythonPackage f (overrideWith newArgs); | ^ 22| }; … from call site at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/lib/trivial.nix:955:7: 954| { # TODO: Should we add call-time "type" checking like built in? 955| __functor = self: f; | ^ 956| __functionArgs = args; … while calling anonymous lambda at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/pkgs/development/interpreters/python/python-packages-base.nix:16:63: 15| # This function introduces `overridePythonAttrs` and it overrides the call to `buildPythonPackage`. 16| makeOverridablePythonPackage = f: lib.mirrorFunctionArgs f (origArgs: | ^ 17| let … from call site at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/lib/trivial.nix:955:7: 954| { # TODO: Should we add call-time "type" checking like built in? 955| __functor = self: f; | ^ 956| __functionArgs = args; … while calling anonymous lambda at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/lib/customisation.nix:136:17: 135| in 136| mirrorArgs (origArgs: | ^ 137| let … from call site at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/pkgs/development/interpreters/python/python-packages-base.nix:18:14: 17| let 18| args = lib.fix (lib.extends | ^ 19| (_: previousAttrs: { … while calling 'fix' at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/lib/fixed-points.nix:93:9: 92| */ 93| fix = f: let x = f x; in x; | ^ 94| … from call site at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/lib/fixed-points.nix:93:20: 92| */ 93| fix = f: let x = f x; in x; | ^ 94| … while calling 'extends' at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/lib/fixed-points.nix:306:7: 305| ( 306| final: | ^ 307| let … from call site at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/lib/fixed-points.nix:308:16: 307| let 308| prev = f final; | ^ 309| in … while calling anonymous lambda at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/pkgs/development/interpreters/python/python-packages-base.nix:24:10: 23| }) 24| (_: origArgs)); | ^ 25| result = f args; … from call site at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/pkgs/development/interpreters/python/python-packages-base.nix:21:76: 20| passthru = (previousAttrs.passthru or { }) // { 21| overridePythonAttrs = newArgs: makeOverridablePythonPackage f (overrideWith newArgs); | ^ 22| }; … while calling 'overrideWith' at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/pkgs/development/interpreters/python/python-packages-base.nix:26:22: 25| result = f args; 26| overrideWith = newArgs: args // (if pkgs.lib.isFunction newArgs then newArgs args else newArgs); | ^ 27| in … from call site at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/pkgs/development/interpreters/python/python-packages-base.nix:26:76: 25| result = f args; 26| overrideWith = newArgs: args // (if pkgs.lib.isFunction newArgs then newArgs args else newArgs); | ^ 27| in … while calling anonymous lambda at /nix/store/papjgkg8lwdad6qrqys31hx2rq87vd1y-source/overrides/default.nix:2231:9: 2230| pyarrow = prev.pyarrow.overridePythonAttrs ( 2231| old: lib.optionalAttrs ((!old.src.isWheel or false) && lib.versionAtLeast old.version "0.16.0") | ^ 2232| ( … from call site at /nix/store/papjgkg8lwdad6qrqys31hx2rq87vd1y-source/overrides/default.nix:2231:14: 2230| pyarrow = prev.pyarrow.overridePythonAttrs ( 2231| old: lib.optionalAttrs ((!old.src.isWheel or false) && lib.versionAtLeast old.version "0.16.0") | ^ 2232| ( … while calling 'optionalAttrs' at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/lib/attrsets.nix:1352:5: 1351| cond: 1352| as: | ^ 1353| if cond then as else {}; … from call site at /nix/store/papjgkg8lwdad6qrqys31hx2rq87vd1y-source/overrides/default.nix:2247:13: 2246| in 2247| lib.throwIf (arrowCppVersion != pyArrowVersion) errorMessage { | ^ 2248| nativeBuildInputs = old.nativeBuildInputs or [ ] ++ [ pkg-config cmake ]; … while calling 'throwIf' at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/lib/trivial.nix:879:19: 878| */ 879| throwIf = cond: msg: if cond then throw msg else x: x; | ^ 880| … from call site at /nix/store/papjgkg8lwdad6qrqys31hx2rq87vd1y-source/overrides/default.nix:2243:33: 2242| ARROW_HOME = _arrow-cpp; 2243| arrowCppVersion = lib.versions.majorMinor _arrow-cpp; | ^ 2244| pyArrowVersion = lib.versions.majorMinor prev.pyarrow; … while calling 'majorMinor' at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/lib/versions.nix:45:16: 44| */ 45| majorMinor = v: | ^ 46| builtins.concatStringsSep "." … from call site at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/lib/versions.nix:47:6: 46| builtins.concatStringsSep "." 47| (lib.take 2 (splitVersion v)); | ^ 48| … while calling 'sublist' at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/lib/lists.nix:1606:5: 1605| count: 1606| list: | ^ 1607| let len = length list; in error: value is a set while a string was expected ```
poetry2nix/templates/app/flake.nix: ```nix { description = "Application packaged using poetry2nix"; inputs = { flake-utils.url = "github:numtide/flake-utils"; nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable-small"; poetry2nix = { url = "github:nix-community/poetry2nix"; inputs.nixpkgs.follows = "nixpkgs"; }; }; outputs = { self, nixpkgs, flake-utils, poetry2nix }: flake-utils.lib.eachDefaultSystem (system: let # see https://github.com/nix-community/poetry2nix/tree/master#api for more functions and examples. pkgs = nixpkgs.legacyPackages.${system}; inherit (poetry2nix.lib.mkPoetry2Nix { inherit pkgs; }) mkPoetryApplication; in { packages = { myapp = mkPoetryApplication { projectDir = self; }; default = self.packages.${system}.myapp; }; # Shell for app dependencies. # # nix develop # # Use this shell for developing your app. devShells.default = pkgs.mkShell { inputsFrom = [ self.packages.${system}.myapp ]; }; # Shell for poetry. # # nix develop .#poetry # # Use this shell for changes to pyproject.toml and poetry.lock. devShells.poetry = pkgs.mkShell { packages = [ pkgs.poetry ]; }; }); } ```
modified `poetry2nix/templates/app/pyproject.toml`: ```toml [tool.poetry] name = "app" version = "0.1.0" description = "" authors = ["Your Name "] license = "MIT" readme = "README.md" [tool.poetry.dependencies] python = "^3.11" pyarrow = "^16.1.0" [build-system] requires = ["poetry-core"] build-backend = "poetry.core.masonry.api" ```
poetry.lock: ```toml # This file is automatically @generated by Poetry 1.6.1 and should not be changed by hand. package = [] [metadata] lock-version = "2.0" python-versions = "^3.11" content-hash = "81b2fa642d7f2d1219cf80112ace12d689d053d81be7f7addb98144d56fc0fb2" ```
TyberiusPrime commented 2 weeks ago

So this is in a check that tries to see if the pyarrow version and the arrow-cpp version (from nixpkgs) match, and it's passing the arrow-cpp attrSet, instead of the verison, to lib.versions.majorMinor.

I have a fix in my mega work in progress PR, but in the meantime, have you tried to use to 'just use a wheel'? (pyarrow.override {preferWheel=True}...)