jupyter-server fails to build

deemp commented 1 year ago

Describe the issue

I expect that I get a devshell.

devShells.p2n =
        let
          myAppEnv = poetry2nix.mkPoetryEnv {
            projectDir = ./.;
            editablePackageSources = {
              my-app = ./src;
            };
            python = pkgs.python310;
            overrides = poetry2nix.overrides.withDefaults (self: super:
              let
                # workaround https://github.com/nix-community/poetry2nix/issues/568
                addBuildInputs = name: buildInputs: super.${name}.overridePythonAttrs (old: {
                  buildInputs = (builtins.map (x: super.${x}) buildInputs) ++ (old.buildInputs or [ ]);
                });
                mkOverrides = pkgs.lib.attrsets.mapAttrs (name: value: addBuildInputs name value);
              in
              mkOverrides {
                jupyter-server-terminals = [ "hatchling" ];
                bs4 = [ "setuptools" ];
              }
            );
          };
        in
        myAppEnv.env.overrideAttrs (oldAttrs: {
          buildInputs = [ pkgs.hello ];
        });

But I receive:

jupyter-server failed with exit code 1;

Initial error in a terminal: ```console error: builder for '/nix/store/wd4y5dfxpx5scszldrhpsn3syb3317w0-python3.10-jupyter-server-2.3.0.drv' failed with exit code 1; last 10 log lines: > full command: /nix/store/0pyymzxf7n0fzpaqnvwv92ab72v3jq8d-python3-3.10.9/bin/python3.10 /nix/store/wf1vywz8lb8xvh27zwcl41kc9hwl2r3p-python3.10-pip-22.3.1/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /build/tmpf9k99i_h > cwd: /build/jupyter_server-2.3.0 > Preparing metadata (pyproject.toml) ... error > error: metadata-generation-failed > > × Encountered error while generating package metadata. > ╰─> See above for output. > > note: This is an issue with the package mentioned above, not pip. > hint: See above for details. For full logs, run 'nix log /nix/store/wd4y5dfxpx5scszldrhpsn3syb3317w0-python3.10-jupyter-server-2.3.0.drv'. error (ignored): error: cannot unlink '/tmp/nix-build-python3.10-mypy-1.0.1.drv-0/mypy-1.0.1/mypyc': Directory not empty error: 1 dependencies of derivation '/nix/store/4ss6j6jkd973ikczlrjvnbr1m5iim2vs-python3-3.10.9-env.drv' failed to build error: 1 dependencies of derivation '/nix/store/ngrksg307p2v70n63gb09nr75h234vxz-interactive-python3-3.10.9-environment-env.drv' failed to build ```` `nix log /nix/store/wd4y5dfxpx5scszldrhpsn3syb3317w0-python3.10-jupyter-server-2.3.0.drv`: ```console Sourcing python-remove-tests-dir-hook Sourcing python-catch-conflicts-hook.sh Sourcing python-remove-bin-bytecode-hook.sh Sourcing pip-install-hook Using pipInstallPhase Sourcing python-imports-check-hook.sh Using pythonImportsCheckPhase Sourcing python-namespaces-hook Sourcing pip-build-hook Using pipBuildPhase Using pipShellHook @nix { "action": "setPhase", "phase": "unpackPhase" } unpacking sources unpacking source archive /nix/store/rdhcdfagi3rkl4zvb8k9c4k3m3cq6464-jupyter_server-2.3.0.tar.gz source root is jupyter_server-2.3.0 setting SOURCE_DATE_EPOCH to timestamp 1580601600 of file jupyter_server-2.3.0/tests/utils.py @nix { "action": "setPhase", "phase": "patchPhase" } patching sources Removing path dependencies Finished removing path dependencies Removing git dependencies Finished removing git dependencies @nix { "action": "setPhase", "phase": "configurePhase" } configuring no configure script, doing nothing @nix { "action": "setPhase", "phase": "buildPhase" } lines 1-26...skipping... Sourcing python-remove-tests-dir-hook Sourcing python-catch-conflicts-hook.sh Sourcing python-remove-bin-bytecode-hook.sh Sourcing pip-install-hook Using pipInstallPhase Sourcing python-imports-check-hook.sh Using pythonImportsCheckPhase Sourcing python-namespaces-hook Sourcing pip-build-hook Using pipBuildPhase Using pipShellHook @nix { "action": "setPhase", "phase": "unpackPhase" } unpacking sources unpacking source archive /nix/store/rdhcdfagi3rkl4zvb8k9c4k3m3cq6464-jupyter_server-2.3.0.tar.gz source root is jupyter_server-2.3.0 setting SOURCE_DATE_EPOCH to timestamp 1580601600 of file jupyter_server-2.3.0/tests/utils.py @nix { "action": "setPhase", "phase": "patchPhase" } patching sources Removing path dependencies Finished removing path dependencies Removing git dependencies Finished removing git dependencies @nix { "action": "setPhase", "phase": "configurePhase" } configuring no configure script, doing nothing @nix { "action": "setPhase", "phase": "buildPhase" } building Executing pipBuildPhase Creating a wheel... Processing /build/jupyter_server-2.3.0 Running command Preparing metadata (pyproject.toml) Traceback (most recent call last): File "/nix/store/wf1vywz8lb8xvh27zwcl41kc9hwl2r3p-python3.10-pip-22.3.1/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 144> hook = backend.prepare_metadata_for_build_wheel AttributeError: module 'hatchling.build' has no attribute 'prepare_metadata_for_build_wheel' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/nix/store/wf1vywz8lb8xvh27zwcl41kc9hwl2r3p-python3.10-pip-22.3.1/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 351> main() File "/nix/store/wf1vywz8lb8xvh27zwcl41kc9hwl2r3p-python3.10-pip-22.3.1/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 333> json_out['return_val'] = hook(**hook_input['kwargs']) File "/nix/store/wf1vywz8lb8xvh27zwcl41kc9hwl2r3p-python3.10-pip-22.3.1/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 148> whl_basename = backend.build_wheel(metadata_directory, config_settings) File "/nix/store/b28ayfb443svjlqxc0gn0wh422p7y0gi-python3.10-hatchling-1.11.1/lib/python3.10/site-packages/hatchling/build.py", line 41, in build_wheel return os.path.basename(next(builder.build(wheel_directory, ['standard']))) File "/nix/store/b28ayfb443svjlqxc0gn0wh422p7y0gi-python3.10-hatchling-1.11.1/lib/python3.10/site-packages/hatchling/builders/plugin/interface.py", line 105> configured_build_hooks = self.get_build_hooks(directory) File "/nix/store/b28ayfb443svjlqxc0gn0wh422p7y0gi-python3.10-hatchling-1.11.1/lib/python3.10/site-packages/hatchling/builders/plugin/interface.py", line 366> raise UnknownPluginError(f'Unknown build hook: {hook_name}') hatchling.plugin.exceptions.UnknownPluginError: Unknown build hook: jupyter-builder error: subprocess-exited-with-error × Preparing metadata (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> See above for output. note: This error originates from a subprocess, and is likely not a problem with pip. full command: /nix/store/0pyymzxf7n0fzpaqnvwv92ab72v3jq8d-python3-3.10.9/bin/python3.10 /nix/store/wf1vywz8lb8xvh27zwcl41kc9hwl2r3p-python3.10-pip-22.3.1/lib/> cwd: /build/jupyter_server-2.3.0 Preparing metadata (pyproject.toml) ... error error: metadata-generation-failed × Encountered error while generating package metadata. ╰─> See above for output. note: This is an issue with the package mentioned above, not pip. hint: See above for details. ```

Additional context

in my flake.nix, I use

poetry2nix = {
      url = "github:nix-community/poetry2nix";
      inputs.nixpkgs.follows = "nixpkgs";
    };

In nix repl, inputs.nixpkgs.rev shows e3f9921357f83dbf45934d1e72e89b928544a4be.

pyproject.toml has:

[tool.poetry.dependencies]
python = "^3.10"
requests = "^2.28.2"
bs4 = "^0.0.1"
aiohttp = "^3.8.4"
grequests = "^0.6.0"
httpx = "^0.23.3"
nltk = "^3.8.1"
pandas = "^1.5.3"
selenium = "^4.8.2"
ipykernel = "^6.21.2"
jupyter = "^1.0.0"
mypy = "^1.0.1"

ryanswrt commented 1 year ago

Have you tried after https://github.com/nix-community/poetry2nix/pull/1001 was merged? I am using jupyter via those overrides, so it would be odd if it doesn't work

deemp commented 1 year ago

Hi, @ryanswrt! I tried now here and got a similar error:

error: builder for '/nix/store/wkx7vxybb18vq7qykbm17isr4lyjhhcz-python3.10-jupyter-server-2.4.0.drv' failed with exit code 1;
       last 10 log lines:
       >   full command: /nix/store/iw1vmh509hcbby8dbpsaanbri4zsq7dj-python3-3.10.10/bin/python3.10 /nix/store/7i9aca5djiprkq30y6qznszfl72dqak4-python3.10-pip-22.3.1/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /build/tmp5h64ojzd
       >   cwd: /build/jupyter_server-2.4.0
       >   Preparing metadata (pyproject.toml) ... error
       > error: metadata-generation-failed
       >
       > × Encountered error while generating package metadata.
       > ╰─> See above for output.
       >
       > note: This is an issue with the package mentioned above, not pip.
       > hint: See above for details.
       For full logs, run 'nix log /nix/store/wkx7vxybb18vq7qykbm17isr4lyjhhcz-python3.10-jupyter-server-2.4.0.drv'.
error (ignored): error: cannot unlink '/tmp/nix-build-python3.10-mypy-1.1.1.drv-1/mypy-1.1.1': Directory not empty
error: 1 dependencies of derivation '/nix/store/g8b8s644ac0vpi1bqig2vxlbzmyii720-python3-3.10.10-env.drv' failed to build
error: 1 dependencies of derivation '/nix/store/8cy7wz4chy7c10vpcwzzpwx72i64xf2w-interactive-python3-3.10.10-environment-env.drv' failed to build

ryanswrt commented 1 year ago

Hmm, could you lock it to jupyter_server 2.3.0? That is the version that works for me

deemp commented 1 year ago

Locked as follows https://github.com/deemp/gists/blob/5bfaa710d8505fca6e84be7ba655ab60309d7650/tryP2N/pyproject.toml#L23

First, I get lots of warnings in red like

trace: warning: 'deps' argument to makeSetupHook is deprecated and will be removed in release 23.11., Please use propagatedBuildInputs instead. content of deps: /nix/store/8gzj48mkr8i824a4ali025hk85gmfdzy-python3.10-pip-22.3.1 /nix/store/0623n2275d43mqpd82djrlcmp296fi7d-python3.10-wheel-0.38.4
trace: warning: 'deps' argument to makeSetupHook is deprecated and will be removed in release 23.11., Please use propagatedBuildInputs instead. content of deps: /nix/store/8gzj48mkr8i824a4ali025hk85gmfdzy-python3.10-pip-22.3.1 /nix/store/0623n2275d43mqpd82djrlcmp296fi7d-python3.10-wheel-0.38.4

Then, I get an error

error: builder for '/nix/store/kl65zzha4zgcg6p0k4xvfdj7rcbbjcy8-python3.10-jupyter-server-2.3.0.drv' failed with exit code 1;
       last 10 log lines:
       >   full command: /nix/store/iw1vmh509hcbby8dbpsaanbri4zsq7dj-python3-3.10.10/bin/python3.10 /nix/store/7i9aca5djiprkq30y6qznszfl72dqak4-python3.10-pip-22.3.1/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /build/tmpkgf6lz88
       >   cwd: /build/jupyter_server-2.3.0
       >   Preparing metadata (pyproject.toml) ... error
       > error: metadata-generation-failed
       >
       > × Encountered error while generating package metadata.
       > ╰─> See above for output.
       >
       > note: This is an issue with the package mentioned above, not pip.
       > hint: See above for details.
       For full logs, run 'nix log /nix/store/kl65zzha4zgcg6p0k4xvfdj7rcbbjcy8-python3.10-jupyter-server-2.3.0.drv'.
error (ignored): error: cannot unlink '/tmp/nix-build-python3.10-mypy-1.1.1.drv-4/mypy-1.1.1': Directory not empty
error: 1 dependencies of derivation '/nix/store/f8ipfqk2n9jxbgxy2v1lzh3m2zyqnv8w-python3-3.10.10-env.drv' failed to build
error: 1 dependencies of derivation '/nix/store/f2lyd4057skxh56wjcx2y8v1siajwymx-interactive-python3-3.10.10-environment-env.drv' failed to build

ryanswrt commented 1 year ago

You seem to have a number of overrides in your flake, which I am unsure are still necessary; have you tried without them?

deemp commented 1 year ago

Yes, I tried. Like here. Then had to add them because of the errors like

error: builder for '/nix/store/ck2rd4wsgqfp0zqjzcbwrl1vdfvl9i6q-python3.10-bs4-0.0.1.drv' failed with exit code 2;
       last 10 log lines:
       >   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
       >   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
       >   File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
       >   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
       >   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
       >   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
       >   File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
       > ModuleNotFoundError: No module named 'setuptools'
       > 
       > 
       For full logs, run 'nix log /nix/store/ck2rd4wsgqfp0zqjzcbwrl1vdfvl9i6q-python3.10-bs4-0.0.1.drv'.
error (ignored): error: cannot unlink '/tmp/nix-build-python3.10-numpy-1.24.2.drv-1/numpy-1.24.2': Directory not empty
error: 1 dependencies of derivation '/nix/store/pba6jffs0h2s3ldb595gsa985rqgv7hf-python3-3.10.10-env.drv' failed to build
error: 1 dependencies of derivation '/nix/store/nrh78nj1pyhf913j6xqjy5qj9f3j1bgf-interactive-python3-3.10.10-environment-env.drv' failed to build

rafa-dot-el commented 1 year ago

I have the same issue, I created a small project with only a few dependencies to reproduce the error.

https://github.com/rafa-dot-el/python-poetry-on-nixos/blob/feat/add-nix-shell/flake.nix#L21

Gives

e propagatedBuildInputs instead. content of deps: /nix/store/micahg85qjjalavnrjsv6yip8m1m4qds-python3.10-pip-23.0.1 /nix/store/if8hmh5jb452r7xrqdabc83gdazx825n-python3.10-wheel-0.38.4
error: builder for '/nix/store/dvy4pm4wwy8v2pyzifr53525c642qys9-python3.10-pandas-ta-0.3.14b.drv' failed with exit code 2;
       last 10 log lines:
       >   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
       >   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
       >   File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
       >   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
       >   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
       >   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
       >   File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
       > ModuleNotFoundError: No module named 'setuptools'
       > 
       > 
       For full logs, run 'nix log /nix/store/dvy4pm4wwy8v2pyzifr53525c642qys9-python3.10-pandas-ta-0.3.14b.drv'.

Full logs: nix log /nix/store/06adx8wbaglgklnar1mcxn2zkdn3j9a0-python3.10-iniconfig-2.0.0.drv

@nix { "action": "setPhase", "phase": "unpackPhase" }
unpacking sources
unpacking source archive /nix/store/di4c31yr68j50ii2lh8qg02nax4x02ny-iniconfig-2.0.0.tar.gz
source root is iniconfig-2.0.0
setting SOURCE_DATE_EPOCH to timestamp 1580601600 of file iniconfig-2.0.0/src/iniconfig/py.typed
@nix { "action": "setPhase", "phase": "patchPhase" }
patching sources
Removing path dependencies
Finished removing path dependencies
Removing git dependencies
Finished removing git dependencies
@nix { "action": "setPhase", "phase": "configurePhase" }
configuring
no configure script, doing nothing
@nix { "action": "setPhase", "phase": "buildPhase" }
building
Executing pipBuildPhase
Creating a wheel...
WARNING: The directory '/homeless-shelter/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should >
Processing /build/iniconfig-2.0.0
  Running command Preparing metadata (pyproject.toml)
  Traceback (most recent call last):
    File "/nix/store/dsrpjlnb2rl5qx2h4kp7aibz6byzfbzb-python3.10-hatchling-1.13.0/lib/python3.10/site-packages/hatchling/metadata/core.py", line 1409, in cached
      self._cached = self.source.get_version_data()['version']
    File "/nix/store/dsrpjlnb2rl5qx2h4kp7aibz6byzfbzb-python3.10-hatchling-1.13.0/lib/python3.10/site-packages/hatchling/metadata/core.py", line 1457, in source
      raise UnknownPluginError(message)
  hatchling.plugin.exceptions.UnknownPluginError: Unknown version source: vcs

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/nix/store/wfgc28if3wg2g01q18f7s9mqzf1a02sh-python3.10-pip-23.0.1/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/nix/store/wfgc28if3wg2g01q18f7s9mqzf1a02sh-python3.10-pip-23.0.1/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/nix/store/wfgc28if3wg2g01q18f7s9mqzf1a02sh-python3.10-pip-23.0.1/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 152, in prepare_metadata_for_build_wheel
      whl_basename = backend.build_wheel(metadata_directory, config_settings)
    File "/nix/store/dsrpjlnb2rl5qx2h4kp7aibz6byzfbzb-python3.10-hatchling-1.13.0/lib/python3.10/site-packages/hatchling/build.py", line 56, in build_wheel
      return os.path.basename(next(builder.build(wheel_directory, ['standard'])))
    File "/nix/store/dsrpjlnb2rl5qx2h4kp7aibz6byzfbzb-python3.10-hatchling-1.13.0/lib/python3.10/site-packages/hatchling/builders/plugin/interface.py", line 93, in build
      self.metadata.validate_fields()
    File "/nix/store/dsrpjlnb2rl5qx2h4kp7aibz6byzfbzb-python3.10-hatchling-1.13.0/lib/python3.10/site-packages/hatchling/metadata/core.py", line 243, in validate_fields
      _ = self.version
    File "/nix/store/dsrpjlnb2rl5qx2h4kp7aibz6byzfbzb-python3.10-hatchling-1.13.0/lib/python3.10/site-packages/hatchling/metadata/core.py", line 128, in version
      self._version = self._get_version()
    File "/nix/store/dsrpjlnb2rl5qx2h4kp7aibz6byzfbzb-python3.10-hatchling-1.13.0/lib/python3.10/site-packages/hatchling/metadata/core.py", line 226, in _get_version
      version = self.hatch.version.cached
    File "/nix/store/dsrpjlnb2rl5qx2h4kp7aibz6byzfbzb-python3.10-hatchling-1.13.0/lib/python3.10/site-packages/hatchling/metadata/core.py", line 1411, in cached
      message = f'Error getting the version from source `{self.source.PLUGIN_NAME}`: {e}'
    File "/nix/store/dsrpjlnb2rl5qx2h4kp7aibz6byzfbzb-python3.10-hatchling-1.13.0/lib/python3.10/site-packages/hatchling/metadata/core.py", line 1457, in source
      raise UnknownPluginError(message)
  hatchling.plugin.exceptions.UnknownPluginError: Unknown version source: vcs
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /nix/store/0n4y44dnaxafqs7cg625aldrb152x7bx-python3-3.10.10/bin/python3.10 /nix/store/wfgc28if3wg2g01q18f7s9mqzf1a02sh-python3.10-pip-23.0.1/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py prepare_m>
  cwd: /build/iniconfig-2.0.0
  Preparing metadata (pyproject.toml) ... error
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

rafa-dot-el commented 1 year ago

I could bypass the error and execute the build with the following workaround.

But during runtime the following exception is thrown:

# ./result/bin/example-nix-poetry 
Traceback (most recent call last):
  File "/nix/store/py8rijy114g14yh5vmp0avw16yyz6syh-python3.10-nix-poetry-example-0.1.0/bin/.example-nix-poetry-wrapped", line 6, in <module>
    from nix_poetry_example.cli import hello
  File "/nix/store/py8rijy114g14yh5vmp0avw16yyz6syh-python3.10-nix-poetry-example-0.1.0/lib/python3.10/site-packages/nix_poetry_example/__init__.py", line 2, in <module>
    import pandas_ta as ta
  File "/nix/store/1n94grfgg3w7w4r6y3qsp25216qbl0jq-python3.10-pandas-ta-0.3.14b/lib/python3.10/site-packages/pandas_ta/__init__.py", line 7, in <module>
    from pkg_resources import get_distribution, DistributionNotFound
ModuleNotFoundError: No module named 'pkg_resources'

nix-community / poetry2nix

jupyter-server fails to build #1008

Describe the issue

Additional context