Python + Poetry + Numpy ImportError

[andy-bell101]

Struggling to get Python and Poetry enabled for my package. I've currently got this setup. It looks similar to some of the issues posted already but if I understand it right they should have been fixed by now.

flake.nix

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
    systems.url = "github:nix-systems/default";
    devenv.url = "github:cachix/devenv";
    devenv.inputs.nixpkgs.follows = "nixpkgs";
    nixpkgs-python.url = "github:cachix/nixpkgs-python";
  };

  nixConfig = {
    extra-trusted-public-keys = "devenv.cachix.org-1:w1cLUi8dv3hnoSPGAuibQv+f9TZLr6cv/Hm9XgU50cw=";
    extra-substituters = "https://devenv.cachix.org";
  };

  outputs = {
    self,
    nixpkgs,
    devenv,
    systems,
    ...
  } @ inputs: let
    forEachSystem = nixpkgs.lib.genAttrs (import systems);
  in {
    packages = forEachSystem (system: {
      devenv-up = self.devShells.${system}.default.config.procfileScript;
    });

    devShells =
      forEachSystem
      (system: let
        pkgs = import nixpkgs {
          inherit system;
          config.allowUnfree = true;
        };
      in {
        default = devenv.lib.mkShell {
          inherit inputs pkgs;
          modules = [
            {
              languages.python = {
                enable = true;
                poetry = {
                  enable = true;
                  activate.enable = true;
                  install = {
                    enable = true;
                    allExtras = true;
                    compile = true;
                  };
                };
              };
            }
          ];
        };
      });
  };
}

pyproject.toml

[tool.poetry]
name = "test-project"
version = "0.1.0"
description = ""
authors = ["Andrew Bell <andrew.bell012276@gmail.com>"]
license = "MIT"
readme = "README.md"

[tool.poetry.dependencies]
python = "~3.11"
numpy = "^1.26.4"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

But running python -c "import numpy" gives me:

Traceback (most recent call last):
  File "/home/andy/tmp/.venv/lib/python3.11/site-packages/numpy/core/__init__.py", line 24, in <module>
    from . import multiarray
  File "/home/andy/tmp/.venv/lib/python3.11/site-packages/numpy/core/multiarray.py", line 10, in <module>
    from . import overrides
  File "/home/andy/tmp/.venv/lib/python3.11/site-packages/numpy/core/overrides.py", line 8, in <module>
    from numpy.core._multiarray_umath import (
ImportError: libz.so.1: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/andy/tmp/.venv/lib/python3.11/site-packages/numpy/__init__.py", line 130, in <module>
    from numpy.__config__ import show as show_config
  File "/home/andy/tmp/.venv/lib/python3.11/site-packages/numpy/__config__.py", line 4, in <module>
    from numpy.core._multiarray_umath import (
  File "/home/andy/tmp/.venv/lib/python3.11/site-packages/numpy/core/__init__.py", line 50, in <module>
    raise ImportError(msg)
ImportError: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.11 from "/home/andy/tmp/.venv/bin/python"
  * The NumPy version is: "1.26.4"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: libz.so.1: cannot open shared object file: No such file or directory

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/andy/tmp/.venv/lib/python3.11/site-packages/numpy/__init__.py", line 135, in <module>
    raise ImportError(msg) from e
ImportError: Error importing numpy: you should not try to import numpy from
        its source directory; please exit the numpy source tree, and relaunch
        your python interpreter from there.

Is this something obvious that I've got wrong?

[domenkozar]

Can you try adding nixpkgs-python.inputs.nixpkgs.follows = "nixpkgs";?

[andy-bell101]

Oops did mean to do that. I've just added it but I get the same error

[alecandido]

I actually had the exact same problem.

However, playing around, it seems to be caused by devenv.inputs.nixpkgs.follows = "nixpkgs";, though I'm not completely sure (removing that line, and including native dependencies, i.e. pkgs.zlib, seems to be enough).

---

This is working for me:

              languages.python = {
                enable = true;
                poetry = {
                  enable = true;
                  install = {
                    enable = true;
                    groups = ["..."];
                    allExtras = true;
                  };
                };
                libraries = with pkgs; [zlib];
                version = "3.11";
              };

without the .follow declaration. I will also test that is not working just adding that.

---

I have to take it back: now it is just working, even with the .follow.

Even before, I tried many times to destroy and recreate the environment, but it was still failing. Now, I checked out an old version (before 1.x) including the lock file, then I reapplied all the changes, and nix flake update again, and it seems to work... (I destroyed the whole folder and cloned again, just to be sure)

I have the two lock files, but even with the old one the problem is not present any longer.

So, the advice could be to destroy everything and restart from scratch, but I expected to have done it even before (I even removed the Poetry cache manually). So, I'm not sure any longer how to reproduce it.

[domenkozar]

@andy-bell101 can you also try be deleting .devenv first and using the follow?

[andy-bell101]

I've tried both with mixed success. Still experimenting with clearing caches and using different nixpkgs providers

[andy-bell101]

I think it was the compile = true; that was causing the problem. Removing that seems to have fixed the problem as far as I can tell