Python + Poetry + Numpy ImportError with multiple python versions

[ianepreston]

This feels very similar to #1095 but I think it's different enough to warrant a new issue, although if you'd prefer me to close this and carry on that thread I'm happy to do so.

I've been trying to build out a devenv with python and poetry that can support libraries like numpy. As with the poster in #1095 I was initially getting the Original error was: libz.so.1: cannot open shared object file: No such file or directory error. Updating my flake.nix to include libraries = with pkgs; [zlib]; in the python section, blowing away .venv, .direnv, flake.lock, and poetry.lock seemed to resolve the issue, I was able to launch python and import numpy. However, I got greedy. I want to be able to develop against multiple versions of python so I have multiple dev shells defined for each python version I want to support in the library I'm developing. Activating one of those environments caused the original import error to return. Switching back to the previously working environment also produces the error now, and deleting all the above mentioned files and restarting no longer resolves the issue for any environments. I'm really at a loss on this one. I get it not working, I get it working, but I don't get it working then not working. I assume there's some hidden file or setting I'm not touching but I don't know what it could be.

My flake.nix looks like this:

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    nixpkgs-python.url = "github:cachix/nixpkgs-python";
    nixpkgs-python.inputs = { nixpkgs.follows = "nixpkgs"; };
    devenv.url = "github:cachix/devenv";
  };

  nixConfig = {
    extra-trusted-public-keys =
      "devenv.cachix.org-1:w1cLUi8dv3hnoSPGAuibQv+f9TZLr6cv/Hm9XgU50cw=";
    extra-substituters = "https://devenv.cachix.org";
  };

  outputs = { self, nixpkgs, devenv, ... }@inputs:
    let
      system = "x86_64-linux";
      pkgs = import nixpkgs { system = system; };
      # A list of shell names and their Python versions
      pythonVersions = {
        python39 = "3.9";
        python310 = "3.10";
        python311 = "3.11";
        default = "3.10";
      };
      # A function to make a shell with a python version
      makePythonShell = shellName: pythonVersion:
        devenv.lib.mkShell {
          inherit inputs pkgs;
          modules = [
            ({ pkgs, config, ... }: {
              languages.python = {
                version = pythonVersion;
                libraries = with pkgs; [ zlib ];
                enable = true;
                venv.enable = true;
                poetry = {
                  enable = true;
                  activate.enable = true;
                  package = pkgs.poetry;
                  install = {
                    enable = true;
                    # compile = true;
                    installRootPackage = true;
                  };
                };
              };
            })
          ];
        };
    in {
      # mapAttrs runs the given function (makePythonShell) against every value
      # in the attribute set (pythonVersions) and returns a new set
      devShells.x86_64-linux = builtins.mapAttrs makePythonShell pythonVersions;
    };
}

pyproject.toml is very simple, I have a couple of basic dependencies that will work to ensure my environment is being activated and then a requirement for pandas:

[tool.poetry]
name = "nixpytesting"
version = "0.1.0"
description = "Figuring out nix and python"
authors = ["Ian Preston"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.9"
requests = "^2.32.3"
pandas = "^2.2.2"

[tool.poetry.group.dev.dependencies]
ruff = "^0.4.8"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Any advice for additional troubleshooting I can try would be appreciated.

[alecandido]

Not sure about your lock file, and which version of NumPy you're using, but I noticed that a "vanilla" flake (similar to https://devenv.sh/guides/using-with-flakes/, with the obvious configs for Python, as in your example) is succeeding to run NumPy 1.x.y, but failing with NumPy 2.0.0.

It is also failing with other compiled packages (e.g. Pandas or QuTiP).

I'm now testing https://github.com/cachix/devenv-nixpkgs

[alecandido]

@ianepreston I believe the main discussion to be still related to https://github.com/cachix/devenv/issues/773, so the problem should be solved there (I believe @Atry was mainly facing it).

However, a highly discouraged workaround that you can find on the NixOS Discourse is to manually set LD_LIBRARY_PATH, to contain the path to the libraries you're looking for.

This is partially what Devenv itself is doing https://github.com/cachix/devenv/blob/f810f8d8cb4e674d7e635107510bcbbabaa755a3/src/modules/languages/python.nix#L13-L27 and it should be the purpose of the libraries option. But in this case, it would be restricted to the Python executable, https://github.com/NixOS/nixpkgs/blob/89c49874fb15f4124bf71ca5f42a04f2ee5825fd/pkgs/development/interpreters/python/wrapper.nix#L45 https://nixos.org/manual/nixpkgs/stable/#fun-makeWrapper while setting manually is exposed to all executables in your shell (which is quite more aggressive).

Not sure why the libraries variable is not working for me (it's not even finding libstdc++.so.6 on NixOS), but setting LD_LIBRARY_PATH is working.

I'm pretty sure it's an ugly solution, with many drawbacks, but it may solve your issues while waiting for a proper one.

(in case you want to know which is the exact path to the library you want to use, assuming it's already available in your /nix/store/, I'd advise to use nix-locate from the nix-index tool, e.g. as nix-locate --top-level libz.so.1)