1.0: Rewrite in Python

[domenkozar]

devenv started as a bash wrapped in Nix to explore the design space.

As a result, there were many subtle bugs and lack of features that were hard to implement.

Previous devenv was 240M in closure and now it's down to 126M.

TODO

• [x] Bump to Nix 2.17
• [x] Investigate a bug when .nix files are caches (eval-cache?)
• [x] Python native libraries support is broken
• [x] macOS intel crashes sometimes
• [ ] Fix devenv gc
• [ ] Fix flake.nix support
• [ ] Run devenv ci things in favor of devenv test.

Fixes

Fixes https://github.com/cachix/devenv/issues/744 Fixes #693 Fixes #658 Fixes #244 Fixes #436 Fixes #629 Fixes #731 Fixes #83 Fixes #80 Fixes #749 Fixes #507 Fixes #258 Fixes #486 Fixes #745 Fixes #599 Fixes #607

Try it out 🚀

Add devenv.yaml:

inputs:
  devenv:
     url: github:cachix/devenv/python-rewrite 

Plain Nix

cachix use devenv
nix-env -if https://install.devenv.sh/python-rewrite

Flakes

nix profile install --accept-flake-config tarball+https://install.devenv.sh/python-rewrite

[metasyn]

This branch, in conjunction with setting enterShell to include:

export LD_LIBRARY_PATH=${pkgs.stdenv.cc.cc.lib}/lib/;

was able to get us past being unable to find libstdc++.so.6 and GLIBC_2.38 when using various python packages (like grpc, or pandas).

[misumisumi]

Yes, I understand that, but the purpose of this branch is supposed to be that LD_LIBRARY_PATH is not transparent throughout the development environment. In fact, python.nix uses makeWrap to show variables only to python.

[mauricege]

I'm sorry if this has already been tracked, but with the current method of declaring LD_LIBRARY_PATH inside the wrapper, when managing a project using poetry, it is difficult to access the shared library because poetry does not use wrapped python. This will cause problems.

As far as I can see, this currently open pull-request over on nixpkgs should fix that issue: nixos/nixpkgs#250935

[cameronraysmith]

This instance of zlib.h not found

```pytb creating build/temp.linux-x86_64-cpython-310/pybedtools/include gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -I/nix/store/ixvka49i2mjnxzk8l8anykc1idilws1r-libxcrypt-4.4.36/include -fPIC -Ipybedtools/include/ -I/tmp/tmpd_84vc8x/.venv/include -I/nix/store/50kabpj0s79040a42b7jj2dxn85wmbfd-python3-3.10.13/include/python3.10 -c pybedtools/cbedtools.cpp -o build/temp.linux-x86_64-cpython-310/pybedtools/cbedtools.o In file included from pybedtools/include/bedFile.h:16, from pybedtools/cbedtools.cpp:802: pybedtools/include/gzstream.h:35:10: fatal error: zlib.h: No such file or directory 35 | #include | ^~~~~~~~ compilation terminated. error: command '/nix/store/6jnx2r93796cnqr5dznrgc1bwgyl488b-devenv-profile/bin/gcc' failed with exit code 1 at /nix/store/1hhzpyy8rvjavf894qy0r98hq8ah59qb-python3.11-poetry-1.7.1/lib/python3.11/site-packages/poetry/installation/chef.py:164 in _prepare 160│ 161│ error = ChefBuildError("\n\n".join(message_parts)) 162│ 163│ if error is not None: → 164│ raise error from None 165│ 166│ return path 167│ 168│ def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path: Note: This error originates from the build backend, and is likely not a problem with poetry but with pybedtools (0.9.1) not supporting PEP 517 builds. You can verify this by running 'pip wheel --no-cache-dir --use-pep517 "pybedtools (==0.9.1)"'. ```

may be similar to https://github.com/cachix/devenv/pull/507#issuecomment-1660105656 or others in this thread,

in this case, from

devenv devshell in flake.nix + pyproject.toml.

```nix devShells = forEachSystem (system: let pkgs = nixpkgs.legacyPackages.${system}; in { default = devenv.lib.mkShell { inherit inputs pkgs; modules = [ { packages = with pkgs; [ atuin bat bedtools gcc gh git gnumake htslib lazygit poethepoet ripgrep starship tree zlib zlib.dev zsh ]; dotenv = { enable = true; filename = ".env"; # disableHint = true; }; pre-commit.hooks = { alejandra.enable = true; ruff.enable = true; pyright.enable = true; }; difftastic.enable = true; languages.python = { enable = true; package = pkgs.python310; poetry = { enable = true; activate.enable = true; install = { enable = true; installRootPackage = true; groups = [ "lint" "test" "docs" "workflows" ]; }; }; }; } ]; }; }); ``` with ```toml [tool.poetry.dependencies] pybedtools = "0.9.1" ```

Certainly this requires an equivalent to making the environment with buildInputs available.

In any case, if we

migrate the poetry env build to poetry2nix

```nix { inputs = { nixpkgs.url = "github:NixOS/nixpkgs/nixos-23.11"; # nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable"; systems.url = "github:nix-systems/default"; devenv.url = "github:cachix/devenv"; flake-utils.url = github:numtide/flake-utils; poetry2nix.url = github:nix-community/poetry2nix; poetry2nix.inputs.nixpkgs.follows = "nixpkgs"; }; nixConfig = { extra-trusted-public-keys = [ "dnadiffusion.cachix.org-1:P20JWJrVBiN5iPBnzJ4UiqLVghGBCYOicXxltPRLaEY=" "devenv.cachix.org-1:w1cLUi8dv3hnoSPGAuibQv+f9TZLr6cv/Hm9XgU50cw=" ]; extra-substituters = [ "https://dnadiffusion.cachix.org" "https://devenv.cachix.org" ]; }; outputs = { self, nixpkgs, devenv, systems, poetry2nix, ... } @ inputs: let forEachSystem = nixpkgs.lib.genAttrs (import systems); in { packages = forEachSystem (system: let pkgs = import nixpkgs { inherit system; overlays = [poetry2nix.overlays.default]; }; in { devenv-up = self.devShells.${system}.default.config.procfileScript; }); devShells = forEachSystem (system: let pkgs = import nixpkgs { inherit system; overlays = [poetry2nix.overlays.default]; }; pyPkgsBuildRequirements = { biofluff = ["setuptools"]; biothings-client = ["setuptools"]; cloudpickle = ["flit-core"]; feather-format = ["setuptools"]; flytekit = ["setuptools"]; flyteidl = ["setuptools"]; genomepy = ["hatchling"]; gimmemotifs = ["setuptools"]; gtfparse = ["setuptools"]; htseq = [pkgs.swig]; hydra-core = ["setuptools"]; hydra-joblib-launcher = ["setuptools"]; hydra-zen = ["setuptools"]; logomaker = ["setuptools"]; marshmallow-jsonschema = ["setuptools"]; mygene = ["setuptools"]; memory-efficient-attention-pytorch = ["setuptools"]; norns = ["setuptools"]; pybedtools = ["setuptools" "cython" pkgs.bedtools pkgs.htslib pkgs.zlib]; pybigwig = [pkgs.zlib pkgs.curl]; pysam = [pkgs.bzip2 pkgs.curl pkgs.htslib pkgs.openssl pkgs.xz]; }; poetry2nixOverrides = pkgs.poetry2nix.overrides.withDefaults ( self: super: let buildInputsOverrides = builtins.mapAttrs ( package: buildRequirements: (builtins.getAttr package super).overridePythonAttrs (old: { buildInputs = (old.buildInputs or []) ++ (builtins.map (pkg: if builtins.isString pkg then builtins.getAttr pkg super else pkg) buildRequirements); }) ) pyPkgsBuildRequirements; in buildInputsOverrides // { hydra-core = super.hydra-core.override {preferWheel = true;}; hydra-joblib-launcher = super.hydra-joblib-launcher.override {preferWheel = true;}; mysql-connector-python = super.mysql-connector-python.override {preferWheel = true;}; pysam = super.pysam.override {preferWheel = true;}; qnorm = super.qnorm.override {preferWheel = true;}; scipy = super.scipy.override {preferWheel = true;}; sourmash = super.sourmash.override {preferWheel = true;}; yarl = super.yarl.override {preferWheel = true;}; } ); poetryEnv = pkgs.poetry2nix.mkPoetryEnv { projectDir = ./.; python = pkgs.python310; preferWheels = false; editablePackageSources = { dnadiffusion = ./src; }; groups = ["lint" "test" "workflows"]; checkGroups = ["test"]; extras = []; overrides = poetry2nixOverrides; }; in { default = devenv.lib.mkShell { inherit inputs pkgs; modules = [ { packages = with pkgs; [ poetryEnv poetry atuin bat gh git gnumake lazygit poethepoet ripgrep starship tree zsh ]; dotenv = { enable = true; filename = ".env"; # disableHint = true; }; pre-commit.hooks = { alejandra.enable = true; ruff.enable = true; # pyright.enable = true; }; difftastic.enable = true; } ]; }; }); }; } ```

everything works as expected.

[mcdonc]

FWIW, unlike on the master branch, on this branch, if the languages.python config is:

  languages.python = {
    enable = true;
    # https://github.com/cachix/devenv/issues/901
    # version = "3.11.6";
    venv = {
      enable = true;
      requirements = ./requirements-devenv.txt;
    };
  };

In flakes mode, it will run pip install of the requirements every time you invoke nix develop --impure as well as when you run devenv up.

I'm trying to investigate.

EDIT: It actually rebuilds the entire venv because it believes the requirements have changed on every invocation of nix develop (or devenv shell in nonflakes mode) and devenv up.

EDIT 2: This is due to the fact that my requirements filename is not requirements.txt and this line https://github.com/cachix/devenv/blob/python-rewrite/src/modules/languages/python.nix#L49 ... We could store the name of the requirements file, if it's a path, in some other derivation and compose the comparison out of it. However, it's going to miss requirement changes composed out of includes like -r requirements.txt, so its actually a lot more complex.

EDIT 3: After poking at it a while, I'm pretty sure I don't understand why https://github.com/cachix/devenv/blob/python-rewrite/src/modules/languages/python.nix#L49 is required at all. I don't see why we have to destroy the venv if the requirements have changed. I'm going to remove it in a PR and see what say you.

EDIT 4: Even with that line removed, it always rebuilds the venv. Good times. Here's why. If I add the following to the venv init shell script:

    echo $(${readlink} "${package.interpreter}")
    echo $(${readlink} "$VENV_PATH/bin/python")

Here's what gets spit out:

/nix/store/yxw6lvjh9vrf2cr6rabcp5pp5n8cs323-python3-3.11.6-env/bin/python3.11
/nix/store/qp5zys77biz7imbk6yy85q5pdv7qk84j-python3-3.11.6/bin/python3.11

Those two are compared to see if it needs to rebuild the venv. They will never match, so the venv is always rebuilt. I couldn't begin to tell you how to generate a venv that will resolve to the wrapper. The things in the wrapper bindir are not just symlinks to the real Python binaries, they are.. hardlinks maybe? Nope. They are wrapper binaries. Ugh. I think we're shit out of luck if we want to use development/interpreters/python/wrapper.nix, I can't see a way to tell the virtualenv its interprer binary is actually the wrapper and not the python the wrapper wraps.

https://github.com/cachix/devenv/pull/904

[domenkozar]

Thanks to @mcdonc we have Python working with native dependencies: https://github.com/cachix/devenv/pull/939

[Nebucatnetzer]

I tested it against our current setup and haven't found any problems so far. locale -a shows now not just the system locales which is nice.

Edit: devenv up still triggers evaluation in Flake, I reckon that's more of a problem with Flakes than with devenv?

[domenkozar]

Continuation of this work is ready for another round of testing: https://github.com/cachix/devenv/pull/1005