[Bug]: Including jupyenv in flake causes nix evaluation to OOM

[turion]

Current Behavior

In https://github.com/tweag/monad-bayes/pull/304/commits/5d08e7c3e790edc331466eeeeaa599528a5e076c I bumped our jupyenv version to revision 97f76f92fa14814e955a4f6847eaa4198525a19e. This causes nix develop to go into a memory-leaking infinite loop.

Expected Behavior

Just evaluating nix develop

Steps To Reproduce

1. Check out https://github.com/tweag/monad-bayes/pull/304/commits/5d08e7c3e790edc331466eeeeaa599528a5e076c
2. nix develop

OS

- system: `"x86_64-linux"`
 - host os: `Linux 6.1.51, NixOS, 23.11 (Tapir), 23.11.20230904.f53ec4f`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.17.0`
 - nixpkgs: `/nix/store/mwy5r2z2c1v4sgya3v5rszs7fmk89hzc-source`

Version

97f76f92fa14814e955a4f6847eaa4198525a19e

Additional Context

Running nix develop -vvv gives this ever repeating loop:

Relevant log output

instantiated 'source' -> '/nix/store/6xf538aynk7v7g573rhdd973vskip143-source.drv'
instantiated 'python3.10-bootstrap-flit-core-3.9.0' -> '/nix/store/ywzpzlkmp0rbmz21ziwwjbwgynqfakg2-python3.10-bootstrap-flit-core-3.9.0.drv'
instantiated 'source' -> '/nix/store/kzp5mfwd5p759mkgwia1dgcgsddbr9my-source.drv'
instantiated 'python3.10-bootstrap-installer-0.7.0' -> '/nix/store/garcn982ah65akbry7wzxpasb13amrqk-python3.10-bootstrap-installer-0.7.0.drv'
instantiated 'pyproject_hooks-1.0.0.tar.gz' -> '/nix/store/7lg3d41h9gslk4k47ydcgvaizki69axy-pyproject_hooks-1.0.0.tar.gz.drv'
instantiated 'python3.10-bootstrap-pyproject-hooks-1.0.0' -> '/nix/store/0j4ksf6yhmasl45wig4ifvys2y218pcy-python3.10-bootstrap-pyproject-hooks-1.0.0.drv'
instantiated 'packaging-23.0.tar.gz' -> '/nix/store/0khi1cnwxi1840lxlbmx91gil1zd1jvc-packaging-23.0.tar.gz.drv'
instantiated 'python3.10-bootstrap-packaging-23.0' -> '/nix/store/0lpni15gfp8acx7a2dga3fhz21qkp2v6-python3.10-bootstrap-packaging-23.0.drv'
instantiated 'tomli-2.0.1.tar.gz' -> '/nix/store/i08s0mz38gj8ixll8qphxg3l4digvxgh-tomli-2.0.1.tar.gz.drv'
instantiated 'python3.10-bootstrap-tomli-2.0.1' -> '/nix/store/b68wnrpfvsgcpirrf74b9rz44wgljbmz-python3.10-bootstrap-tomli-2.0.1.drv'
instantiated 'source' -> '/nix/store/w6fl1dqjvc7q74ris8m6kprmajnqsd4y-source.drv'
instantiated 'python3.10-bootstrap-build-0.10.0' -> '/nix/store/ss8bagnad1p9gvkj48yhx077jkihm87x-python3.10-bootstrap-build-0.10.0.drv'
instantiated 'pypa-build-hook.sh' -> '/nix/store/syv9j21gnivl1yvbqbl1cgl55kmky8j4-pypa-build-hook.sh.drv'
instantiated 'python-setup-hook.sh' -> '/nix/store/1ggi1d0idq4ikpxqv886sd7f9467psfc-python-setup-hook.sh.drv'
instantiated 'Python-3.10.12.tar.xz' -> '/nix/store/a5hx93i2r3x2g00kd5ywb2phj91hfq59-Python-3.10.12.tar.xz.drv'
instantiated 'python3-3.10.12' -> '/nix/store/fbyjcpdqv92z8fc85plz3m6j4f0m2jv2-python3-3.10.12.drv'
instantiated 'python-setup-hook.sh' -> '/nix/store/1ggi1d0idq4ikpxqv886sd7f9467psfc-python-setup-hook.sh.drv'
instantiated 'Python-3.10.12.tar.xz' -> '/nix/store/a5hx93i2r3x2g00kd5ywb2phj91hfq59-Python-3.10.12.tar.xz.drv'
instantiated 'python3-3.10.12' -> '/nix/store/fbyjcpdqv92z8fc85plz3m6j4f0m2jv2-python3-3.10.12.drv'
instantiated 'wrap-python-hook' -> '/nix/store/ipxz7lkjp6jjr6xkcvz33jal354154gh-wrap-python-hook.drv'
instantiated 'python-remove-tests-dir-hook' -> '/nix/store/13cyxa5v1syj13xmzsccs0s9bq98gxcg-python-remove-tests-dir-hook.drv'
instantiated 'python-remove-bin-bytecode-hook' -> '/nix/store/9ch8hyg0zjccvav10pmrs92a7lkswmrl-python-remove-bin-bytecode-hook.drv'

[turion]

I can observe a memory leak with v0.1.0 (37c9075875c4d1c1466465afd5e2c3ea1cc50325) as well. The last commit that worked was: 3ad2c9512c9efd586cf63adde454e734a8ce049c

[turion]

Sorry, it seems there is some interaction with a particular nixpkgs version, maybe it's not jupyenv's fault. Let me close until I understand it better.

[turion]

This problem seems to be solved after removing nixpkgs.follows = "nixpkgs"; in my code.

[turion]

I'm afraid the problem does indeed persist. After updating with nix flake update, the problem is back because for some reason I don't understand, it does update the jupyenv/nixpkgs input.

[turion]

It seems that jupyenv doesn't work with nixpkgs revision f5892ddac112a1e9b3612c39af1b72987ee5783a.

[turion]

A working combination is jupyenv https://github.com/tweag/jupyenv/commit/97f76f92fa14814e955a4f6847eaa4198525a19e with nixpkgs revision 18036c0be90f4e308ae3ebcab0e14aae0336fe42.

[turion]

My minimal reproducer:

{
  description = "A very basic flake";

  inputs = {
    jupyenv = {
      inputs.nixpkgs.follows = "nixpkgs";
      url = "github:tweag/jupyenv";
    };
  };

  outputs = { self, nixpkgs, jupyenv }:
  let system = "x86_64-linux";
  in {
    devShells.x86_64-linux.default = with import nixpkgs { inherit system; }; mkShell {
      packages = [
        (jupyenv.lib.${system}.mkJupyterlabNew {})
      ];
    };

    packages.x86_64-linux.hello = nixpkgs.legacyPackages.x86_64-linux.hello;

    packages.x86_64-linux.default = self.packages.x86_64-linux.hello;

  };
}

Running nix develop in a folder with this file triggers the bug.

The two input locks:

jupyenv: github:tweag/jupyenv/97f76f92fa14814e955a4f6847eaa4198525a19e
nixpkgs: github:NixOS/nixpkgs/f902cb49892d300ff15cb237e48aa1cad79d68c3

[GTrunSec]

  # FIXME: pkgs.poetry is segfaulting on nixpkgs-unstable & poetry2nix
https://github.com/nix-community/poetry2nix/issues/1291
https://github.com/nix-community/poetry2nix/issues/1316

see #524

[GTrunSec]

please try: https://github.com/nix-community/poetry2nix/pull/1329

[turion]

Indeed, adding inputs.jupyenv.inputs.poetry2nix.url = "github:nix-community/poetry2nix/?ref=refs/pull/1329/head"; fixes the issue.

[turion]

To illustrate the bug, I created https://github.com/turion/jupyenv-bug/. Reproduce like this:

$ git clone git@github.com:turion/jupyenv-bug.git

$  git branch --show-current
main

$ nix develop # This will work fine, feel free to abort

$ git checkout follow

$  git diff main -- flake.nix
diff --git a/flake.nix b/flake.nix
index 123f264..946ad46 100644
--- a/flake.nix
+++ b/flake.nix
@@ -3,6 +3,7 @@

   inputs = {
     jupyenv = {
+      inputs.nixpkgs.follows = "nixpkgs";
       url = "github:tweag/jupyenv";
     };
   };

$ nix develop # This will OOM, be sure to cancel before your machine becomes unresponsive!

$ git checkout resolution

$  git diff main -- flake.nix
diff --git a/flake.nix b/flake.nix
index 123f264..7b13f75 100644
--- a/flake.nix
+++ b/flake.nix
@@ -3,6 +3,8 @@

   inputs = {
     jupyenv = {
+      inputs.nixpkgs.follows = "nixpkgs";
+      inputs.poetry2nix.url = "github:nix-community/poetry2nix/?ref=refs/pull/1329/head";
       url = "github:tweag/jupyenv";
     };
   };

$ nix develop # This will work again