search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
18.2k stars 14.2k forks source link

deeptools interferes with jupyter lab (problem with propagated inputs?) #261552

Open MatrixManAtYrService opened 1 year ago

MatrixManAtYrService commented 1 year ago

Describe the bug

Jupyter Lab kernels fail to start, but only when deeptools is installed.

Steps To Reproduce

Here'a flake.nix

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
    flake-utils.url = "github:numtide/flake-utils";
  };

  outputs = {
    self,
    nixpkgs,
    flake-utils,
  }:
    flake-utils.lib.eachDefaultSystem (system: let
      pkgs = import nixpkgs {
        inherit system;
      };
    in
    {
      devShells.default = with pkgs; mkShell {
        packages = [
          pkgs.nodejs
          pkgs.python311
          pkgs.python311Packages.jupyterlab

          # pkgs.deeptools
        ];
      };
    });
}

Steps to reproduce the behavior:

  1. Run nix develop
  2. Run jupyter lab
  3. Create a code cell with something like: 
    print("hi")
  4. Run it and notice that it prints hi
  5. Rejoyce, it's working
  6. Stop jupyter lab
  7. Exit the devShell
  8. Uncomment the reference to deeptools
  9. Repeat from step 1
  10. Notice that this time, the kernel never comes up

Expected behavior

Jupyter Lab should work whether or not deeptools is installed

Logs

The jupyter labs logs do not exactly contain a smoking gun:

[E 2023-10-16 20:59:46.166 ServerApp] Uncaught exception GET /api/kernels/ec555597-a181-40e4-8548-b237c147b230/channels?session_id=b4394471-5548-47bb-8ec4-889546d9f5db (127.0.0.1)
    HTTPServerRequest(protocol='http', host='localhost:8888', method='GET', uri='/api/kernels/ec555597-a181-40e4-8548-b237c147b230/channels?session_id=b4394471-5548-47bb-8ec4-889546d9f5db', version='HTTP/1.1', remote_ip='127.0.0.1')
    Traceback (most recent call last):
      File "/nix/store/8vbhki9i0s7cfylinpznxm680p477gl2-python3.11-tornado-6.3.3/lib/python3.11/site-packages/tornado/web.py", line 1786, in _execute
        result = await result
                 ^^^^^^^^^^^^
      File "/nix/store/qlvmpxhkcwxrp3i5jlmvpm8wf4jyqk49-python3.11-jupyter-server-2.7.3/lib/python3.11/site-packages/jupyter_server/services/kernels/websocket.py", line 64, in get
        await self.pre_get()
      File "/nix/store/qlvmpxhkcwxrp3i5jlmvpm8wf4jyqk49-python3.11-jupyter-server-2.7.3/lib/python3.11/site-packages/jupyter_server/services/kernels/websocket.py", line 59, in pre_get
        await self.connection.prepare()
      File "/nix/store/qlvmpxhkcwxrp3i5jlmvpm8wf4jyqk49-python3.11-jupyter-server-2.7.3/lib/python3.11/site-packages/jupyter_server/services/kernels/connection/channels.py", line 312, in prepare
        raise TimeoutError(msg)
    TimeoutError: Kernel never reached an 'alive' state.

Additional context

I think that this problem is similar to the one solved here: https://github.com/NixOS/nixpkgs/pull/235123

Here is a workaround:

  ...
  outputs = {
    self,
    nixpkgs,
    flake-utils,
  }:
    flake-utils.lib.eachDefaultSystem (system: let
      overlay = self: super: {
        deeptools = super.deeptools.overrideAttrs (oldAttrs: {
          postFixup = oldAttrs.postFixup or "" + ''
            rm $out/nix-support/propagated-build-inputs
          '';
        });
      };
      pkgs = import nixpkgs {
        inherit system;
        overlays = [ overlay ];
      };
  ...

I can't be sure that $out/nix-support/propagated-build-inputs didn't break something, but the tool I need, bamCoverage seems to work.

Notify maintainers

@scalavision

Metadata

❯ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.1.52, NixOS, 23.11 (Tapir), 23.11.20230911.3a2786e`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.17.0`
 - channels(root): `"home-manager, nixos"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`
scalavision commented 1 year ago

I think your are right, I can reproduce the issue. Do you have any more pointers to what's wrong? Is it mix of different python versions used by the tools and jupyter lab? Deeptools is a fairly complex build. I can try to upgrade the tool during the next day, maybe it helps?

scalavision commented 1 year ago

Sorry, I didn't see this:

https://github.com/NixOS/nixpkgs/pull/251418

Really nice work, do you think this would fix the issue?

MatrixManAtYrService commented 1 year ago

Pre-commit is a weird case. I thought I fixed it with https://github.com/NixOS/nixpkgs/pull/235123 by making it more self-contained, but that caused other problems which I thought I fixed with https://github.com/NixOS/nixpkgs/pull/251418 which adds special cases in which it it it less self contained.

I don't know deeptools well, but I somewhat doubt that the contents of https://github.com/NixOS/nixpkgs/pull/251418 are relevant to it (unless it somewhere has to call a program which is not expected to be packaged alongside it--that's what I broke).

My impression is that deeptools contains utilities that are run as standalone progams. That is, like so:

$ bamCoverage

And not imported like a module:

from deeptools import bamCoverage

So the ideal case is that the executables that end up on the user's PATH are wrapper scripts which put the various python dependencies on the PATH only for the process which is executing that particular... deep tool.

Since the deeptools derivation uses propagatedBuildInputs, the paths to these dependencies are being passed along to jupyter-lab as if it needs them. As it turns out, it depends on something else, and the versions from deeptools are breaking it. I think the fix should be as simple as deleting the propagatedBuildInputs in a build step, just like I have done in https://github.com/NixOS/nixpkgs/pull/235123. That way buildPythonApplication includes them in the wrapper script that it creates (i.e. they become available at runtime when the wrapper gets called), but they don't propagate so far that they end up interfering with jupyter-lab.

Perhaps this is obvious, but when I first encountered this kind of problem I benefitted from looking at the wrapper script which the problematic package generates:

❯ nix-shell -p deeptools
these 2 paths will be fetched (1.33 MiB download, 8.61 MiB unpacked):
  /nix/store/40dlqj7irwh0iihnb5ykcs16950qrxg0-deepTools-3.5.1
  /nix/store/8pzf49yq6v0ki772fhpgvrvn053gmbpz-bash-interactive-5.2-p15
copying path '/nix/store/8pzf49yq6v0ki772fhpgvrvn053gmbpz-bash-interactive-5.2-p15' from 'https://cache.nixos.org'...
copying path '/nix/store/40dlqj7irwh0iihnb5ykcs16950qrxg0-deepTools-3.5.1' from 'https://cache.nixos.org'...

[nix-shell:~/src/nixpkgs]$ cat $(which bamCoverage)
#! /nix/store/3ln9yvs9pg78qvabh89y0a4m1hqzyl1h-bash-5.2-p15/bin/bash -e
PATH=${PATH:+':'$PATH':'}
PATH=${PATH/':''/nix/store/fabx2mv2akxbci8274l05p1fm9va68qc-python3.10-fonttools-4.38.0/bin'':'/':'}
PATH='/nix/store/fabx2mv2akxbci8274l05p1fm9va68qc-python3.10-fonttools-4.38.0/bin'$PATH
PATH=${PATH#':'}
PATH=${PATH%':'}
export PATH
PATH=${PATH:+':'$PATH':'}
PATH=${PATH/':''/nix/store/77jkj30xrd566wpdj66rz21g8zz4nvxs-python3.10-cython-0.29.36/bin'':'/':'}
PATH='/nix/store/77jkj30xrd566wpdj66rz21g8zz4nvxs-python3.10-cython-0.29.36/bin'$PATH
PATH=${PATH#':'}
PATH=${PATH%':'}
export PATH
PATH=${PATH:+':'$PATH':'}
PATH=${PATH/':''/nix/store/4y06nybvndzax9flil8pipbbqdgd707w-python3.10-pbr-5.11.1/bin'':'/':'}
PATH='/nix/store/4y06nybvndzax9flil8pipbbqdgd707w-python3.10-pbr-5.11.1/bin'$PATH
PATH=${PATH#':'}
PATH=${PATH%':'}
export PATH
PATH=${PATH:+':'$PATH':'}
PATH=${PATH/':''/nix/store/j5wpnynag3sbhl4fkx6lw7dw4gysg3cj-python3.10-charset-normalizer-3.0.1/bin'':'/':'}
PATH='/nix/store/j5wpnynag3sbhl4fkx6lw7dw4gysg3cj-python3.10-charset-normalizer-3.0.1/bin'$PATH
PATH=${PATH#':'}
PATH=${PATH%':'}
export PATH
PATH=${PATH:+':'$PATH':'}
PATH=${PATH/':''/nix/store/2ddkqq4y94dbh41hdj9b5w5mz86hqqz3-python3.10-docutils-0.19/bin'':'/':'}
PATH='/nix/store/2ddkqq4y94dbh41hdj9b5w5mz86hqqz3-python3.10-docutils-0.19/bin'$PATH
PATH=${PATH#':'}
PATH=${PATH%':'}
export PATH
PATH=${PATH:+':'$PATH':'}
PATH=${PATH/':''/nix/store/2ikwq24yd21i516hz2k6c6xg7vmshgm2-python3.10-pygments-2.15.1/bin'':'/':'}
PATH='/nix/store/2ikwq24yd21i516hz2k6c6xg7vmshgm2-python3.10-pygments-2.15.1/bin'$PATH
PATH=${PATH#':'}
PATH=${PATH%':'}
export PATH
PATH=${PATH:+':'$PATH':'}
PATH=${PATH/':''/nix/store/yxzrgs9h0pf51g2n9h139z71y9xvbfvm-python3.10-sphinx-5.3.0/bin'':'/':'}
PATH='/nix/store/yxzrgs9h0pf51g2n9h139z71y9xvbfvm-python3.10-sphinx-5.3.0/bin'$PATH
PATH=${PATH#':'}
PATH=${PATH%':'}
export PATH
PATH=${PATH:+':'$PATH':'}
PATH=${PATH/':''/nix/store/nk5z9mjm2xxmg2yvkgv7ap4dxrfjsndw-python3.10-babel-2.12.1/bin'':'/':'}
PATH='/nix/store/nk5z9mjm2xxmg2yvkgv7ap4dxrfjsndw-python3.10-babel-2.12.1/bin'$PATH
PATH=${PATH#':'}
PATH=${PATH%':'}
export PATH
PATH=${PATH:+':'$PATH':'}
PATH=${PATH/':''/nix/store/x80hp101kwdc8f6218lh1nbx5k1119kf-python3.10-numpy-1.25.1/bin'':'/':'}
PATH='/nix/store/x80hp101kwdc8f6218lh1nbx5k1119kf-python3.10-numpy-1.25.1/bin'$PATH
PATH=${PATH#':'}
PATH=${PATH%':'}
export PATH
PATH=${PATH:+':'$PATH':'}
PATH=${PATH/':''/nix/store/40dlqj7irwh0iihnb5ykcs16950qrxg0-deepTools-3.5.1/bin'':'/':'}
PATH='/nix/store/40dlqj7irwh0iihnb5ykcs16950qrxg0-deepTools-3.5.1/bin'$PATH
PATH=${PATH#':'}
PATH=${PATH%':'}
export PATH
PATH=${PATH:+':'$PATH':'}
PATH=${PATH/':''/nix/store/aa3nnkfyif67k6861vd77g4cm4rgbqh8-python3-3.10.12/bin'':'/':'}
PATH='/nix/store/aa3nnkfyif67k6861vd77g4cm4rgbqh8-python3-3.10.12/bin'$PATH
PATH=${PATH#':'}
PATH=${PATH%':'}
export PATH
export PYTHONNOUSERSITE='true'
exec -a "$0" "/nix/store/40dlqj7irwh0iihnb5ykcs16950qrxg0-deepTools-3.5.1/bin/.bamCoverage-wrapped"  "$@"

So there they are, the packages which bamCoverage expects to have on its path. And here they are, also in the devShell:

Screen Shot 2023-10-25 at 8 41 33 PM

So the solution I'm suggesting is to change the derivation such that these dependencies stay in the wrapper script, but are not in the devShell. If you don't get around to it, I'll try to revisit this in a few weeks.

Also, be aware that I'm not blocked by this. I have submitted that homework assignment and moved on. I just reported the issue because it felt like the helpful thing to do. It's possible that the real bug here is that I'm trying to use a devShell as a way to hack dependencies into a jupyter notebook in the first place. I've been meaning to study https://github.com/tweag/jupyenv, which likely provides a smarter way to do this.

I just wanted to point this out because if you decided that this wasn't a bug after all, I wouldn't argue with you.