Currently, TRs have a pip install statement where the packages that are required for the notebook itself. The packages are pinned to a specific version in an attempt to make the notebook reproducible, but indirect dependencies are not pinned. This results in bugs like ComPWA/compwa.github.io#272.
Unfortunately, Jupyter notebooks do not support launching sub-kernels. That would be the ideal solution: you have a main developer environment for the repository that you can for instance use to start Jupyter Lab, and then each notebook launches a subkernel with it's own specific set of dependencies.
The closest we can get to an isolated kernel is by doing a pip install with --prefix and then using sys.path.insert to import from that directory. You can create a constraint file for all indirect dependencies with pip compile (or faster: uv pip compile). Providing a constraint file for each notebook then makes each technical report reproducible.
Disadvantages:
The notebook cannot be run on its own: you need the constraint file. That works fine on Binder, but Google Colab (and Deepnote?) only have access to the notebook, not the whole repo.
You need to pip compile the constraint files. Is that done through the notebook as well? How to provide a mechanism where the writer of the report can specify which direct dependencies are required and then compile a constraint file from that?
redeboer
commented
1 day ago
Currently, TRs have a
pip installstatement where the packages that are required for the notebook itself. The packages are pinned to a specific version in an attempt to make the notebook reproducible, but indirect dependencies are not pinned. This results in bugs like ComPWA/compwa.github.io#272.Unfortunately, Jupyter notebooks do not support launching sub-kernels. That would be the ideal solution: you have a main developer environment for the repository that you can for instance use to start Jupyter Lab, and then each notebook launches a subkernel with it's own specific set of dependencies.
The closest we can get to an isolated kernel is by doing a pip install with
--prefixand then usingsys.path.insertto import from that directory. You can create a constraint file for all indirect dependencies withpip compile(or faster:uv pip compile). Providing a constraint file for each notebook then makes each technical report reproducible.Disadvantages:
pip compilethe constraint files. Is that done through the notebook as well? How to provide a mechanism where the writer of the report can specify which direct dependencies are required and then compile a constraint file from that?