more flexible ways to specify preinstalled packages

[parmentelat]

Problem

as per ,https://jupyterlite-sphinx.readthedocs.io/en/latest/configuration.html#pre-installed-packages> one can specify the preinstalled packages in a file named environment.yml i see 2 limitations:

• given that this filename is already heavily used, it might help if one could decide to store this information under a different filename, (like e.g. jupyterlite.yml)
• also, it would be really helpful to support for a pip-compliant format (like requirements.txt)

• my main use case is for writing courses, and in that sense what is usually referred to as docs is the main folder in my case
• in that context I use environment.yml already to store the "usual" - i.e. not lite - environment to use the material (typically jupyterlab and similar)

Proposed Solution

to this end, I would propose to support a new configuration variable in conf.py named jupyterlite_environment or similar that would use conda or pip depending on the file extension (txt→pip and yml or yaml → conda);

or alternatively a more elaborate scheme where e.g.

jupyterlite_environment = {'pip': 'requirements-lite.txt'}
# or
jupyterlite_environment = {'conda': 'environment-lite.yml'}

I understand that the conda flavour is more powerful in that it contains the python version; however not everybody is fluent in conda, so maybe the pip flavour makes sense too

Additional context

as far as I am concerned, supporting pip is less crucial than just being able to use another yaml file, so this could be implemented in 2 separate steps

[steppi]

Thanks for the suggestion @parmentelat. The instructions to use environment.yml are specific to the xeus kernel, so I think this issue would be a better fit over there.

[parmentelat]

Hi @steppi thanks for pointing me in that direction however as per https://github.com/jupyterlite/xeus there is already a way to redirect xeus to load from some other place, so I'd say that as far as xeus is concerned, the interface is clean

jupyter lite build --XeusAddon.environment_file=some_path/to/environment.yaml

as a jupyterlite-sphinx user, I am not the one who invokes jupyter lite build directly, so what seems to be missing is for me to instruct jupyterlite-sphinx to pass this flag along

as an aside, I can't seem to find any equivalent for when using pyodide under jupyterlite, did I miss somthing ? is that on the roadmap ?

[steppi]

Ah, I see. Thanks @parmentelat. It seems what should be added is a way to pass additional options to jupyter lite build in jupyterlite-sphinx. This would be pretty useful I think. The invocation of jupyter lite build is here:

https://github.com/jupyterlite/jupyterlite-sphinx/blob/53fc56eca5197d20ba7e0b69dce1a82f3c08853f/jupyterlite_sphinx/jupyterlite_sphinx.py#L601-L615

@agriyakhetarpal, do you have bandwidth to work on that? It should be fairly straightforward, except we might want to document how to avoid having options that conflict with the ones that are currently included, and put in place safeguards to prevent conflicting options

as an aside, I can't seem to find any equivalent for when using pyodide under jupyterlite, did I miss somthing ? is that on the roadmap ?

As far as I'm aware, the pyodide kernel doesn't allow for pre-installing packages in the kernel environment like the xeus kernel does. I don't know if there's anything like this on the roadmap for pyodide; my involvement in this project is almost entirely on the Sphinx side, so I'm not well informed on such matters.

[martinRenou]

also, it would be really helpful to support for a pip-compliant format (like requirements.txt)

pip does not know anything about the WASM-based platforms, it just knows Windows and Unix (for now). So supporting requirements.txt in jupyterlite-xeus is possible, though only in a hacky way (not using pip under the hood). There is actually experimental support for pip packages but please don't use it if you can use conda-forge and emscripten-forge instead.

[parmentelat]

thank you @steppi and @martinRenou for your answers

[agriyakhetarpal]

Thanks for tagging me, @steppi. I shall take a look at this very soon – and I'll post here if I have some questions. I had thought of doing this earlier myself and was editing the source code to add additional options in a few cases. Now that we have a feature request for it, it seems worthwhile to add.

[steppi]

@parmentelat. A fix for this has made it into main. See the dev docs here.