Support for package data

[nthiery]

Description

I have a Python package which includes package data, to be accessed through importlib.resources. However, within jupyterlite, importlib.resources fails do find the data.

Reproduce

1. Clone https://github.com/jupyterlite/xeus-python-demo

2. Add these dependencies to environment.yml

     - pip
     - pip:
       - "git+https://gitlab.dsi.universite-paris-saclay.fr/nicolas.thiery/Laby.git"

3. Build JupyterLite

   mamba env create --force -f build-environment.yml
   conda activate build-env
   cp README.md content
   jupyter lite build --contents content --output-dir dist

4. Check the log, and notice the copying of the package data when building the wheel:

   creating build/bdist.linux-x86_64/wheel/laby/resources/tiles_png
   copying build/lib/laby/resources/tiles_png/nweb.png -> build/bdist.linux-x86_64/wheel/laby/resources/tiles_png
   ...

5. Browse dist/extensions/@jupyterlite/xeus-python-kernel/static/laby_python-0.0.5-pip.tar.gz

   Expected behavior: the archive contain the package data. Got: the package contains only the Python code

6. Run JupyterLite and from a python kernel run:

    tiles_dir = importlib.resources.files("laby").joinpath("resources", "tiles_png")
    list(tiles_dir.glob("*"))

   Expected behavior: a list of files including 'nweb.png`.
   Got: the empty list

Context

• JupyterLite version: 0.1.2
• Operating System and version: Ubuntu 22.04
• Browser and version: Firefox/chromium

[martinRenou]

Thank you for opening an issue. Would you be able to confirm that os.path.exists("/lib/python3.10/site-packages/laby/resources/tiles_png/nweb.png") returns True?

I misread the issue. This will definitely return False if

Expected behavior: the archive contain the package data. Got: the package contains only the Python code

So this has to do with either the empack filtering logic or the way we add package data for the pip dependencies. I'd bet on the empack filtering logic.

[nthiery]

Hello! Bumping this issue, as this is a blocker for making our «Introduction to programming Python course» available outside of Paris-Saclay using JupyterLite.

Is this a hard problem? Do we have some estimate of when this could be fixed?

Thank you in advance!

[DerThorsten]

As a workaround , one can mount any file/dir "by hand"

https://github.com/jupyterlite/xeus?tab=readme-ov-file#mounting-additional-files

jupyter lite build \
    --XeusAddon.environment_file=environment.yaml \
    --XeusAddon.mounts=/some/path/on/host_machine:/some/path/in/virtual/filesystem

@martinRenou reminder: we should get rid of empack completely and just load the conda-pkg itself (and do the relocation on the fly), or at least completely drop the empack filtering...

[martinRenou]

or at least completely drop the empack filtering...

Yeah 👍🏽

I'd be curious to see what impact removing filtering would have in terms of size, but it would definitely make this situation better.

[martinRenou]

I think I commented this somewhere, but I believe the filtering logic should be an opt-in and not an opt-out. We should not filter any file unless explicitly stated.

[martinRenou]

I can have a look at this

[nthiery]

Thank you @DerThorsten and @martinRenou for the quick feedback! Happy to beta test anytime.

[martinRenou]

Released empack with the fix and bumping the dependency in https://github.com/jupyterlite/xeus/pull/96

[nthiery]

I confirm that this works with #96 on my use case. Closing ticket. Thank you!!!