Kernel binaries missing #61

Closed martinRenou closed 2 months ago

Originally opened by @Alberto199810 in voici-demo https://github.com/voila-dashboards/voici-demo/issues/23

Hello guys, I'm working on GitLab, and I'm using your library to deploy the Jupyter Notebook on GL Pages. The Pipeline end in success, and everything seems fine, apart from the fact that when I try to launch the page, it is stuck at the "Running" phase.

It seemed to me that the Kernel was not starting, and indeed in the deploy pipeline I've found this row: """ {...}/python3.10/site-packages/jupyterlite_xeus/add_on.py: UserWarning: kernel binaries not found for xpython-raw """

And then again same thing:

""" {...}/python3.10/site-packages/jupyterlite_xeus/add_on.py: UserWarning: kernel binaries not found for xpython """

So, my feeling is that apparently it is not recognizing the kernel. Any idea in how to solve this?

Thanks again for opening an issue!

Do you have a link to this deployment or is it private?

We'd need the content of the environment.yml file you used.

Thanks again for opening an issue!

Do you have a link to this deployment or is it private?

We'd need the content of the environment.yml file you used.

Unfortunately it's private. This is the environment.yml: """ name: voici channels:

...

dependencies:

xeus-python
jupyterlite-xeus>=0.1.5,<0.2
pandas <2
numpy
matplotlib
plotly==5.18.0
ipywidgets==8.1.1
statsmodels==0.14.1 """

While build-environment.yml is: """ name: build-env channels:

... dependencies:
python==3.10
pip
jupyterlite-xeus>=0.1.5,<0.2
pip:
- voici>=0.5,<0.6 """

Hope this can help.

Thank you.

I see one weird thing, jupyterlite-xeus should only be in the build environment, not the run environment. Otherwise everything looks good.

You omitted the channels, was that on purpose because you use private channels?

You probably want to make sure to use the emscripten-forge channel.

Thank you.

I see one weird thing, jupyterlite-xeus should only be in the build environment, not the run environment. Otherwise everything looks good.

You omitted the channels, was that on purpose because you use private channels?

You probably want to make sure to use the emscripten-forge channel.

Yes, I omitted the channels due to private channels. But everything works fine, the packages are in my channels.

I tried to even remove the jupyterlite-xeus from the run environment, but still it's not working.

Maybe the mistake is in the .gitlab-ci.yml? """ image: python

pages: stage: deploy before script:

apt-get install -y curl
curl -L {channel}
chmod +x bin/micromamba
export PATH=$PWD/bin:$PATH
eval "$(micromamba shell hook --shell bash)"

script:
micromamba create -n build-env --file build-environment.yml
micromamba activate build-env
voici build --contents content --output-dir public

artifacts: paths:

public/

only:

main """

I really struggle to understand the reason of the error. I took a look at the add_on.py but don't see a way to overcome this.

Are you mirroring the emscripten-forge channel?

The private channels may be the culprit here. If you're mirroring the emscripten-forge channel I wonder if some files are missing from the xeus-python build.

Are you mirroring the emscripten-forge channel?

The private channels may be the culprit here. If you're mirroring the emscripten-forge channel I wonder if some files are missing from the xeus-python build.

Yes, I actually am... I'm using jupyter-xeus==0.1.6 and xeus-python 0.15.13

Maybe it's another weird thing but I see that after creating the environment and downloading the packages, once I run the "voici" commands restarts the downloads from scratch. Seems like he is not using the environment. Can this affect? I don't know. At the end of the installation, it says

""" To activate this environment, use: micromamba activate voici """

Immediately after this, it throws the warning of the kernel binaries not found. Does this ring a bell?

To activate this environment, use: micromamba activate voici

You can ignore this. This is just micromamba creating the environment for the emscripten runtime. You should not try to activate it.

Are you able to download the xeus-python package from your emscripten-forge mirror and make sure it's exactly the same as in https://beta.mamba.pm/get/emscripten-forge/emscripten-wasm32/xeus-python-0.15.13-h1be4ea2_1.tar.bz2 ?

just to double check, you named the file environment.yml and not environment.yaml (since https://github.com/jupyterlite/xeus/blob/de72e20dbe263751aa19aabf1fb91878df03e29a/jupyterlite_xeus/add_on.py#L80C9-L80C25)

just to double check, you named the file environment.yml and not environment.yaml

Taking a mental note that automatically supporting both would be nice

lets not have any default and throw an error when nothing is given

To activate this environment, use:

micromamba activate voici

You can ignore this. This is just micromamba creating the environment for the emscripten runtime. You should not try to activate it.

Are you able to download the xeus-python package from your emscripten-forge mirror and make sure it's exactly the same as in https://beta.mamba.pm/get/emscripten-forge/emscripten-wasm32/xeus-python-0.15.13-h1be4ea2_1.tar.bz2 ?

Thanks for the replies. Im seeing here that my package is stored as xeus-python-0.15.13-py{...}.conda.

Can this create a difference? Unfortunately I struggle to go deeper than this, I cannot find the .tar.bz2 for the xeus-python.

lets not have any default and throw an error when nothing is given

Sounds also good, but another breaking change.

just to double check, you named the file environment.yml and not environment.yaml

(since https://github.com/jupyterlite/xeus/blob/de72e20dbe263751aa19aabf1fb91878df03e29a/jupyterlite_xeus/add_on.py#L80C9-L80C25)

Yes, it's .yml.

Can this create a difference?

IIRC the (new?) .conda format is just a zipped file so you should be able to unzip it and compare its content with the .tar.bz2 I pointed to.

I have a feeling this is not coming from your emscripten-forge mirror though? We don't ship .conda files on emscripten-forge. Not sure how your mirror works but I feel like you may have mirrored conda-forge instead.

Another clue that makes me think you mirrored conda-forge instead is this:

 {...}/python3.10/site-packages/jupyterlite_xeus/add_on.py: UserWarning: kernel binaries not found for xpython-raw

We explicitly remove the xpython-raw kernel in emscripten-forge so jupyter-xeus should really not look for it.

Can this create a difference?

IIRC the (new?) .conda format is just a zipped file so you should be able to unzip it and compare its content with the .tar.bz2 I pointed to.

I have a feeling this is not coming from your emscripten-forge mirror though? We don't ship .conda files on emscripten-forge. Not sure how your mirror works but I feel like you may have mirrored conda-forge instead.

you're right, I think it's mirroring conda-forge most likely. Can this cause a serious difference?

[x] I'm just thinking of way to overcome this. Before, I was using pyodide, but it slows down things quite a lot. Can a previous version of xeus-python work as well? Or what would you suggest? I need the installment of the packages while creating the app, and not at runtime.

is there some way to reproduce this in an open-source fashion? Otherwise, we should close this as this is some problem with your mirroring

is there some way to reproduce this in an open-source fashion? Otherwise, we should close this as this is some problem with your mirroring

Nope. At this stage I am convinced it is an issue of mirroring. But maybe before closing you have some suggestions on what else can be used that is in conda-forge.

you're right, I think it's mirroring conda-forge most likely. Can this cause a serious difference?

Yeah you should also mirror emscripten-forge.

Though I'm a bit skeptical this would fix your issue. You are in a weird situation where micromamba is not respecting the specified platform whatsoever. Otherwise it would complain earlier saying it cannot install xeus-python from the specified channels.

maybe this helps, but for pyjs I create a sphinx documentation using an explicit jupyterlite config https://github.com/emscripten-forge/pyjs/blob/main/docs/conf.py#L34

There I create the prefix first, build & install stuff, and then use that in the docs

Fine, thank you for help. We can close this and I'll try to find an alternative way to achieve the goal, otherwise I'll stick to the "slower" solution, which is still fine.

@martinRenou re: using enviroment.yml, the README of this repo says "From environment.yaml" ?

Also, if you do use environment.yaml, then that config file seems to be used by the Github Action to build the parent Jupyterlite environment?

The README lies about the file extension, the code tells the truth https://github.com/jupyterlite/xeus/blob/main/jupyterlite_xeus/add_on.py#L80

we should fix the README

Answered and the environment thingy was fixed in #92

jupyterlite / xeus

Kernel binaries missing #61