Closed michaelweinold closed 11 months ago
michaelweinold
commented
11 months ago ...potentially due to
Only
no-archpackages fromconda-forgeandemscripten-forgecan be installed.
(which was added to the readme recently). @martinRenou, is there a reason you're restricting installs to emscripten-forge and conda-forge channels?
martinRenou
commented
11 months ago Upgrading to xeus-python-kernel>0.9.0 breaks the build process of our site, throwing this error:
- package 'bw2io-0.9.dev21-py_0' is excluded by strict repo priority
I see you closed the issue, did you find the reason for this error?
(which was added to the readme recently). @martinRenou, is there a reason you're restricting installs to emscripten-forge and conda-forge channels?
As long as the package is either no-arch or built for the emscripten platform, you should be able to install packages from any channel. Note that most jupyterlite/xeus-python maintainers don't maintain anything else than conda-forge or emscripten-forge.
michaelweinold
commented
11 months ago We added the dev versions of our packages to the conda-forge channel, thereby circumventing the issue.
As long as the package is either no-arch or built for the emscripten platform, you should be able to install packages from any channel.
This is what we had in the environment.yaml:
name: xeus-python-kernel
channels:
- https://repo.mamba.pm/emscripten-forge # check packages at https://beta.mamba.pm/channels
- https://conda.anaconda.org/conda-forge
- nodefaults
dependencies:
# DATA SCIENCE
- pandas
- numpy
# PLOTTING
- matplotlib
# BRIGHTWAY DEPENDENCIES (NOT PURE PYTHON)
- https://repo.mamba.pm/emscripten-forge::peewee # required by Brightway
- https://repo.mamba.pm/emscripten-forge::lxml # required by Brightway
# BRIGHTWAY CORE
- https://conda.anaconda.org/cmutel::bw2io=0.9.dev21 # https://anaconda.org/cmutel/bw2io/files
- https://conda.anaconda.org/cmutel::bw2data=4.0.0.dev21 # https://anaconda.org/cmutel/bw2data/files
- https://conda.anaconda.org/cmutel::bw2calc=1.0.dev13 # https://anaconda.org/cmutel/bw2calc/fileseven though the packages are all noarch, this raised the
- package 'bw2io-0.9.dev21-py_0' is excluded by strict repo priorityerror. I closed the issue because I only later read this part of the documentation:
Only no-arch packages from conda-forge and emscripten-forge can be installed.
Thereby assuming our packages (at the time) not being on the conda-forge channel was the underlying issue.
michaelweinold
commented
11 months ago It seems that our adding the dev versions of our packages to conda-forge didn't really solve the issue. After this issue:
was solved by an upgrade of empack, we were just able to test the build. Unfortunately, we get the error:
__
__ ______ ___ ____ _____ ___ / /_ ____ _
/ / / / __ `__ \/ __ `/ __ `__ \/ __ \/ __ `/
/ /_/ / / / / / / /_/ / / / / / / /_/ / /_/ /
/ .___/_/ /_/ /_/\__,_/_/ /_/ /_/_.___/\__,_/
/_/
warning libmamba Problem type not implemented SOLVER_RULE_STRICT_REPO_PRIORITY
error libmamba Could not solve for environment specs
The following packages are incompatible
├─ bw2calc 2.0.dev13* does not exist (perhaps a typo or a missing channel);
├─ bw2data 4.0.dev21* is uninstallable because it conflicts with any installable versions previously reported;
└─ bw2io 0.9.dev21* is uninstallable because it requires
└─ bw2calc >=1.7.4 , which does not exist (perhaps a missing channel).If I use either this syntax:
name: xeus-python-kernel
channels:
- https://repo.mamba.pm/emscripten-forge # check packages at https://beta.mamba.pm/channels
- https://conda.anaconda.org/conda-forge
- nodefaults
dependencies:
# DATA SCIENCE
- pandas
- numpy
# PLOTTING
- matplotlib
# BRIGHTWAY DEPENDENCIES (NOT PURE PYTHON)
- peewee # required by Brightway, loaded from emscripten-forge
- lxml # required by Brightway, loaded from emscripten-forge
# BRIGHTWAY CORE
- https://conda.anaconda.org/conda-forge/label/bw2io_dev::bw2io=0.9.dev21 # https://anaconda.org/conda-forge/bw2io/files
- https://conda.anaconda.org/conda-forge/label/bw2data_dev::bw2data=4.0.dev21 # https://anaconda.org/conda-forge/bw2data/files
- https://conda.anaconda.org/conda-forge/label/bw2calc_dev::bw2calc=2.0.dev13 # https://anaconda.org/conda-forge/bw2calc/filesor this syntax:
name: xeus-python-kernel
channels:
- https://repo.mamba.pm/emscripten-forge # check packages at https://beta.mamba.pm/channels
- https://conda.anaconda.org/conda-forge
- https://conda.anaconda.org/conda-forge/label/bw2io_dev
- https://conda.anaconda.org/conda-forge/label/bw2data_dev
- https://conda.anaconda.org/conda-forge/label/bw2calc_dev
- nodefaults
dependencies:
# DATA SCIENCE
- pandas
- numpy
# PLOTTING
- matplotlib
# BRIGHTWAY DEPENDENCIES (NOT PURE PYTHON)
- peewee # required by Brightway, loaded from emscripten-forge
- lxml # required by Brightway, loaded from emscripten-forge
# BRIGHTWAY CORE
- bw2io=0.9.dev21 # https://anaconda.org/conda-forge/bw2io/files
- bw2data=4.0.dev21 # https://anaconda.org/conda-forge/bw2data/files
- bw2calc=2.0.dev13 # https://anaconda.org/conda-forge/bw2calc/filesin the environment.yaml file. We are following CFEP-05 in our use of the dev labels here.
martinRenou
commented
11 months ago This may be a combination of a bug on jupyterlite-xeus-python and a wrong order of channels for you.
Can you try applying the changes from https://github.com/jupyterlite/xeus-python-kernel/pull/149 in your jupyterlite-xeus-python and using the following environment file:
name: xeus-python-kernel
channels:
- https://conda.anaconda.org/conda-forge/label/bw2io_dev
- https://conda.anaconda.org/conda-forge/label/bw2data_dev
- https://conda.anaconda.org/conda-forge/label/bw2calc_dev
- https://repo.mamba.pm/emscripten-forge # check packages at https://beta.mamba.pm/channels
- https://conda.anaconda.org/conda-forge
- nodefaults
dependencies:
# DATA SCIENCE
- pandas
- numpy
# PLOTTING
- matplotlib
# BRIGHTWAY DEPENDENCIES (NOT PURE PYTHON)
- peewee # required by Brightway, loaded from emscripten-forge
- lxml # required by Brightway, loaded from emscripten-forge
# BRIGHTWAY CORE
- bw2io=0.9.dev21 # https://anaconda.org/conda-forge/bw2io/files
- bw2data=4.0.dev21 # https://anaconda.org/conda-forge/bw2data/files
- bw2calc=2.0.dev13 # https://anaconda.org/conda-forge/bw2calc/files...just tried to jupyter lite build the environment file you provided above, using your changes from https://github.com/jupyterlite/xeus-python-kernel/pull/149, but I still get:
error libmamba Could not solve for environment specs
The following packages are incompatible
├─ bw2calc 2.0.dev13** does not exist (perhaps a typo or a missing channel);
├─ bw2data 4.0.dev21** is not installable because it requires
│ └─ requests >=1.1.0 but there are no viable options
│ ├─ requests [2.23.0|2.24.0|...|2.29.0] would require
│ │ └─ urllib3 [>=1.21.1,<1.26,!=1.25.0,!=1.25.1 |>=1.21.1,<1.27 ] but there are no viable options
│ │ ├─ urllib3 [1.24.3|2.0.0|2.0.1|2.0.2|2.0.3] would require
│ │ │ └─ pysocks >=1.5.6,<2.0,!=1.5.7 but there are no viable options
│ │ │ ├─ pysocks 1.7.1 would require
│ │ │ │ └─ __win, which is missing on the system;
│ │ │ └─ pysocks 1.7.1 would require
│ │ │ └─ __unix, which is missing on the system;
│ │ └─ urllib3 [1.25.10|1.25.11|...|1.26.9] would require
│ │ └─ brotlipy >=0.6.0 , which does not exist (perhaps a missing channel);
│ └─ requests 2.31.0 would require
│ └─ urllib3 >=1.21.1,<3 but there are no viable options
│ ├─ urllib3 [1.24.3|2.0.0|2.0.1|2.0.2|2.0.3], which cannot be installed (as previously explained);
│ ├─ urllib3 [1.25.10|1.25.11|...|1.26.9], which cannot be installed (as previously explained);
│ └─ urllib3 [2.0.3|2.0.4] would require
│ └─ brotli-python >=1.0.9 , which does not exist (perhaps a missing channel);
└─ bw2io 0.9.dev21** is not installable because it requires
└─ bw2calc >=1.7.4 , which does not exist (perhaps a missing channel).
critical libmamba Could not solve for environment specsI think this is what's wrong: https://github.com/conda-forge/bw2calc-feedstock/pull/14
bw2calc should probably be noarch, am I correct saying it's pure Python?
michaelweinold
commented
11 months ago It's pure Python, but I think we have multiple platforms because we have to use different linear algebra solvers on different platforms (@cmutel knows more about this). Despite this, I erroneously assumed a noarch build would be available on the conda-forge channel also due to this platform overview on the package site:
But this seems not to be the case.
martinRenou
commented
11 months ago Not being no-arch, the package cannot be installed on the emscripten-32 platform for xeus-python to use it. That is why it's complaining for it not being available.
If your package is pure-python, it should specify noarch in the conda-forge recipe. If it needs to compile code, you would need to contribute your package to emscripten-forge by opening a PR on https://github.com/emscripten-forge/recipes st. it's available for the emscripten-32 platform.
michaelweinold
commented
11 months ago Ok, understood - as I mentioned, I was under the assumption that we were providing a noarch version of bw2calc. I'll try to figure out the details in
and hope this resolves the issue. Thanks!
michaelweinold
commented
11 months ago Not being no-arch, the package cannot be installed on the emscripten-32 platform for xeus-python to use it. That is why it's complaining for it not being available.
@martinRenou, we resolved the issue with bw2calc. The library is now available as noarch under the dev label in the conda-forge channel. However, it seems that there is an issue loading another package, which is already noarch: pysocks
error libmamba Could not solve for environment specs
The following packages are incompatible
├─ bw2data 4.0.dev21** is uninstallable because it requires
│ └─ requests >=1.1.0 but there are no viable options
│ ├─ requests [2.23.0|2.24.0|...|2.29.0] would require
│ │ └─ urllib3 [>=1.21.1,<1.26,!=1.25.0,!=1.25.1 |>=1.21.1,<1.27 ] but there are no viable options
│ │ ├─ urllib3 [1.24.3|2.0.0|2.0.1|2.0.2|2.0.3] would require
│ │ │ └─ pysocks >=1.5.6,<2.0,!=1.5.7 but there are no viable options
│ │ │ ├─ pysocks 1.7.1 would require
│ │ │ │ └─ __win , which is missing on the system;
│ │ │ └─ pysocks 1.7.1 would require
│ │ │ └─ __unix , which is missing on the system;
│ │ └─ urllib3 [1.25.10|1.25.11|...|1.26.9] would require
│ │ └─ brotlipy >=0.6.0 , which does not exist (perhaps a missing channel);
│ └─ requests 2.31.0 would require
│ └─ urllib3 >=1.21.1,<3 but there are no viable options
│ ├─ urllib3 [1.24.3|2.0.0|2.0.1|2.0.2|2.0.3], which cannot be installed (as previously explained);
│ ├─ urllib3 [1.25.10|1.25.11|...|1.26.9], which cannot be installed (as previously explained);
│ └─ urllib3 [2.0.3|2.0.4] would require
│ └─ brotli-python >=1.0.9 , which does not exist (perhaps a missing channel);
├─ bw2io 0.9.dev21** is uninstallable because it requires
│ └─ bw2data >=3.5.1 but there are no viable options
│ ├─ bw2data 4.0.dev21, which cannot be installed (as previously explained);
│ └─ bw2data [3.6.5|3.6.6] would require
│ └─ psutil , which does not exist (perhaps a missing channel);
└─ pysocks 1.7.1** , which cannot be installed (as previously explained).Why can't the package be loaded? It's available as noarch on conda-forge?
martinRenou
commented
11 months ago ├─ pysocks 1.7.1 would require │ └─ win , which is missing on the system; └─ pysocks 1.7.1 would require | └─ unix , which is missing on the system;
cc. @wolfv @DerThorsten it looks like the pysocks recipe depends on a variable named virtual_pkg while rendering the recipe. Does that ring a bell? What should be the equivalent for emscripten-32 and where should we set this (boa?)?.
michaelweinold
commented
11 months ago Related conda documentation:
Will we need to replace the requests library with the internal urllib? We have 3 functions in bw2data using it, so it should be doable.
I also saw that @martinRenou and @DerThorsten created a drop-in replacement for the requests library (altough likely for different reasons): requests-wasm-polyfill
michaelweinold
commented
11 months ago Thank you for the assistance in this issue - since it was originally opened related to the error package is excluded by strict repo priority, I have created three individual issues to track the errors we're seeing now:
These seem now all related to the JupterLite infrastructure rather than our packages.
The upside is that we'll likely be able to use Pyodide 0.24.0 (as soon as it is released) to avoid all of these.
Description
Upgrading to
xeus-python-kernel>0.9.0breaks the build process of our site, throwing this error:Full Output
``` [LiteBuildApp] No archive (yet): brightway-hub-jupyterlite.tgz [LiteBuildApp] jupyter-lite.(json|ipynb): 1 files /home/runner/micromamba-root/envs/build-env/lib/python3.10/site-packages/jupyterlite_core/optional.py:22: UserWarning: install libarchive-c for better perfomance when working with archives: No module named 'libarchive' warnings.warn(hint.format(error=error)) /home/runner/micromamba-root/envs/build-env/lib/python3.10/site-packages/jupyterlite_core/optional.py:22: UserWarning: [lite] [translation] install `jupyterlab_server` to load translations: No module named 'jupyterlab_server' warnings.warn(hint.format(error=error)) __ __ ______ ___ ____ _____ ___ / /_ ____ _ / / / / __ `__ \/ __ `/ __ `__ \/ __ \/ __ `/ / /_/ / / / / / / /_/ / / / / / / /_/ / /_/ / / .___/_/ /_/ /_/\__,_/_/ /_/ /_/_.___/\__,_/ /_/ error libmamba Error opening for writing "/tmp/xeus-python-kernel/envs/xeus-python-kernel/.mambarc": No such file or directory error libmamba Selected channel specific (or force-reinstall) job, but package is not available from channel. Solve job will fail. error libmamba Selected channel specific (or force-reinstall) job, but package is not available from channel. Solve job will fail. Encountered problems while solving: - package 'bw2io-0.9.dev21-py_0' is excluded by strict repo priority The environment can't be solved, aborting the operation error libmamba Could not solve for environment specs critical libmamba UnsatisfiableError [LiteBuildApp] ERROR | [lite] [post_build] [jupyterlite-xeus-python] [ERR] Command '['/home/runner/micromamba-bin/micromamba', 'create', '--yes', '--root-prefix', '/tmp/xeus-python-kernel', '--name', 'xeus-python-kernel', '--platform=emscripten-32', '-c', 'https://repo.mamba.pm/emscripten-forge', '-c', 'https://repo.mamba.pm/conda-forge', '-c', 'https://conda.anaconda.org/conda-forge', '-c', 'nodefaults', 'python=3.10', 'xeus-lite', 'xeus-python=0.15.9', 'pandas', 'numpy', 'matplotlib', 'https://repo.mamba.pm/emscripten-forge::peewee', 'https://repo.mamba.pm/emscripten-forge::lxml', 'https://conda.anaconda.org/cmutel::bw2io=0.9.dev21', 'https://conda.anaconda.org/cmutel::bw2data=4.0.0.dev21', 'https://conda.anaconda.org/cmutel::bw2calc=1.0.dev13']' returned non-zero exit status 1. Traceback (most recent call last): File "/home/runner/micromamba-root/envs/build-env/lib/python3.10/site-packages/doit/doit_cmd.py", line 294, in run return command.parse_execute(args) File "/home/runner/micromamba-root/envs/build-env/lib/python3.10/site-packages/doit/cmd_base.py", line 150, in parse_execute return self.execute(params, args) File "/home/runner/micromamba-root/envs/build-env/lib/python3.10/site-packages/doit/cmd_base.py", line 570, in execute return self._execute(**exec_params) File "/home/runner/micromamba-root/envs/build-env/lib/python3.10/site-packages/doit/cmd_run.py", line 265, in _execute return runner.run_all(self.control.task_dispatcher()) File "/home/runner/micromamba-root/envs/build-env/lib/python3.10/site-packages/doit/runner.py", line 254, in run_all self.run_tasks(task_dispatcher) File "/home/runner/micromamba-root/envs/build-env/lib/python3.10/site-packages/doit/runner.py", line 213, in run_tasks node = task_dispatcher.generator.send(node) File "/home/runner/micromamba-root/envs/build-env/lib/python3.10/site-packages/doit/control.py", line 629, in _dispatcher_generator next_step = node.step() File "/home/runner/micromamba-root/envs/build-env/lib/python3.10/site-packages/doit/control.py", line 336, in step return next(self.generator) File "/home/runner/micromamba-root/envs/build-env/lib/python3.10/site-packages/doit/control.py", line 345, in _func for value in decorated(*args, **kwargs): File "/home/runner/micromamba-root/envs/build-env/lib/python3.10/site-packages/doit/control.py", line 473, in _add_task new_tasks = generate_tasks(to_load, task_gen, ref.__doc__) File "/home/runner/micromamba-root/envs/build-env/lib/python3.10/site-packages/doit/loader.py", line 390, in generate_tasks for task_dict, x_doc in flat_generator(gen_result, gen_doc): File "/home/runner/micromamba-root/envs/build-env/lib/python3.10/site-packages/doit/loader.py", line 27, in flat_generator for item in gen: File "/home/runner/micromamba-root/envs/build-env/lib/python3.10/site-packages/jupyterlite_core/manager.py", line 145, in _delayed_gather for task in _gather(): File "/home/runner/micromamba-root/envs/build-env/lib/python3.10/site-packages/jupyterlite_core/manager.py", line 138, in _gather raise error File "/home/runner/micromamba-root/envs/build-env/lib/python3.10/site-packages/jupyterlite_core/manager.py", line 128, in _gather for task in getattr(addon, attr)(self): File "/home/runner/micromamba-root/envs/build-env/lib/python3.10/site-packages/jupyterlite_xeus_python/env_build_addon.py", line 79, in post_build env_prefix = build_and_pack_emscripten_env( File "/home/runner/micromamba-root/envs/build-env/lib/python3.10/site-packages/jupyterlite_xeus_python/build.py", line 249, in build_and_pack_emscripten_env raise e File "/home/runner/micromamba-root/envs/build-env/lib/python3.10/site-packages/jupyterlite_xeus_python/build.py", line 193, in build_and_pack_emscripten_env create_env(env_name, root_prefix, specs, channels) File "/home/runner/micromamba-root/envs/build-env/lib/python3.10/site-packages/jupyterlite_xeus_python/build.py", line 69, in create_env run( File "/home/runner/micromamba-root/envs/build-env/lib/python3.10/subprocess.py", line 526, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['/home/runner/micromamba-bin/micromamba', 'create', '--yes', '--root-prefix', '/tmp/xeus-python-kernel', '--name', 'xeus-python-kernel', '--platform=emscripten-32', '-c', 'https://repo.mamba.pm/emscripten-forge', '-c', 'https://repo.mamba.pm/conda-forge', '-c', 'https://conda.anaconda.org/conda-forge', '-c', 'nodefaults', 'python=3.10', 'xeus-lite', 'xeus-python=0.15.9', 'pandas', 'numpy', 'matplotlib', 'https://repo.mamba.pm/emscripten-forge::peewee', 'https://repo.mamba.pm/emscripten-forge::lxml', 'https://conda.anaconda.org/cmutel::bw2io=0.9.dev21', 'https://conda.anaconda.org/cmutel::bw2data=4.0.0.dev21', 'https://conda.anaconda.org/cmutel::bw2calc=1.0.dev13']' returned non-zero exit status 1. static:jupyter-lite.json . pre_status:static:jupyter-lite.json tarball: jupyterlite-app-0.1.1.tgz 7MB output: /home/runner/work/brightway-hub/brightway-hub/dist lite dir: /home/runner/work/brightway-hub/brightway-hub apps: sourcemaps: True unused packages: True archive:archive contents:contents lite:jupyter-lite.json mathjax:status mimetypes:jupyter-lite.json serve:contents settings:overrides translation:translation . status:archive:archive . status:contents:contents contents: 3 files . status:lite:jupyter-lite.json . status:mathjax:status jupyter-server-mathjax: None configured MathJax dir: /home/runner/work/brightway-hub/brightway-hub/traitlets.Undefined effective MathJax dir: None . status:mimetypes:jupyter-lite.json filetypes: 26 . status:serve:contents url: http://127.0.0.1:8000/ server: tornado headers: . status:settings:overrides overrides.json: 0 . status:translation:translation translation files: 0 files static:output_dir . pre_init:static:output_dir static:unpack . init:static:unpack federated_extensions:copy:ext:jupyterlab_pygments federated_extensions:copy:ext:@jupyterlite/xeus-python-kernel . pre_build:federated_extensions:copy:ext:jupyterlab_pygments . pre_build:federated_extensions:copy:ext:@jupyterlite/xeus-python-kernel contents:copy:README.md contents:copy:demo.ipynb contents:copy:peewee_test.ipynb lite:patch:jupyter-lite.json translation:copy . build:contents:copy:README.md . build:contents:copy:demo.ipynb . build:contents:copy:peewee_test.ipynb . build:lite:patch:jupyter-lite.json MERGED /home/runner/work/brightway-hub/brightway-hub/dist/jupyter-lite.json from [PosixPath('/home/runner/work/brightway-hub/brightway-hub/dist/jupyter-lite.json'), PosixPath('/home/runner/work/brightway-hub/brightway-hub/jupyter-lite.json')] . build:translation:copy contents:contents:. federated_extensions:patch jupyterlite-xeus-python:xeus:copy:ext:@jupyterlite/xeus-python-kernel Error: Process completed with exit code 3. ```Reproduce
For a working version, compare branch
mainof our repo. For the broken version, compare branchenvironment_upgradebranch of our repo.build-environment.yml(this breaks the site build):build-environment.yml(this works fine):environment.yml:Expected behavior
Site build completes successfully.
Context
0.1.1Version 16.6 (18615.3.12.11.2)Browser Output