Cannot create conda environment with `conda-environment.yml` for Python 3.8 or Python 3.9

[fabcor-maxiv]

As spotted in https://github.com/mxcube/mxcubeweb/issues/1380#issue-2492532184 and https://github.com/mxcube/mxcubeweb/pull/1382#discussion_r1738507140, it appears that it is now impossible to install conda-environment.yml for Python 3.8 nor Python 3.9.

[fabcor-maxiv]

It used to be possible. I used to run commands such as micromamba create -f conda-environment.yml 'python==3.8' regularly without any issue. But today it does not resolve. I do not know when it stopped working.

[fabcor-maxiv]

See also: https://github.com/mxcube/mxcubeweb/issues/1384

[fabcor-maxiv]

Seems like it works without issues today. I had not written down what was the exact issue that was happenning at the time (what packages could not resolve). But today all of those seem to work fine:

• micromamba create --file conda-environment.yml 'python=3.8'`

• micromamba create --file conda-environment.yml 'python=3.9'`

• conda env create --file conda-environment.yml
  conda install --name mxcubeweb 'python=3.8'

• conda env create --file conda-environment.yml
  conda install --name mxcubeweb 'python=3.9'

[fabcor-maxiv]

@marcus-oscarsson @woutdenolf can you confirm?

[marcus-oscarsson]

I tried:

conda env create --file conda-environment.yml
conda install --name mxcubeweb 'python=3.9'

It takes forever to resolve conda install --name mxcubeweb 'python=3.9', did someone else try ? @walesch-yan, @beteva ?

[walesch-yan]

I tried:

conda env create --file conda-environment.yml
conda install --name mxcubeweb 'python=3.9'

It takes forever to resolve conda install --name mxcubeweb 'python=3.9', did someone else try ? @walesch-yan, @beteva ?

I tried this and it worked for me (python 3.8 also works). Note: I'm using miniconda

[marcus-oscarsson]

Ill try again, but then we have confirmed that the above works. Its maybe just my connection that was slow somehow. I also usually use micromamba because its much faster, we should perhaps recommend to use that instead in that case ? Because it seems like that there are several people that prefers micromamba

[walesch-yan]

I ran into another issue afterwards, when installing with python 3.8 or 3.9. Running poetry install didn't work for me as the following two modules were not correctly installed by conda: filelock and packaging

[marcus-oscarsson]

So for me conda still does not work, it just takes forever to install python, nothing happens in about 20 minutes. @walesch-yan is using miniconda so that is perhaps why the install process works for him, or maybe I have some strange configuration that makes things extremely slow. Although it seems to work for other things.

However, I usually use micromamba and micromamba create --file conda-environment.yml 'python=3.8' does work great.

So how about then finally propose to use micromamba instead, seeing that conda seems to be problematic and that the dependencies does not seem to be installed correctly in the end (As observed by @walesch-yan) ?

[fabcor-maxiv]

I ran into another issue afterwards, when installing with python 3.8 or 3.9. Running poetry install didn't work for me as the following two modules were not correctly installed by conda: filelock and packaging

This is worrying. Seems like it is a different issue. I hope I can find time to investigate this. Can you open a ticket for this, @walesch-yan?

So how about then finally propose to use micromamba instead, seeing that conda seems to be problematic and that the dependencies does not seem to be installed correctly in the end (As observed by @walesch-yan) ?

I'd prefer if we had a solution that works with everything (in principle it should work with everything, I do not see any reason why, what we are doing wrong). And in general I prefer when projects stay out of the business of recommending one tool over another. If it were me I would remove any mention of conda, mamba, micromamba in our documentation because it is not a strict requirement. But seems like in this case it is better that we do give one recommendation that we know works.

I can update the documentation.

[marcus-oscarsson]

If it were me I would remove any mention of conda, mamba, micromamba in our documentation because it is not a strict requirement. But seems like in this case it is better that we do give one recommendation that we know works.

What we were aiming for was to give the developers in the community a good working solution, also especially since some developers does not necessarily have a developer background. What you would prefer if I understand you correctly then is to simply state; "Install the project using your favorite virtual environment tool" ?

[rhfogh]

I would really vote for having a documented solution that is known to work - and that is fixed if it breaks. Preferably with conda, since a lot of people (myself included) use it already. Navigating all these dependencies is not for the fainthearted, so it is not optimal to let each person sort it out for himself.

[fabcor-maxiv]

What you would prefer if I understand you correctly then is to simply state; "Install the project using your favorite virtual environment tool" ?

Yes. But of course I won't. ; )

"If it were only me" is what I meant. : D

Of course, I had understood that for this particular project and its audience it makes sense to have those helpers and instructions. But I assume you also get my point that maintaining those things seems a bit of a pointless exercise at times. I assumed most people use conda so I wrote conda instructions, but I do not use it myself, and the instructions were wrong for an apparently very long time and no one noticed although a lot of people claim to be following the instructions from this document (according to the questionnaire). So I am confused... : D

I would recommend micromamba to everyone but I also do not want to be responsible, do support, and hand-holding for all this. So if conda is what people are comfortable with, then conda it is. I always assumed mamba and micromamba were drop-in replacements for conda but now I have confirmation that it is not so simple, there are a bunch of incompatibilities and diverging behaviours.

I will rewrite this and actually test it with conda this time. : D

[marcus-oscarsson]

I see, I think people don't really change the python version and use whatever is in the conda-environment.yml file and that's why it works. Its great if we can get it to work with conda because then its very likely that its going to work with the other "drop in" replacement tools.

Perhaps the easiest solution is in the end to state that you need to update the conda-environment file if you like a specific version of python.

I don't mind taking care of that section of the documentation if needed. I think everybody otherwise understands and appreciates that the documentation is of best effort nature and that it was working when it was written. During the course of time software changes and those changes might not always be reflected in the documentation.

As you mentioned above the mxcube collaboration is much more than just the mxcube software itself. I think we can save a tremendous amount of time an effort by acknowledging that.

[fabcor-maxiv]

Anyways... back to the actual topic of this ticket: the conda resolution issue...

I was able to create conda environments for Python 3.8 and 3.9 multiple times. Then one time it did fail again with some resolution issue. I retried the exact same command and it succeeded. It feels like there is some issue that is beyond our understanding and reach. Something transient maybe. Maybe depends on the cache. No idea.

That was the trace for the failed attempt:

$ conda install --name mxcubeweb 'python=3.8'
Channels:
 - maxiv-kits
 - mini-conda-forge
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: failed

LibMambaUnsatisfiableError: Encountered problems while solving:
  - package python-ldap-3.4.3-py39hd1e30aa_0 requires python_abi 3.9.* *_cp39, but none of the providers can be installed

Could not solve for environment specs
The following packages are incompatible
├─ python-ldap 3.4.3  is installable with the potential options
│  ├─ python-ldap 3.4.3 would require
│  │  └─ python_abi 3.9.* *_cp39, which can be installed;
│  └─ python-ldap 3.4.3 would require
│     └─ python_abi 3.11.* *_cp311, which can be installed;
└─ python 3.8**  is not installable because it requires
   └─ python_abi 3.8.* *_cp38, which conflicts with any installable versions previously reported.

Maybe it is because I do not have conda-forge in my default channels (as can be seen in this trace output). I do not know.

These were the commands:

conda env create --file conda-environment.yml
conda install --name mxcubeweb 'python=3.8'

For whom is it still failing? @marcus-oscarsson ?

[woutdenolf]

For reference I'm using miniforge3 which only has conda-forge in the channels by default, not default.

Python 3.8

This installs fine but does not run:

cd mxcubeweb
conda clean -a -y
conda env create --file conda-environment.yml
conda activate mxcubeweb
conda install -y 'python=3.8'
poetry install
pnpm --prefix ui install
pnpm --prefix ui build
cd ..
mxcubeweb-server -r $(pwd)/mxcubeweb/demo/ --static-folder $(pwd)/mxcubeweb/ui/build/ -L debug

Traceback (most recent call last):
  File "/home/denolf/.local/wdnbashutils/virtualenvs/mxcubeweb/bin/mxcubeweb-server", line 3, in <module>
    from mxcubeweb import main
  File "/home/denolf/projects/sb/mxcubeweb/mxcubeweb/__init__.py", line 14, in <module>
    from mxcubeweb.app import MXCUBEApplication as mxcube  # noqa: E402
  File "/home/denolf/projects/sb/mxcubeweb/mxcubeweb/app.py", line 36, in <module>
    from mxcubeweb.core.components.harvester import Harvester
  File "/home/denolf/projects/sb/mxcubeweb/mxcubeweb/core/components/harvester.py", line 14, in <module>
    class Harvester(ComponentBase):
  File "/home/denolf/projects/sb/mxcubeweb/mxcubeweb/core/components/harvester.py", line 189, in Harvester
    def get_sample_info(self, sampleID: str) -> dict[str]:
TypeError: 'type' object is not subscriptable

These type annotations need

from __future__ import annotations

Python 3.9

This takes too long:

cd mxcubeweb
conda clean -a -y
conda env create --file conda-environment.yml
conda activate mxcubeweb
conda install -y 'python=3.9'
poetry install  # building opencv 100%CPU, 100%MEM -> CTRL-C no time

That's all I have time for today.

[rhfogh]

Surely the problem with Python 3.8 -0s not a matter of conda - but a problem that the python code is incompatible with Python 3.8 in the first place?

[woutdenolf]

but a problem that the python code is incompatible with Python 3.8 in the first place?

at this point in time yes: it installs but is not compatible with python 3.8 (but could be when adding from __future__ import annotations)

[fabcor-maxiv]

Argh... people are conflating different issues into one.

This ticket is about not being able to select or switch Python version (to 3.8 or 3.9?) inside the conda environment.

If everyone is able to either:

• (with micromamba) create the environment for a specific Python version (with micromamba)
• or (with conda) create the environment (with default Python version) and then switch to a different Python version

then we can close this ticket.

So here in this ticket let's leave out Poetry, pnpm, and even running the application. One thing at a time. Otherwise it gets too complex, especially since everyone seems to have a different workflow with different symptoms. : )

[marcus-oscarsson]

micromamba worked fine for me conda still takes forever but thats perhaps something local. It seems to work fine for yourself, @walesch-yan and @woutdenolf, so I think we can conclude that the python version and conda environment issue is solved. There still seems to be some intermittent problem as you mention that are out of our control, however that we can't really do anything about at the moment. Your proposed fixed using conda looks perfect to me, thanks !

[marcus-oscarsson]

The python issue, is indeed something separate. I guess the type hint issue is something recent that snuck in in the Harvester code and should be easy to fix. Perhaps @jbflo could have a look ?

[fabcor-maxiv]

OK, I close this then.

Can always be reopened, if/when we have more concrete failing scenario of course.