Installation with conda and pip

[antolinos]

Currently, documentation recommends to use conda in order to setup python (https://ispyb.github.io/py-ispyb/run/#setup-python) and then pip to install the dependencies (https://ispyb.github.io/py-ispyb/run/#install-requirements).

Even if it is technically feasible It seems to me that there is a sort of consensus in the python community about not to mix conda and pip. If the latest is true and everybody agrees then I'd propose to change that point in the documentation.

It can be decided to support pip, conda or both but it should be clearly explained how to achieve the installation without mixing them.

[rhfogh]

This link suggests it may be growing more acceptable https://stackoverflow.com/questions/55340279/its-ok-to-mix-conda-install-and-pip-install

Anyway, the dilemma is that conda is much nicer to work with - but sometimes you need packages that are only available on pip. The conda advice is just to build those packages into conda, but that does sound a little heavy. This may not be a problem in ISPyB, but for MXCuBE conda-only is definitely not feasible.

[stufisher]

The reason conda is recommended is that it decouples you from the system python install. If you're unlucky your linux flavour may not provide you with the needed python version, with conda you can install whatever version you like and run one or more in isolation.

The other useful thing is that the dependencies are isolated in the conda environment so don't pollute the system environment. We could do this with virtualenv instead but it doesnt solve the python versioning question as virtualenv is still tied to the system python afaik.

When i setup the fastapi side i kept pip for deps as that was what being used. We could switch over to conda dependencies if you prefer. I didn't do an extensive check of what available/missing in conda. Any outliers we could install with a second pip requirements file (which is what we do in bcu on the beamlines). This would add an additional install step but maybe make things cleaner.

[rhfogh]

@stufisher That makes sense. Anyway, I am sure that you and the others (who work much more with ISPyB code than I do) will come up with something sensible.

[antolinos]

The cleanest would be to use only conda but I agree with Rasmus that it might be too heavy if packages need to be created. pip alone does not solve the issue with python versions but there are some alternatives like pyenv + pip that makes the trick

[antolinos]

I've just gave a try to pyenv . The most "difficult" is the installation itself. The rest is straight-forward.

pyenv install 3.10
pyenv global 3.10
pip install -r requirements
. uvicorn.sh

pyenv allows to change the global python version as well as per-project that could be quite useful.

I'd like to have a clear, concise and easy as possible documentation about how to install and run py-ispyb (ideally following the best practices or recommendations) mostly oriented to newcomers. Obviously, everybody is free to use their favorite way.

Would pyenv work for you?

[rhfogh]

If possible I'd vote for using conda as much as possible with additional pip for the few packages that need it, as @stufisher proposed above. As I understand it, you can install pip packages inside the conda environment, so that you are not loading them into your system. My main reason is that mxcube uses conda (which means that so do I). For one thing it would be nicer to have fewer tools to use; for another would it not be problematical to call ISPyB from MXCuBE if ISPyB relies on pyenv and MXCuBE relies on conda?

Do we know how many packages would not be available in conda? Do we have an actual current problem with conda, or is it abstract best practice?

[KarlLevik]

The mariadb package which we are not using yet, but will inevitably need to use in the future (in place of mysql-connector-python) is not available for conda, AFAIK.

I reported it as a feature request in the MariaDB issue tracker, but it doesn't seem to be a priority for them yet: https://jira.mariadb.org/browse/CONPY-113

By the way, it would be helpful if that feature request got some votes, just saying :)

[stufisher]

I don't have a strong preference, as long as its easy for developers to get going and contained outside the system python environment. I used conda because its what we use in bcu but i'm open to whatever as long as its well supported and well used in the python community. I've never used pyenv so maybe worth asking some of our other colleagues what they think too?

[stufisher]

I read about this recently too https://code.visualstudio.com/docs/devcontainers/containers which is pretty interesting (to be able to cross debug its useful for everyone to have the same environment)

[antolinos]

I'd say that there are few packages missing for conda so far. At least, I did not manage to install it with conda. Has anyone tried yet?

As I understand it, you can install pip packages inside the conda environment, so that you are not loading them into your system.

I use conda on daily basis and I like it however, as I said, it seems not to be recommended to mix both. It does not mean that it does not work but you can eventually break things because conda is not aware of what it is installed with pip and vice-versa. Nice article explaining why and best practices checklist: https://www.anaconda.com/blog/using-pip-in-a-conda-environment

Good news is that they are addressing the problem that might be fixed from conda > 4.6.0 (currently in beta)

My main reason is that mxcube uses conda (which means that so do I). For one thing it would be nicer to have fewer tools to use;

IMO, MxCube use case is different to ISPyB. MxCube has to be installed on any beamline and it might be quite delicate in terms of dependencies and software environment. On the other hand ISPyB, in general, will be installed in a dedicated machine that is most likely under our full control in terms of software installed.

for another would it not be problematical to call ISPyB from MXCuBE if ISPyB relies on pyenv and MXCuBE relies on conda?

MxCube calls ISPyB via webservices (http), it does not matter in which environment it is running.

In the documentation, If there is no consensus and as a compromise we could say that py-ispyb requires python 3.10 and dependencies can be installed via pip install. It is strictly correct.

In a subsection, we could mention the virtual environments and how it could be installed with conda+pip and pyenv

What do you think?