gpelouze
commented
3 weeks ago Hi @stanleesocca, can you give an concrete example of the issue you are trying to solve by manipulating sys.path? Perhaps you are trying to add modules that cannot be installed with pip?
The conda environments are indeed found at different locations when you run the notebook (/opt/conda/envs/<conda environment name>/lib/python3.11/) and in the containerized cell (/venv/lib/python3.11). This allows us have multiple environments for notebooks, but keep a single one in containerized cells (which reduces the image size).
However, this implementation detail should not matter.
- If you use packages from the environment, or even install new ones using
piporconda install, they will automatically go to the right location. In this case, you do not need to manipulate Python'ssys.pathor the system's$PATH. - If you manually install Python modules to a custom location (eg.
/my/python/modules/), they will be completely independent from the conda environment. In this case, you have to manage the path yourself, as you found out (eg. withsys.path.append('/my/python/modules/')). Keep in mind: this needs to be added to each cell, because they run as independent Python scripts in the containerized context. I suspect that it is why the appended path doesn’t show up in your Argo example.
The following should work, with cell 2 printing hello, world!:
# cell 1
import os
custom_path = '/tmp/data/my/python/modules/'
os.makedirs(custom_path, exist_ok=True)
with open(os.path.join(custom_path, 'something_custom.py'), 'w') as f:
f.write('print("hello, world!")\n')# cell 2
import sys
sys.path.append(custom_path)
import something_customHere is a general note for people stumbling upon this issue: the second approach should be avoided if possible. Try to rely on packages available on PyPI or conda-forge (these are detected during containerization and automatically added to the environment), or run pip install git+... from within the cell.
stanleesocca
One issue with using a custom conda environment is that it doesn't work well in containerization mode. For example, with a conda environment called lter-life-wadden and the following listing:
I see the following system path where all libraries and binaries are located:
Now that works fine for interactive usage. But as soon as the cells are containerized even after specifying the exact base image lter-life-wadden to use during the containerization process, the result list of library/binary and system path is the following in argo logs:
This look like there are two different system search paths for environment in place here. One for the interactive work and another for the dockerization work. This is not an issue if I want to only use library already installed in the
/ven/lib/but an issue if I need packages not installed there. Ideally, we would want both system paths to be the same (and more importantly, the user's defined sys.path). This will make it possible for user specific environment in the workflow building pipeline.