Native system python being called over conda env python

[olivermccallion]

Hi,

I've done a fresh panpipes install on ARC following the option 1 instructions here. The install completes without error. This gives me a conda environment named pp_env at the path /data/ndos-trig/mdiv1297/condaenvs/pp_env/

I'm trying to run the integration step using a pre-existing .h5mu with the following commands:

conda deactivate
conda activate /data/ndos-trig/mdiv1297/condaenvs/pp_env/
panpipes integration make full --local

This gives the following import error:

    Traceback (most recent call last):
      File "/data/ndos-trig/mdiv1297/condaenvs/pp_env/lib/python3.10/site-packages/panpipes/python_scripts/batch_correct_scvi.py", line 9, in <module>
        import scvi
    ModuleNotFoundError: No module named 'scvi'

If I start python within the environment and import scvi it imports without issue.

My sys.path within pp_env is:

 ['', '/data/ndos-trig/mdiv1297/condaenvs/pp_env/lib/python310.zip', '/data/ndos-trig/mdiv1297/condaenvs/pp_env/lib/python3.10', '/data/ndos-trig/mdiv1297/condaenvs/pp_env/lib/python3.10/lib-dynload', '/home/mdiv1297/.local/lib/python3.10/site-packages', '/data/ndos-trig/mdiv1297/condaenvs/pp_env/lib/python3.10/site-packages']

If I print sys.path from batch_correct_scvi.py I get:

 ['/data/ndos-trig/mdiv1297/condaenvs/pp_env/lib/python3.10/site-packages/panpipes/python_scripts', '/apps/system/easybuild/software/Anaconda3/2022.05/lib/python39.zip', '/apps/system/easybuild/software/Anaconda3/2022.05/lib/python3.9', '/apps/system/easybuild/software/Anaconda3/2022.05/lib/python3.9/lib-dynload', '/home/mdiv1297/.local/lib/python3.9/site-packages', '/apps/system/easybuild/software/Anaconda3/2022.05/lib/python3.9/site-packages']

It seems like panpipes is not using the python (3.10) installed in pp_env but instead defaulting to the system python.

Any help on how to fix this would be really appreciated! Thanks :)

[bio-la]

hi @olivermccallion, thanks for using panpipes. Do you explicitly set the conda environment in the configuration file that is passed to the integration workflow?

#--------------------------
# Compute resources options
#--------------------------

condaenv: /data/ndos-trig/mdiv1297/condaenvs/pp_env

[olivermccallion]

Hi @bio-la

Yes, I've explicitly set the conda environment in the yml, although I've also tried without doing so and get the same behaviour.

[bio-la]

You're submitting a job on a cluster? I'm not familiar with ARC, but setting the same conda env in the .cgat.yml configuration file should prevent this behavior. If I understand correctly you're not running the integration workflow on the submission node (--local would do) where your conda env is active with panpipes installation, and in some systems the node on which the job lands doesn't inherit the configuration of the submission node, hence the discrepancy. Have you checked the documentation for running on HPC?

Can you check your .cgat.yml configuration file?

[olivermccallion]

It's a HPC cluster, but it doesn't have DRMAA configured so I'm running panpipes on an interactive node. So jobs aren't being managed by slurm, they're being run on the node running the interactive session. Sorry, I should have made that clearer. But I've updated my .cgat.yml and am still getting the same behaviour.

[bio-la]

Ok thanks for clarifying. i've seen some instances of this, and exporting the PYTHONPATH in your .bashrc should force the shell to use the correct one. I'm wondering if in your case you'd rather use a python venv instead of the conda (although exporting the PYTHONPATH with your conda python path should have the same effect) We wrote specific instructions for one of the clusters we systematically run panpipes, can you check if it's helpful for you? https://github.com/DendrouLab/panpipes/blob/main/docs/installation_rescomp.md

thanks for your patience with this, I'm happy to help troubleshoot!

[olivermccallion]

Hi,

Bit of a frustrating resolution but just a quick message to say that a fresh install from github fixed this behaviour for me. Thanks for the support and getting back to me so quickly!