search

opain / GenoPred

Genotype-based Prediction (GenoPred)
https://opain.github.io/GenoPred/
GNU General Public License v3.0
68 stars 23 forks source link

Error while installing dependencies #98

Closed unchowdhury closed 2 months ago

unchowdhury commented 5 months ago

I was trying to install the genopred and it produces the following error while installing dependencies:

Command: (genopred) [uquchowd@bunya1 pipeline]$ snakemake --restart-times 3 -j 1 --use-conda --conda-frontend conda get_dependencies /sw/auto/rocky8c/epyc3/software/Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/paramiko/transport.py:236: CryptographyDeprecationWarning: Blowfish has been deprecated "class": algorithms.Blowfish, Building DAG of jobs... Your conda installation is not configured to use strict channel priorities. This is however crucial for having robust and correct environments (for details, see https://conda-forge.org/docs/user/tipsandtricks.html). Please consider to configure strict priorities by executing 'conda config --set channel_priority strict'. Creating conda environment envs/pgscatalog_utils.yaml... Downloading and installing remote packages. CreateCondaEnvironmentException: Could not create conda environment from /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/rules/../envs/pgscatalog_utils.yaml: Command: conda env create --quiet --file "/scratch/projectmnt/S0007/uquchowd/GenoPred/pipeline/.snakemake/conda/9f4c78026677de4a934c7c783c6d8dc0.yaml" --prefix "/scratch/projectmnt/S0007/uquchowd/GenoPred/pipeline/.snakemake/conda/9f4c78026677de4a934c7c783c6d8dc0" Output: Channels:

CondaEnvException: Pip failed

N.B. Previously I was successful to install GenoPred in this platform but now it's generating this error.

opain commented 5 months ago

Hello,

This is odd. Lets try and understand.

The error seems to suggest that you do not have required permission to modify files within the directory that environment is being installed. This doesn’t make sense though as the environment should be created within a subdirectory of the GenoPred repo. Particularly confusing that you had installed before, so something must have changed (permissions, working directory, TMPDIR), or the previous installation is somehow conflicting with the new setup.

To try and get more information, could you try and build the conda environment directly (without snakemake). Hopefully this will give a clearer error. Use the verbose. 

conda env create -f /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/pgscatalog_utils.yaml --prefix /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/.snakemake/conda/9f4c78026677de4a934c7c783c6d8dc0_ --verbose

Thanks!

unchowdhury commented 5 months ago

Thanks for your response. I tried to build the conda environment directly with verbose. This produced the following error: (genopred) [uquchowd@bunya1 pipeline]$ conda env create -f /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/pgscatalog_utils.yaml --prefix /scratch/projectmnt/S0007/uquchowd/GenoPred/pipeline/.snakemake/conda/9f4c78026677de4a934c7c783c6d8dc0 --verbose Collecting package metadata (repodata.json): ...working... done Solving environment: ...working... done

==> WARNING: A newer version of conda exists. <== current version: 23.9.0 latest version: 24.5.0

Please update conda by running

$ conda update -n base -c defaults conda

Or to minimize the number of packages updated during conda update use

 conda install conda=24.5.0

Preparing transaction: ...working... done Verifying transaction: ...working... done Executing transaction: ...working... done Installing pip dependencies: ...working... Ran pip subprocess with arguments: ['/scratch/projectmnt/S0007/uquchowd/GenoPred/pipeline/.snakemake/conda/9f4c78026677de4a934c7c783c6d8dc0/bin/python', '-m', 'pip', 'install', '-U', '-r', '/scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/condaenv.uxwjmdiz.requirements.txt', '--exists-action=b'] Pip subprocess output: Collecting poetry (from -r /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/condaenv.uxwjmdiz.requirements.txt (line 1)) Using cached poetry-1.8.3-py3-none-any.whl.metadata (6.8 kB) Collecting build<2.0.0,>=1.0.3 (from poetry->-r /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/condaenv.uxwjmdiz.requirements.txt (line 1)) Using cached build-1.2.1-py3-none-any.whl.metadata (4.3 kB) Collecting cachecontrol<0.15.0,>=0.14.0 (from cachecontrol[filecache]<0.15.0,>=0.14.0->poetry->-r /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/condaenv.uxwjmdiz.requirements.txt (line 1)) Using cached cachecontrol-0.14.0-py3-none-any.whl.metadata (3.1 kB) Collecting cleo<3.0.0,>=2.1.0 (from poetry->-r /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/condaenv.uxwjmdiz.requirements.txt (line 1)) Using cached cleo-2.1.0-py3-none-any.whl.metadata (12 kB) Collecting crashtest<0.5.0,>=0.4.1 (from poetry->-r /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/condaenv.uxwjmdiz.requirements.txt (line 1)) Using cached crashtest-0.4.1-py3-none-any.whl.metadata (1.1 kB) Collecting dulwich<0.22.0,>=0.21.2 (from poetry->-r /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/condaenv.uxwjmdiz.requirements.txt (line 1)) Using cached dulwich-0.21.7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.3 kB) Collecting fastjsonschema<3.0.0,>=2.18.0 (from poetry->-r /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/condaenv.uxwjmdiz.requirements.txt (line 1)) Using cached fastjsonschema-2.19.1-py3-none-any.whl.metadata (2.1 kB) Collecting installer<0.8.0,>=0.7.0 (from poetry->-r /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/condaenv.uxwjmdiz.requirements.txt (line 1)) Using cached installer-0.7.0-py3-none-any.whl.metadata (936 bytes) Collecting keyring<25.0.0,>=24.0.0 (from poetry->-r /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/condaenv.uxwjmdiz.requirements.txt (line 1)) Using cached keyring-24.3.1-py3-none-any.whl.metadata (20 kB) Collecting packaging>=23.1 (from poetry->-r /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/condaenv.uxwjmdiz.requirements.txt (line 1)) Using cached packaging-24.0-py3-none-any.whl.metadata (3.2 kB) Collecting pexpect<5.0.0,>=4.7.0 (from poetry->-r /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/condaenv.uxwjmdiz.requirements.txt (line 1)) Using cached pexpect-4.9.0-py2.py3-none-any.whl.metadata (2.5 kB) Collecting pkginfo<2.0,>=1.10 (from poetry->-r /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/condaenv.uxwjmdiz.requirements.txt (line 1)) Using cached pkginfo-1.10.0-py3-none-any.whl.metadata (11 kB) Collecting platformdirs<5,>=3.0.0 (from poetry->-r /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/condaenv.uxwjmdiz.requirements.txt (line 1)) Using cached platformdirs-4.2.2-py3-none-any.whl.metadata (11 kB) Collecting poetry-core==1.9.0 (from poetry->-r /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/condaenv.uxwjmdiz.requirements.txt (line 1)) Using cached poetry_core-1.9.0-py3-none-any.whl.metadata (3.5 kB) Collecting poetry-plugin-export<2.0.0,>=1.6.0 (from poetry->-r /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/condaenv.uxwjmdiz.requirements.txt (line 1)) Using cached poetry_plugin_export-1.8.0-py3-none-any.whl.metadata (2.8 kB) Collecting pyproject-hooks<2.0.0,>=1.0.0 (from poetry->-r /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/condaenv.uxwjmdiz.requirements.txt (line 1)) Using cached pyproject_hooks-1.1.0-py3-none-any.whl.metadata (1.3 kB) Collecting requests<3.0,>=2.26 (from poetry->-r /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/condaenv.uxwjmdiz.requirements.txt (line 1)) Downloading requests-2.32.3-py3-none-any.whl.metadata (4.6 kB) Collecting requests-toolbelt<2.0.0,>=1.0.0 (from poetry->-r /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/condaenv.uxwjmdiz.requirements.txt (line 1)) Using cached requests_toolbelt-1.0.0-py2.py3-none-any.whl.metadata (14 kB) Collecting shellingham<2.0,>=1.5 (from poetry->-r /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/condaenv.uxwjmdiz.requirements.txt (line 1)) Using cached shellingham-1.5.4-py2.py3-none-any.whl.metadata (3.5 kB) Collecting tomli<3.0.0,>=2.0.1 (from poetry->-r /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/condaenv.uxwjmdiz.requirements.txt (line 1)) Using cached tomli-2.0.1-py3-none-any.whl.metadata (8.9 kB) Collecting tomlkit<1.0.0,>=0.11.4 (from poetry->-r /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/condaenv.uxwjmdiz.requirements.txt (line 1)) Using cached tomlkit-0.12.5-py3-none-any.whl.metadata (2.7 kB) Collecting trove-classifiers>=2022.5.19 (from poetry->-r /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/condaenv.uxwjmdiz.requirements.txt (line 1)) Using cached trove_classifiers-2024.5.22-py3-none-any.whl.metadata (2.2 kB) Collecting virtualenv<21.0.0,>=20.23.0 (from poetry->-r /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/condaenv.uxwjmdiz.requirements.txt (line 1)) Using cached virtualenv-20.26.2-py3-none-any.whl.metadata (4.4 kB) Collecting msgpack<2.0.0,>=0.5.2 (from cachecontrol<0.15.0,>=0.14.0->cachecontrol[filecache]<0.15.0,>=0.14.0->poetry->-r /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/condaenv.uxwjmdiz.requirements.txt (line 1)) Using cached msgpack-1.0.8-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (9.1 kB) Collecting filelock>=3.8.0 (from cachecontrol[filecache]<0.15.0,>=0.14.0->poetry->-r /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/condaenv.uxwjmdiz.requirements.txt (line 1)) Using cached filelock-3.14.0-py3-none-any.whl.metadata (2.8 kB) Collecting rapidfuzz<4.0.0,>=3.0.0 (from cleo<3.0.0,>=2.1.0->poetry->-r /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/condaenv.uxwjmdiz.requirements.txt (line 1)) Downloading rapidfuzz-3.9.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (12 kB) Collecting urllib3>=1.25 (from dulwich<0.22.0,>=0.21.2->poetry->-r /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/condaenv.uxwjmdiz.requirements.txt (line 1)) Using cached urllib3-2.2.1-py3-none-any.whl.metadata (6.4 kB) Collecting jaraco.classes (from keyring<25.0.0,>=24.0.0->poetry->-r /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/condaenv.uxwjmdiz.requirements.txt (line 1)) Using cached jaraco.classes-3.4.0-py3-none-any.whl.metadata (2.6 kB) Collecting importlib-metadata>=4.11.4 (from keyring<25.0.0,>=24.0.0->poetry->-r /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/condaenv.uxwjmdiz.requirements.txt (line 1)) Using cached importlib_metadata-7.1.0-py3-none-any.whl.metadata (4.7 kB) Collecting SecretStorage>=3.2 (from keyring<25.0.0,>=24.0.0->poetry->-r /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/condaenv.uxwjmdiz.requirements.txt (line 1)) Using cached SecretStorage-3.3.3-py3-none-any.whl.metadata (4.0 kB) Collecting jeepney>=0.4.2 (from keyring<25.0.0,>=24.0.0->poetry->-r /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/condaenv.uxwjmdiz.requirements.txt (line 1)) Using cached jeepney-0.8.0-py3-none-any.whl.metadata (1.3 kB) Collecting ptyprocess>=0.5 (from pexpect<5.0.0,>=4.7.0->poetry->-r /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/condaenv.uxwjmdiz.requirements.txt (line 1)) Using cached ptyprocess-0.7.0-py2.py3-none-any.whl.metadata (1.3 kB) Collecting charset-normalizer<4,>=2 (from requests<3.0,>=2.26->poetry->-r /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/condaenv.uxwjmdiz.requirements.txt (line 1)) Using cached charset_normalizer-3.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (33 kB) Collecting idna<4,>=2.5 (from requests<3.0,>=2.26->poetry->-r /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/condaenv.uxwjmdiz.requirements.txt (line 1)) Using cached idna-3.7-py3-none-any.whl.metadata (9.9 kB) Collecting certifi>=2017.4.17 (from requests<3.0,>=2.26->poetry->-r /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/condaenv.uxwjmdiz.requirements.txt (line 1)) Using cached certifi-2024.2.2-py3-none-any.whl.metadata (2.2 kB) Collecting distlib<1,>=0.3.7 (from virtualenv<21.0.0,>=20.23.0->poetry->-r /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/condaenv.uxwjmdiz.requirements.txt (line 1)) Using cached distlib-0.3.8-py2.py3-none-any.whl.metadata (5.1 kB) Collecting zipp>=0.5 (from importlib-metadata>=4.11.4->keyring<25.0.0,>=24.0.0->poetry->-r /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/condaenv.uxwjmdiz.requirements.txt (line 1)) Using cached zipp-3.19.0-py3-none-any.whl.metadata (3.5 kB) Collecting cryptography>=2.0 (from SecretStorage>=3.2->keyring<25.0.0,>=24.0.0->poetry->-r /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/condaenv.uxwjmdiz.requirements.txt (line 1)) Using cached cryptography-42.0.7-cp39-abi3-manylinux_2_28_x86_64.whl.metadata (5.3 kB) Collecting more-itertools (from jaraco.classes->keyring<25.0.0,>=24.0.0->poetry->-r /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/condaenv.uxwjmdiz.requirements.txt (line 1)) Using cached more_itertools-10.2.0-py3-none-any.whl.metadata (34 kB) Collecting cffi>=1.12 (from cryptography>=2.0->SecretStorage>=3.2->keyring<25.0.0,>=24.0.0->poetry->-r /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/condaenv.uxwjmdiz.requirements.txt (line 1)) Using cached cffi-1.16.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.5 kB) Collecting pycparser (from cffi>=1.12->cryptography>=2.0->SecretStorage>=3.2->keyring<25.0.0,>=24.0.0->poetry->-r /scratch/project_mnt/S0007/uquchowd/GenoPred/pipeline/envs/condaenv.uxwjmdiz.requirements.txt (line 1)) Using cached pycparser-2.22-py3-none-any.whl.metadata (943 bytes) Using cached poetry-1.8.3-py3-none-any.whl (249 kB) Using cached poetry_core-1.9.0-py3-none-any.whl (309 kB) Using cached build-1.2.1-py3-none-any.whl (21 kB) Using cached cachecontrol-0.14.0-py3-none-any.whl (22 kB) Using cached cleo-2.1.0-py3-none-any.whl (78 kB) Using cached crashtest-0.4.1-py3-none-any.whl (7.6 kB) Using cached dulwich-0.21.7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (514 kB) Using cached fastjsonschema-2.19.1-py3-none-any.whl (23 kB) Using cached installer-0.7.0-py3-none-any.whl (453 kB) Using cached keyring-24.3.1-py3-none-any.whl (38 kB) Using cached packaging-24.0-py3-none-any.whl (53 kB) Using cached pexpect-4.9.0-py2.py3-none-any.whl (63 kB) Using cached pkginfo-1.10.0-py3-none-any.whl (30 kB) Using cached platformdirs-4.2.2-py3-none-any.whl (18 kB) Using cached poetry_plugin_export-1.8.0-py3-none-any.whl (10 kB) Using cached pyproject_hooks-1.1.0-py3-none-any.whl (9.2 kB) Downloading requests-2.32.3-py3-none-any.whl (64 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 64.9/64.9 kB 3.0 MB/s eta 0:00:00 Using cached requests_toolbelt-1.0.0-py2.py3-none-any.whl (54 kB) Using cached shellingham-1.5.4-py2.py3-none-any.whl (9.8 kB) Using cached tomli-2.0.1-py3-none-any.whl (12 kB) Using cached tomlkit-0.12.5-py3-none-any.whl (37 kB) Using cached trove_classifiers-2024.5.22-py3-none-any.whl (13 kB) Using cached virtualenv-20.26.2-py3-none-any.whl (3.9 MB) Using cached certifi-2024.2.2-py3-none-any.whl (163 kB) Using cached charset_normalizer-3.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (142 kB) Using cached distlib-0.3.8-py2.py3-none-any.whl (468 kB) Using cached filelock-3.14.0-py3-none-any.whl (12 kB) Using cached idna-3.7-py3-none-any.whl (66 kB) Using cached importlib_metadata-7.1.0-py3-none-any.whl (24 kB) Using cached jeepney-0.8.0-py3-none-any.whl (48 kB) Using cached msgpack-1.0.8-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (385 kB) Using cached ptyprocess-0.7.0-py2.py3-none-any.whl (13 kB) Downloading rapidfuzz-3.9.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.4 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.4/3.4 MB 27.9 MB/s eta 0:00:00 Using cached SecretStorage-3.3.3-py3-none-any.whl (15 kB) Using cached urllib3-2.2.1-py3-none-any.whl (121 kB) Using cached jaraco.classes-3.4.0-py3-none-any.whl (6.8 kB) Using cached cryptography-42.0.7-cp39-abi3-manylinux_2_28_x86_64.whl (3.8 MB) Using cached zipp-3.19.0-py3-none-any.whl (8.3 kB) Using cached more_itertools-10.2.0-py3-none-any.whl (57 kB) Using cached cffi-1.16.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (443 kB) Using cached pycparser-2.22-py3-none-any.whl (117 kB) Installing collected packages: trove-classifiers, ptyprocess, fastjsonschema, distlib, zipp, urllib3, tomlkit, tomli, shellingham, rapidfuzz, pyproject-hooks, pycparser, poetry-core, platformdirs, pkginfo, pexpect, packaging, msgpack, more-itertools, jeepney, installer, idna, filelock, crashtest, charset-normalizer, certifi, virtualenv, requests, jaraco.classes, importlib-metadata, dulwich, cleo, cffi, build, requests-toolbelt, cryptography, cachecontrol, SecretStorage, keyring, poetry-plugin-export, poetry Attempting uninstall: fastjsonschema Found existing installation: fastjsonschema 2.16.1 Uninstalling fastjsonschema-2.16.1:

Pip subprocess error: ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: 'METADATA' Consider using the --user option or check the permissions.

failed

CondaEnvException: Pip failed

unchowdhury commented 5 months ago

I tried to start things from the begining once again by deleting GenoPred folder, removing genopred environment, deleting leftover environment files and clearing cache from conda. I downloaded GenoPred in different directory and now it generates a new error: (genopred) [uquchowd@bun099 pipeline]$ snakemake --restart-times 3 -j 1 --use-conda --conda-frontend conda get_dependencies
ImportError in file /scratch/project_mnt/S0007/uquchowd/AGHA/PRS/GenoPred/pipeline/rules/dependencies.smk, line 5:
Unable to import required dependencies:
numpy:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at: 

https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: No module named 'numpy.core._multiarray_umath'

File "/scratch/project_mnt/S0007/uquchowd/AGHA/PRS/GenoPred/pipeline/Snakefile", line 18, in
File "/scratch/project_mnt/S0007/uquchowd/AGHA/PRS/GenoPred/pipeline/rules/dependencies.smk", line 5, in
File "/sw/auto/rocky8c/epyc4/software/SciPy-bundle/2022.05-foss-2022a/lib/python3.10/site-packages/pandas/init.py", line 16, in

N.B. I am trying to install on HPC using an interactive job.

opain commented 5 months ago

Hi,

Sorry the installation is not working well for you.

The latest error is due to having multiple versions of python in your PATH, as it says it is using "/home/uquchowd/.conda/envs/genopred/bin/python3.8" at one point, but then "/sw/auto/rocky8c/epyc4/software/SciPy-bundle/2022.05-foss-2022a/lib/python3.10" at another. Start with a fresh interactive session, with no modules loaded, to avoid conflicts in your PATH. Or you can modify your PATH variable to look at the conda environment. I mention this issue on the website here.

If the conda installation continues to fail, you may want to consider using a containerised version of GenoPred. These containers already have GenoPred installed. If you are working on an HPC, you will most likely use singularity to run the container. I mention containers here, and provide some example of me running the pipeline within a container. A downside of using the container is that it is harder distribute analyses across your HPC using the --profile parameter - It can still easily run analyses using multiple cores though (using the -j parameter), so just run the pipeline in an instance with access to as many cores as you can/need. Depending on your needs I could try and document how to using the --profile parameter inside a container.

Let me know if you would like further advice.

opain commented 3 months ago

I had another user contact me regarding similar conda related issues. This person was working on the KCL CREATE HPC. The solution in their case was using the preinstalled miniforge module, instead of installing miniconda.

I would suggest to users working on HPCs to check that conda/mamba is not already installed as a module, as it may be less likely to run into issues.

Furthermore, it might be worth moving away from miniconda, and instead use miniforge, as I see it is no longer recommended to install mamba using conda, so this might also be leading to errors in some cases.