AI MARRVEL HPC Issues - Internet Access Not Allowed

[yr542]

I am on an HPC and therefore, I cannot have this accessing the internet. Is there a way for me to be able to create a conda environment or work around considering you have multiple times that your files try to access the internet (and multiple different versions of python)?

[sasidharPasupuleti]

@yr542 is it possible to provide example of the script(slurm or any job submitting scheduler) that you may be using to run the pipeline on HPC.

Also could you you try setting env variable ENV NXF_OFFLINE=true in your job and show what errors you are facing.

[yr542]

Dr. @sasidharPasupuleti I could try this but I would not be able to resolve the fact that it's asking for 2 completely different versions of python in miniconda? Is there any way to get around this? Preferably is there a singularity with the modules it requires inside?

[allyssonallan]

@sasidharPasupuleti thanks for your help!

I previously had this behavior: 'wget --quiet http://hgdownload.soe.ucsc.edu/goldenPath/${params.ref_ver}/bigZips/${params.ref_ver}.fa.gz '

The workaround that I made: 1 - downloaded the file inside the db-S3 directory 2 - pointed the workflow to the file 3 - it worked in HPC without internet access to the worker nodes

Best of the worlds:

• [ ] S3 reference-based (VEP chunks are insane in number)
• [ ] Docker reference-based
• [ ] Singularity pull option

Are PRs welcome?

[yr542]

@allyssonallan Could I ask what you mean by db-S3? I am just an HPC user (not an admin). This plus what @sasidharPasupuleti and a way to help with the disparate python versions might help me to use this software.

[allyssonallan]

@allyssonallan Could I ask what you mean by db-S3? I am just an HPC user (not an admin). This plus what @sasidharPasupuleti and a way to help with the disparate python versions might help me to use this software.

Sorry @yr543; db-S3 below:

https://github.com/LiuzLab/AI_MARRVEL/blob/e193d8ae1ece143d7ff877616b5251e4b0b30af4/README.md?plain=1#L40C1-L44C1

[yr542]

@allyssonallan So just a reference. Thank you for the response :) to the question I posed to you:

@allyssonallan Could I ask what you mean by db-S3? I am just an HPC user (not an admin).

[yr542]

Could I ask if there is a singularity of AI-MARRVEL so that I could perhaps use that as a work around? Or a created docker equivalent that I could turn into a singularity? @sasidharPasupuleti