run without internet access

[OH-AU]

Describe the bug This may be either a bug or a feature request. In this case it is not uncommon for clusters to have network access restricted on actual compute nodes but allow download access on specific restricted nodes. After creating the sif image and attempting to run on a firewalled machine, the following error occurs:

./pgap.py --no-internet -r -o mg37_results ./test_genomes/MG37/input.yaml
--no-internet flag enabled, not checking remote versions.
<urlopen error [Errno 111] Connection refused>
Failed to update ./pgap.py, ignoring
Something has gone wrong, please manually download: https://github.com/ncbi/pgap/raw/prod/scripts/pgap.py

When running a node that does have internet access it works as expected:

./pgap.py --no-internet -r -o mg37_results ./test_genomes/MG37/input.yaml
--no-internet flag enabled, not checking remote versions.
Output will be placed in: /dir/pgap/mg37_results.1
INFO:    Using cached SIF image

Not after a successful run, there is ~29GB of data in a folder input-2022-08-11.build6275, however it still doesn't run when you attempt to rerun. If the above is expected, could you consider an offline mode to running the pipeline after the data has been downloaded once. Better yet, split the data download from the rest of the pipeline and allow that to be run separately?

To Reproduce See above

Expected behavior Once the sif image has been created, no more internet access is required. Using the --no-internet option to me implies that it is possible to run without a network connection once the image is created elsewhere.

Software versions (please complete the following information):

• OS: SuSE 15SP3
• pgap.py --version: PGAP version 2022-08-11.build6275 is up to date.
• Docker (or other container runner) version: apptainer version 1.0.1

Log Files error happens before debug output starts

Additional context

[azat-badretdin]

Apologies for not replying earlier. We have no excuse.

Thank you for your report! It is not clear from your description if you managed to install it (obviously you need internet access for that).

[EricDeveaud]

same problem here. compute node -> no internet access installed tools on Read only shared drive no docker available only singularity // apptainer

we used to install tools on our cluster from a specific node that have internet access and RW permission on our tool tree. tools are installed and provided via module environnement

is there a was to have pgap use aspecific .sif image. let's saywe have it lying on PAG_PREFIX/share and we wouldlike to have pgap.py use this specific on when called ?

regards

Eric

[EricDeveaud]

seems to my understanding that when pgap.py is run trhough docker and//or singularity the generated command will always use docker container rties to pull it from cach and in case of singularity convert it to sif one.

it would be nice to have singularity native behavoiour. eg when called with -D singularity that already built .sif container cold be given as argument to be used.

regards

Eric

[azat-badretdin]

seems to my understanding that when pgap.py is run trhough docker and//or singularity the generated command will always use docker container rties to pull it from cach and in case of singularity convert it to sif one.

Eric, this is correct. You can also trying specifying a path to existing .sif using --container-path

[EricDeveaud]

seems there is a problem. see:

rpm_maker:PGAP/2021-05-19 > unshare -n pgap.py  --no-internet --no-self-update --container-name /opt/gensoft/exe/PGAP/2021-05-19/libexec/pgap_2021-05-19.build5429.sif -D /opt/gensoft/exe/singularity/3.7.3/bin/singularity  -v -r -o MG37_results MG37/input.yaml
--no-internet flag enabled, not checking remote versions.
Output will be placed in: /opt/gensoft/tests/PGAP/2021-05-19/MG37_results.7
FATAL:   Unable to handle docker://ncbi/pgap:2021-05-19.build5429 uri: failed to get checksum for docker://ncbi/pgap:2021-05-19.build5429: error pinging docker registry registry-1.docker.io: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 192.168.149.102:53: dial udp 192.168.149.102:53: connect: network is unreachable

[azat-badretdin]

Did you point PGAP_INPUT_DIR envar to the directory where you installed it from the machine that had internet access?

[EricDeveaud]

installing version 2022-09-30 right now --container_path was not available on the PGAP/2021-05-19 I was working with previously

stay tunned

[EricDeveaud]

@azat-badretdin thanks works well.

so in order to run pgap.py without internet using singularity // apptainer 1) install from an internet connected computer as container runner 2) run

PGAP_INPUT_DIR=/install/directory pgap.py --no-self-update --update -D `command -v singularity`

this will génerate the singularity//apptainer .sif container

then just run on network less nodes, eg:

export PGAP_INPUT_DIR=/install/directory
pgap.py  --no-internet --no-self-update --container-path $PGAP_INPUT_DIR/pgap_2022-08-11.build6275.sif -D `command -v singularity`  -v -r -o MG37_results MG37/input.yaml

it is currently running smoothly with 2022-08-11 I donwgraded to this one instead of 2022-09-30 as this one brings back data from the 2022-08-11 ??

anyway it is now running and I willprovide it to my users.

thanks again for the valuable information and help

Eric

[azat-badretdin]

That's correct to the dot, Eric! Including your wise decision to ditch 2022-09-30 and return back to August release. 2022-09-30 was a premature release because I accidentally published it instead of saving a draft. I deleted that release already.

Have in mind that are we getting ready for a new release in the next couple of days.

Cheers!

[EricDeveaud]

for the next release maybee you can provide a way, during --update to download the --taxcheck required archive. ie: https://s3.amazonaws.com/pgap/input-2022-08-11.build6275.ani.tgz

I had to download it, expand and touch .ani_complete file in order to be abble to run with --taxcheck

regards Eric

[azat-badretdin]

That you can do right now, Eric: ./pgap.py --update --taxcheck should do the trick

[EricDeveaud]

ooops should have read the docs//code more in depth

best regards

Eric

[hgingras]

Hello, I also need to run PGAP on compute node without internet access.

I tried to follow direction from this issue to test everything with test_genomes provided here.

Here is what I did:

wget https://github.com/ncbi/pgap/raw/prod/scripts/pgap.py chmod +x pgap.py ./pgap.py --update -D /opt/software/singularity-3.8/bin/singularity

Got the .sif file installed.

Here is the .sh file I run on the compute node:

---

!/bin/bash

#

SBATCH --time=02:00:00

SBATCH --ntasks=16

SBATCH --mem-per-cpu=64GB

module load python/3.10 module load singularity/3.8

export PGAP_INPUT_DIR=/home/helene/projects/def-helene/helene/Tickets/0188208/PGAP

./pgap.py --no-internet --no-self-update --container-path $PGAP_INPUT_DIR/pgap_2022-12-13.build6494.sif -D /opt/software/singularity-3.8/bin/singularity -n -o mg37_results $HOME/.pgap/test_genomes/MG37/input.yaml

---

Here is my output file:

---

Traceback (most recent call last): File "/lustre03/project/6078354/helene/Tickets/0188208/PGAP/./pgap.py", line 917, in main params = Setup(args) File "/lustre03/project/6078354/helene/Tickets/0188208/PGAP/./pgap.py", line 546, in init if (self.local_version != self.use_version) or not self.check_install_data(): File "/lustre03/project/6078354/helene/Tickets/0188208/PGAP/./pgap.py", line 740, in check_install_data if self.use_version > "2019-11-25.build4172": TypeError: '>' not supported between instances of 'NoneType' and 'str' The latest version of PGAP is None, you have nothing installed locally. Not trying to update self, because the --no-self-update flag is enabled.

---

When I do on login nodes (with internet access I get):

./pgap.py --update PGAP version 2022-12-13.build6494 is up to date.

Thanks for your help,

Hélène

[azat-badretdin]

Thanks, Hélène, for your report!

I opened internal ticket for this. We will have a look at this.

[hgingras]

Hello there. For a reason I cannot explain, I didn't have at first the VERSION file and that was the source of the problem. On another attempt, all the files were uploaded and could run the pipeline. Thanks.

[azat-badretdin]

Thanks for update, Hélène!