search

databio / bulker

Manager for multi-container computing environments
https://bulker.io
BSD 2-Clause "Simplified" License
24 stars 2 forks source link

Loading/activating PEPATAC crate_Errors #53

Open Kmrouj opened 4 years ago

Kmrouj commented 4 years ago

Hi Nathan, I'm working on sherlock (Stanford cluster) I'm trying to load PEPATAC crate using the "bulker load databio/pepatac -b -r" command. During the load, I had this return 'INFO: Creating SIF file... FATAL: While making image from oci registry: while building SIF from layers: while creating squashfs: create command failed: signal: killed: mv: cannot stat ‘rpipe’: No such file or directory ------ Error building. Build script used: ------

!/bin/sh

if [ ! -f "/home/users/karim90/simages/databio/rpipe" ]; then singularity pull rpipe docker://databio/rpipe mv rpipe /home/users/karim90/simages/databio/rpipe fi

Container available at: /home/users/karim90/simages/databio/rpipe Container available at: /home/users/karim90/simages/quay.io/biocontainers/samtools:1.9--h91753b0_8 Container available at: /home/users/karim90/simages/quay.io/biocontainers/ucsc-wigtobigwig:357--h35c10e6_3 Populating host commands Loading manifest: 'databio/pepatac:default'. Activate with 'bulker activate databio/pepatac:default'. Commands available: bedGraphToBigWig, bedToBigBed, bedtools, bigWigCat, bowtie2, fastqc, java, macs2, pigz, preseq, samblaster, skewer, R, Rscript, samtools, wigToBigWig Host commands available: python3, Perl"

When I tried to activate bulker using the "bulker activate databio/pepatac" command on sherlock, I had this error

[karim90@sh02-ln03 login ~]$ bulker activate databio/pepatac Bulker config: /home/users/karim90/bulker_config.yaml Activating bulker crate: databio/pepatac Error for command "pull": unknown shorthand flag: 'n' in -n "Run 'singularity pull --help' for more detailed usage information. mv: cannot stat ‘alpine-coreutils’: No such file or directory FATAL: could not open image /home/users/karim90/simages/databio/alpine-coreutils: failed to retrieve path for /home/users/karim90/simages/databio/alpine-coreutils: lstat /home/users/karim90/simages/databio/alpine-coreutils: no such file or directory databio/pepatac|~$ exit"

Thank you for your help

nsheff commented 4 years ago

singularity is not correctly pulling that image... can you do this independet of bulker?

singularity pull rpipe docker://databio/rpipe
Kmrouj commented 4 years ago

Hi Nathan, I tried: "singularity pull rpipe docker://databio/rpipe" but it did not work INFO: Creating SIF file... "FATAL: While making image from oci registry: while building SIF from layers: while creating squashfs: create command failed: signal: killed:"

I managed to edit the "singularity_build.jinja2 " file to remove the -n flag:

!/bin/sh

if [ ! -f "{{ pkg.singularity_fullpath }}" ]; then singularity pull {{ pkg.singularity_image }} docker://{{ pkg.docker_image }} mv {{ pkg.singularity_image }} {{ pkg.singularity_fullpath }} fi However when I started again loading and activating I had the same type of error: [karim90@sh01-ln03 login ~]$ bulker activate databio/pepatac Bulker config: /home/users/karim90/bulker_config.yaml Activating bulker crate: databio/pepatac Error for command "pull": unknown shorthand flag: 'n' in -n

Run 'singularity pull --help' for more detailed usage information. mv: cannot stat ‘alpine-coreutils’: No such file or directory FATAL: could not open image /home/users/karim90/simages/databio/alpine-coreutils: failed to retrieve path for /home/users/karim90/simages/databio/alpine-coreutils: lstat /home/users/karim90/simages/databio/alpine-coreutils: no such file or directory databio/pepatac|~$

nsheff commented 4 years ago

If you can't do "singularity pull rpipe docker://databio/rpipe" in general, then there's some problem with singularity that is not related to bulker. might need to ask the sys admins or something, maybe they disabled creation of singularity images?

Kmrouj commented 4 years ago

Hi, Good news, with the help of Stanford personal (cluster) we were able to figure out where was the problem (with singularity) we used 4 cores to pull the singularity image:

[karim90@sh02-ln01 login ~]$ srun -c 4 --pty bash [karim90@sh02-01n30 ~]$ cd $PI_HOME [karim90@sh02-01n30 /home/groups/pbeachy]$ singularity pull rpipe docker://databio/rpipe INFO: Converting OCI blobs to SIF format INFO: Starting build...

INFO: Creating SIF file... INFO: Build complete: rpipe

I want to ask you since I pulled directly the singularity image and now I have it in my $PI_HOME/Karim Directory I guess I don’t need to do: bulker activate databio/pepatac

Can I start with the test project, or there’s another step to do before.

Thanks

nsheff commented 4 years ago

well, bulker puts it in a specific spot that it manages, so you'd be best of now using bulker to build it... you can type: bulker load -b databio/pepatac, for example, and it will build all of the singularity containers and put them in the correct places.

Kmrouj commented 4 years ago

Just a quick question regarding my directory. Right now the pulled image is here (I made a special dir for this): [karim90@sh02-01n30 /home/groups/pbeachy/Karim/singularity/pepatac]$ ls pipe

Can I stay in this same directory to "bulker load -b databio/pepatac" or move to the parent directory in my case /home/groups/pbeachy/Karim

Thanks

nsheff commented 4 years ago

the directory shouldn't matter, bulker will manage the locations according what you have listed in your bulker config file.

Kmrouj commented 4 years ago

It did not work

karim90@sh02-01n30 /home/groups/pbeachy/Karim/singularity/pepatac]$ bulker load -b databio/pepatac Using default config. No config found in env var: BULKERCFG Bulker config: /home/users/karim90/.local/lib/python3.6/site-packages/bulker/templates/bulker_config.yaml Building images with template: /home/users/karim90/.local/lib/python3.6/site-packages/bulker/templates/docker_build.jinja2 Importing crate 'bulker/alpine:default' from '/home/users/karim90/bulker_crates/bulker/alpine/default'. Importing crate 'bulker/coreutils:default' from '/home/users/karim90/bulker_crates/bulker/coreutils/default'. sh: line 2: docker: command not found ------ Error building. Build script used: ------

!/bin/sh

docker pull quay.io/biocontainers/ucsc-bedgraphtobigwig:357--h35c10e6_3

Traceback (most recent call last): File "/home/users/karim90/.local/lib/python3.6/site-packages/attmap/ordattmap.py", line 45, in getitem return super(OrdAttMap, self).getitem(item) KeyError: 'singularity_fullpath'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/users/karim90/.local/bin/bulker", line 8, in sys.exit(main()) File "/home/users/karim90/.local/lib/python3.6/site-packages/bulker/bulker.py", line 870, in main force=args.force) File "/home/users/karim90/.local/lib/python3.6/site-packages/bulker/bulker.py", line 426, in bulker_load _LOGGER.info("Container available at: {cmd}".format(cmd=pkg["singularity_fullpath"])) File "/home/users/karim90/.local/lib/python3.6/site-packages/attmap/pathex_attmap.py", line 51, in getitem v = super(PathExAttMap, self).getitem(item) File "/home/users/karim90/.local/lib/python3.6/site-packages/attmap/ordattmap.py", line 47, in getitem return AttMap.getitem(self, item) File "/home/users/karim90/.local/lib/python3.6/site-packages/attmap/attmap.py", line 32, in getitem return self.dict[item] KeyError: 'singularity_fullpath'

Kmrouj commented 4 years ago

Here's my configuration for bulker_config.yaml:

[karim90@sh02-01n30 /home/groups/pbeachy]$ cat bulker_config.yaml bulker: volumes: ['$HOME'] envvars: ['DISPLAY'] registry_url: http://hub.bulker.io/ shell_path: ${SHELL} shell_rc: ${HOME}/.bashrc rcfile: templates/start.sh rcfile_strict: templates/start_strict.sh default_crate_folder: ${HOME}/bulker_crates singularity_image_folder: ${HOME}/simages container_engine: singularity default_namespace: bulker executable_template: templates/singularity_executable.jinja2 shell_template: templates/singularity_shell.jinja2 build_template: templates/singularity_build.jinja2 crates: bulker: demo: default: /home/users/karim90/bulker_crates/bulker/demo/default [karim90@sh02-01n30 /home/groups/pbeachy]$

nsheff commented 4 years ago

hi, you need to make sure to set up the config correctly. See the error message: Using default config. No config found in env var: BULKERCFG

So, it's not using the config. and therefore trying to use docker (the default), which you don't have installed. You have to pass it as -c or set up a BULKERCFG environment variable. I'd recommend reading the documentation, docs.bulker.io. You can find the part on how to configure bulker here: https://bulker.databio.org/en/latest/install/

Kmrouj commented 4 years ago

I followed the documentation by defining BULKERCFG environment variable but it did not work

[karim90@sh01-ln04 login ~]$ export BULKERCFG=~/home/users/karim90/bulker_config.yaml [karim90@sh01-ln04 login ~]$ export BULKERCFG="bulker_config.yaml" [karim90@sh01-ln04 login ~]$ cd $PI_HOME [karim90@sh01-ln04 login /home/groups/pbeachy]$ cd /home/groups/pbeachy/Karim [karim90@sh01-ln04 login /home/groups/pbeachy/Karim]$ bulker load -b databio/pepatac Using default config. No config found in env var: BULKERCFG Bulker config: /home/users/karim90/.local/lib/python3.6/site-packages/bulker/templates/bulker_config.yaml Building images with template: /home/users/karim90/.local/lib/python3.6/site-packages/bulker/templates/docker_build.jinja2 Importing crate 'bulker/alpine:default' from '/home/users/karim90/bulker_crates/bulker/alpine/default'. Importing crate 'bulker/coreutils:default' from '/home/users/karim90/bulker_crates/bulker/coreutils/default'. sh: line 2: docker: command not found ------ Error building. Build script used: ------

!/bin/sh

docker pull quay.io/biocontainers/ucsc-bedgraphtobigwig:357--h35c10e6_3

Traceback (most recent call last): File "/home/users/karim90/.local/lib/python3.6/site-packages/attmap/ordattmap.py", line 45, in getitem return super(OrdAttMap, self).getitem(item) KeyError: 'singularity_fullpath'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/users/karim90/.local/bin/bulker", line 8, in sys.exit(main()) File "/home/users/karim90/.local/lib/python3.6/site-packages/bulker/bulker.py", line 870, in main force=args.force) File "/home/users/karim90/.local/lib/python3.6/site-packages/bulker/bulker.py", line 426, in bulker_load _LOGGER.info("Container available at: {cmd}".format(cmd=pkg["singularity_fullpath"])) File "/home/users/karim90/.local/lib/python3.6/site-packages/attmap/pathex_attmap.py", line 51, in getitem v = super(PathExAttMap, self).getitem(item) File "/home/users/karim90/.local/lib/python3.6/site-packages/attmap/ordattmap.py", line 47, in getitem return AttMap.getitem(self, item) File "/home/users/karim90/.local/lib/python3.6/site-packages/attmap/attmap.py", line 32, in getitem return self.dict[item] KeyError: 'singularity_fullpath'

Kmrouj commented 4 years ago

To give you a clear idea; Here's the full list of commands I have done (mainly by following the documentation) I also included the error message after try to load:

ml python/3.6.1 export BULKERCFG="bulker_config.yaml" bulker init -c $BULKERCFG export BULKERCFG=/home/users/karim90/bulker_config.yaml bulker load -b databio/pepatac

Error

Populating host commands Traceback (most recent call last): File "/home/users/karim90/.local/bin/bulker", line 8, in sys.exit(main()) File "/home/users/karim90/.local/lib/python3.6/site-packages/bulker/bulker.py", line 870, in main force=args.force) File "/home/users/karim90/.local/lib/python3.6/site-packages/bulker/bulker.py", line 441, in bulker_load os.symlink(local_exe, path) FileExistsError: [Errno 17] File exists: '/share/software/user/open/python/3.6.1/bin/python3' -> '/home/users/karim90/bulker_crates/databio/pepatac/default/python3'