search

databio / bulker

Manager for multi-container computing environments
https://bulker.io
BSD 2-Clause "Simplified" License
24 stars 2 forks source link

Using apptainer instead of singularity #93

Closed rcorces closed 8 months ago

rcorces commented 8 months ago

We've finally completely lost singularity from our server due to deprecation for apptainer. I tried getting this to work by updating the bulker templates but something still isnt working.

My bulker_config.yaml file looks like this:

bulker:
  volumes: 
   - /corces
   - /dev
   - /tmp
   - /opt
   - /gladstone/corces/lab
   - /gladstone/corces/sequencing
  envvars: 
   - DISPLAY
   - GENOMES
  registry_url: http://hub.bulker.io/
  shell_path: ${SHELL}
  shell_rc: ${HOME}/.bashrc
  rcfile: templates_corceslab/start.sh
  rcfile_strict: templates_corceslab/start_strict.sh
  default_crate_folder: /corces/home/shared/tools/bulker/bulker_crates
  singularity_image_folder: /corces/home/shared/tools/apptainer/simages
  container_engine: apptainer
  default_namespace: bulker
  executable_template: templates_corceslab/singularity_executable.jinja2
  shell_template: templates_corceslab/singularity_shell.jinja2
  build_template: templates_corceslab/singularity_build.jinja2
  crates:
    bulker:
      demo:
        default: /corces/home/shared/tools/bulker/bulker_crates/bulker/demo/default

singularity_build.jinja2 looks like this:

#!/bin/sh

if [ ! -f "{{ pkg.singularity_fullpath }}" ]; then
  apptainer pull {{ pkg.singularity_image }} docker://{{ pkg.docker_image }}
  mv {{ pkg.singularity_image }} {{ pkg.singularity_fullpath }}
fi

singularity_executable.jinja2 looks like this:

#!/bin/sh

if [ ! -f "{{ pkg.singularity_fullpath }}" ]; then
  apptainer pull {{ pkg.singularity_image }} docker://{{ pkg.docker_image }}
  mv {{ pkg.singularity_image }} {{ pkg.singularity_fullpath }}
fi

LC_ALL=C apptainer exec \{% if pkg.singularity_args %}
  {{ pkg.singularity_args }} \{% endif %}{% for volume in pkg.volumes %}{% if volume != "$HOME"  and volume != "${HOME}" %}
  -B "{{volume}}:{{volume}}" \{% endif %}{% endfor %}
  {{ pkg.singularity_fullpath }}{% if pkg.singularity_command %} {{ pkg.singularity_command }}{% elif pkg.docker_command %} {{ pkg.docker_command }}{% elif pkg.command %} {{ pkg.command }}{% endif %} "$@"

and singularity_shell.jinja2 looks like this:

#!/bin/sh

LC_ALL=C apptainer shell \{% if pkg.singularity_args %}
  {{ pkg.singularity_args }} \{% endif %}{% for volume in pkg.volumes %}{% if volume != "$HOME"  and volume != "${HOME}" %}
  -B "{{volume}}:{{volume}}" \{% endif %}{% endfor %}
  {{ pkg.singularity_fullpath }}

effectively just replacing singularity for apptainer

When I try to run the demo for fortune | cowsay, I get a couple of weird behaviors. First, despite having the path for singularity_image_folder: /corces/home/shared/tools/apptainer/simages, bulker creates .sif files in the working directory. I think this must be because this is what normally happens but then the file gets moved and the mv command is failing here. There is also a problem with apptainer exec presumably because my assumption that I could swap singularity directly for apptainer in the templates is not correct. Output below. Any help would be greatly appreciated!

(p3.6.8) [rcorces@pelayo bulker]$ eval $(bulker activate demo -p -e)
Activating bulker crate: demo
(p3.6.8) [rcorces@pelayo bulker]$ fortune | cowsay
INFO:    Converting OCI blobs to SIF format
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
INFO:    Starting build...
Getting image source signatures
Getting image source signatures
Copying blob f6761dbba445 [--------------------------------------] 0.0b / 33.7MiB
Copying blob 01b8b12bad90 [--------------------------------------] 0.0b / 847.0b
Copying blob b6b268720157 [--------------------------------------] 0.0b / 849.0b
Copying blob c64513b74145 [--------------------------------------] 0.0b / 30.2MiB
Copying blob c5d85cf7a05f [--------------------------------------] 0.0b / 468.0b
Copying blob e12192999ff1 [--------------------------------------] 0.0b / 163.0b
Copying blob 834a54f7272b done  
Copying blob c5d85cf7a05f done  
Copying blob 01b8b12bad90 done  
Copying blob b6b268720157 done  
Copying blob c64513b74145 done  
Copying blob e12192999ff1 done  
Copying config 702c087bf9 done  
Writing manifest to image destination
Storing signatures
Copying config 5e30789bdf done  
Writing manifest to image destination
Storing signatures
FATAL:   While making image from oci registry: error fetching image to cache: while building SIF from layers: conveyor failed to get: while getting config: no descriptor found for reference "9244157d5b97c62a135aec0582848cd65dec07717281a9122653d6c281bef2d1"
mv: missing file operand
Try 'mv --help' for more information.
Error for command "exec": requires at least 2 arg(s), only received 1

Usage:
  apptainer [global options...] exec [exec options...] <container> <command>

Run 'apptainer --help' for more detailed usage information.
2024/03/12 12:26:27  info unpack layer: sha256:c64513b741452f95d8a147b69c30f403f6289542dd7b2b51dd8ba0cb35d0e08b
2024/03/12 12:26:27  warn rootless{dev/full} creating empty file in place of device 1:7
2024/03/12 12:26:27  warn rootless{dev/null} creating empty file in place of device 1:3
2024/03/12 12:26:27  warn rootless{dev/ptmx} creating empty file in place of device 5:2
2024/03/12 12:26:27  warn rootless{dev/random} creating empty file in place of device 1:8
2024/03/12 12:26:27  warn rootless{dev/tty} creating empty file in place of device 5:0
2024/03/12 12:26:27  warn rootless{dev/urandom} creating empty file in place of device 1:9
2024/03/12 12:26:27  warn rootless{dev/zero} creating empty file in place of device 1:5
2024/03/12 12:26:28  info unpack layer: sha256:01b8b12bad90b51d9f15dd4b63103ea6221b339ac3b3e75807c963e678f28624
2024/03/12 12:26:28  info unpack layer: sha256:c5d85cf7a05fec99bb829db84dc5a21cc0aca569253f45d1ea10ca9e8a03fa9a
2024/03/12 12:26:28  info unpack layer: sha256:b6b268720157210d21bbe49f6112f815774e6d2a6144b14911749fadfdb034f0
2024/03/12 12:26:28  info unpack layer: sha256:e12192999ff18f01315563c63333d7c1059cd8e64dffe75fffe504b95eeb093c
2024/03/12 12:26:28  info unpack layer: sha256:834a54f7272b02b4924affec1dfae6d380640225961d10b63c2ac1832fe53918
INFO:    Creating SIF file...
mv: missing file operand
Try 'mv --help' for more information.
Error for command "exec": requires at least 2 arg(s), only received 1

Usage:
  apptainer [global options...] exec [exec options...] <container> <command>

Run 'apptainer --help' for more detailed usage information.
nsheff commented 8 months ago

Thanks -- we've also switched from singularity to apptainer now, and I was just messing around with this... I hadn't quite gotten as far as you, though. I'll take a look at it...

nsheff commented 8 months ago

did you try just using the original bulker, but aliasing singularity to apptainer? This is actually working for me, in simple cases...

nsheff commented 8 months ago

It worked for me, using apptainer version 1.2.2, but just using the 'singularity' executable with no changes to bulker:

 bulker activate demo
Activating bulker crate: demo
bulker/demo|~$ cowsay "boo"
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Getting image source signatures
Copying blob 834a54f7272b done  
Copying blob c64513b74145 done  
Copying blob 01b8b12bad90 done  
Copying blob e12192999ff1 done  
Copying blob b6b268720157 done  
Copying blob c5d85cf7a05f done  
Copying config 5e30789bdf done  
Writing manifest to image destination
Storing signatures
2024/03/12 16:05:17  info unpack layer: sha256:c64513b741452f95d8a147b69c30f403f6289542dd7b2b51dd8ba0cb35d0e08b
2024/03/12 16:05:17  warn rootless{dev/full} creating empty file in place of device 1:7
2024/03/12 16:05:17  warn rootless{dev/null} creating empty file in place of device 1:3
2024/03/12 16:05:17  warn rootless{dev/ptmx} creating empty file in place of device 5:2
2024/03/12 16:05:17  warn rootless{dev/random} creating empty file in place of device 1:8
2024/03/12 16:05:17  warn rootless{dev/tty} creating empty file in place of device 5:0
2024/03/12 16:05:17  warn rootless{dev/urandom} creating empty file in place of device 1:9
2024/03/12 16:05:17  warn rootless{dev/zero} creating empty file in place of device 1:5
2024/03/12 16:05:18  warn xattr{etc/gshadow} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
2024/03/12 16:05:18  warn xattr{/scratch/ns5bc/tmp/build-temp-3448195226/rootfs/etc/gshadow} destination filesystem does not support xattrs, further warnings will be suppressed
2024/03/12 16:05:40  info unpack layer: sha256:01b8b12bad90b51d9f15dd4b63103ea6221b339ac3b3e75807c963e678f28624
2024/03/12 16:05:40  info unpack layer: sha256:c5d85cf7a05fec99bb829db84dc5a21cc0aca569253f45d1ea10ca9e8a03fa9a
2024/03/12 16:05:40  info unpack layer: sha256:b6b268720157210d21bbe49f6112f815774e6d2a6144b14911749fadfdb034f0
2024/03/12 16:05:40  info unpack layer: sha256:e12192999ff18f01315563c63333d7c1059cd8e64dffe75fffe504b95eeb093c
2024/03/12 16:05:40  info unpack layer: sha256:834a54f7272b02b4924affec1dfae6d380640225961d10b63c2ac1832fe53918
2024/03/12 16:05:57  warn xattr{var/cache/apt/archives/partial} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
2024/03/12 16:05:57  warn xattr{/scratch/ns5bc/tmp/build-temp-3448195226/rootfs/var/cache/apt/archives/partial} destination filesystem does not support xattrs, further warnings will be suppressed
INFO:    Creating SIF file...
INFO:    gocryptfs not found, will not be able to use gocryptfs
INFO:    underlay of /etc/localtime required more than 50 (70) bind mounts
 _____
< boo >
 -----
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||
bulker/demo|~$ singularity --version
apptainer version 1.2.2
bulker/demo|~$ fortune | cowsay
INFO:    gocryptfs not found, will not be able to use gocryptfs
INFO:    underlay of /etc/localtime required more than 50 (70) bind mounts
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Getting image source signatures
Copying blob f6761dbba445 done  
Copying blob c5d85cf7a05f skipped: already exists  
Copying blob 01b8b12bad90 skipped: already exists  
Copying blob b6b268720157 skipped: already exists  
Copying blob c64513b74145 skipped: already exists  
Copying blob e12192999ff1 skipped: already exists  
Copying config 702c087bf9 done  
Writing manifest to image destination
Storing signatures
2024/03/12 16:10:07  info unpack layer: sha256:c64513b741452f95d8a147b69c30f403f6289542dd7b2b51dd8ba0cb35d0e08b
2024/03/12 16:10:07  warn rootless{dev/full} creating empty file in place of device 1:7
2024/03/12 16:10:07  warn rootless{dev/null} creating empty file in place of device 1:3
2024/03/12 16:10:07  warn rootless{dev/ptmx} creating empty file in place of device 5:2
2024/03/12 16:10:07  warn rootless{dev/random} creating empty file in place of device 1:8
2024/03/12 16:10:07  warn rootless{dev/tty} creating empty file in place of device 5:0
2024/03/12 16:10:07  warn rootless{dev/urandom} creating empty file in place of device 1:9
2024/03/12 16:10:07  warn rootless{dev/zero} creating empty file in place of device 1:5
2024/03/12 16:10:08  warn xattr{etc/gshadow} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
2024/03/12 16:10:08  warn xattr{/scratch/ns5bc/tmp/build-temp-1213169494/rootfs/etc/gshadow} destination filesystem does not support xattrs, further warnings will be suppressed
2024/03/12 16:10:30  info unpack layer: sha256:01b8b12bad90b51d9f15dd4b63103ea6221b339ac3b3e75807c963e678f28624
2024/03/12 16:10:30  info unpack layer: sha256:c5d85cf7a05fec99bb829db84dc5a21cc0aca569253f45d1ea10ca9e8a03fa9a
2024/03/12 16:10:30  info unpack layer: sha256:b6b268720157210d21bbe49f6112f815774e6d2a6144b14911749fadfdb034f0
2024/03/12 16:10:30  info unpack layer: sha256:e12192999ff18f01315563c63333d7c1059cd8e64dffe75fffe504b95eeb093c
2024/03/12 16:10:30  info unpack layer: sha256:f6761dbba4453fc878ed718984ebcd878db14933d8e4cdd5f78f34834c40cea7
2024/03/12 16:10:30  warn xattr{var/cache/apt/archives/partial} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
2024/03/12 16:10:30  warn xattr{/scratch/ns5bc/tmp/build-temp-1213169494/rootfs/var/cache/apt/archives/partial} destination filesystem does not support xattrs, further warnings will be suppressed
INFO:    Creating SIF file...
INFO:    gocryptfs not found, will not be able to use gocryptfs
INFO:    underlay of /etc/localtime required more than 50 (65) bind mounts
 ________________________________________
/ Just because the message may never be  \
| received does not mean it is not worth |
\ sending.                               /
 ----------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||
nsheff commented 8 months ago

The problem you're having appears to be actually just that it couldn't download the SIF file, or something:

FATAL:   While making image from oci registry: error fetching image to cache: while building SIF from layers: conveyor failed to get: while getting config: no descriptor found for reference "9244157d5b97c62a135aec0582848cd65dec07717281a9122653d6c281bef2d1"

Can you use the pull command directly?

apptainer pull cowsay docker://nsheff/cowsay
rcorces commented 8 months ago

I'm not sure exactly what is going on with what I was doing previously but I'm also not able to get it working using an alias for singularity.

I tried starting from scratch with bulker, creating a new config etc. and getting fortune | cowsay to work but I'm getting an error, presumably from docker.

(p3.6.8) [rcorces@pelayo bulker]$ bulker activate demo
Activating bulker crate: demo
bulker/demo|/corces/home/shared/tools/bulker$ which fortune
/corces/home/shared/tools/bulker/bulker_crates/bulker/demo/default/fortune
bulker/demo|/corces/home/shared/tools/bulker$ which cowsay
/corces/home/shared/tools/bulker/bulker_crates/bulker/demo/default/cowsay
bulker/demo|/corces/home/shared/tools/bulker$ fortune | cowsay
FATAL:   "fortune": executable file not found in $PATH
FATAL:   "cowsay": executable file not found in $PATH

This is what my config file looks like

bulker/demo|/corces/home/shared/tools/bulker$ cat bulker_config.yaml 
bulker:
  volumes: 
   - $HOME
   - /corces
   - /dev
   - /tmp
   - /opt
   - /gladstone/corces/lab
   - /gladstone/corces/sequencing
  envvars: 
   - DISPLAY
   - GENOMES
  registry_url: http://hub.bulker.io/
  shell_path: ${SHELL}
  shell_rc: $HOME/.bashrc
  rcfile: templates/start.sh
  rcfile_strict: templates/start_strict.sh
  default_crate_folder: /corces/home/shared/tools/bulker/bulker_crates
  singularity_image_folder: /corces/home/shared/tools/singularity/simages/
  container_engine: singularity
  default_namespace: bulker
  executable_template: templates/singularity_executable.jinja2
  shell_template: templates/singularity_shell.jinja2
  build_template: templates/singularity_build.jinja2
  crates:
    bulker:
      demo:
        default: /corces/home/shared/tools/bulker/bulker_crates/bulker/demo/default

This shows (I think) that my alias is working fine

bulker/demo|/corces/home/shared/tools/bulker$ which singularity 
alias singularity='apptainer'
        /usr/bin/apptainer
bulker/demo|/corces/home/shared/tools/bulker$
rcorces commented 8 months ago

Actually to answer your previous question, whenever I try to pull with apptainer, it tells me it is using a cached copy.

(p3.6.8) [rcorces@pelayo temp]$ apptainer pull cowsay docker://nsheff/cowsay
INFO:    Using cached SIF image

So if I clear the cache, I am able to pull just fine.

(p3.6.8) [rcorces@pelayo nsheff]$ apptainer pull fortune docker://nsheff/fortune
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Getting image source signatures
Copying blob f6761dbba445 done  
Copying blob c5d85cf7a05f skipped: already exists  
Copying blob c64513b74145 skipped: already exists  
Copying blob 01b8b12bad90 skipped: already exists  
Copying blob e12192999ff1 skipped: already exists  
Copying blob b6b268720157 skipped: already exists  
Copying config 702c087bf9 done  
Writing manifest to image destination
Storing signatures
2024/03/13 09:50:15  info unpack layer: sha256:c64513b741452f95d8a147b69c30f403f6289542dd7b2b51dd8ba0cb35d0e08b
2024/03/13 09:50:15  warn rootless{dev/full} creating empty file in place of device 1:7
2024/03/13 09:50:15  warn rootless{dev/null} creating empty file in place of device 1:3
2024/03/13 09:50:15  warn rootless{dev/ptmx} creating empty file in place of device 5:2
2024/03/13 09:50:15  warn rootless{dev/random} creating empty file in place of device 1:8
2024/03/13 09:50:15  warn rootless{dev/tty} creating empty file in place of device 5:0
2024/03/13 09:50:15  warn rootless{dev/urandom} creating empty file in place of device 1:9
2024/03/13 09:50:15  warn rootless{dev/zero} creating empty file in place of device 1:5
2024/03/13 09:50:16  info unpack layer: sha256:01b8b12bad90b51d9f15dd4b63103ea6221b339ac3b3e75807c963e678f28624
2024/03/13 09:50:16  info unpack layer: sha256:c5d85cf7a05fec99bb829db84dc5a21cc0aca569253f45d1ea10ca9e8a03fa9a
2024/03/13 09:50:16  info unpack layer: sha256:b6b268720157210d21bbe49f6112f815774e6d2a6144b14911749fadfdb034f0
2024/03/13 09:50:16  info unpack layer: sha256:e12192999ff18f01315563c63333d7c1059cd8e64dffe75fffe504b95eeb093c
2024/03/13 09:50:16  info unpack layer: sha256:f6761dbba4453fc878ed718984ebcd878db14933d8e4cdd5f78f34834c40cea7
INFO:    Creating SIF file...

Oddly if I pull in the context of running the fortune | cowsay command after activating bulker demo, then I get those same errors as before

(p3.6.8) [rcorces@pelayo bulker]$ bulker activate demo
Activating bulker crate: demo
bulker/demo|/corces/home/shared/tools/bulker$ fortune | cowsay
INFO:    Converting OCI blobs to SIF format
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
INFO:    Starting build...
Getting image source signatures
Getting image source signatures
Copying blob 834a54f7272b [=>------------------------------------] 1.8MiB / 44.4MiB
Copying blob 01b8b12bad90 [--------------------------------------] 0.0b / 847.0b
Copying blob b6b268720157 [--------------------------------------] 0.0b / 849.0b
Copying blob e12192999ff1 [--------------------------------------] 0.0b / 163.0b
Copying blob c5d85cf7a05f [--------------------------------------] 0.0b / 468.0b
Copying blob c64513b74145 [--------------------------------------] 0.0b / 30.2MiB
Copying blob 01b8b12bad90 done  
Copying blob f6761dbba445 done  
Copying blob c64513b74145 done  
Copying blob c5d85cf7a05f done  
Copying blob e12192999ff1 done  
Copying blob b6b268720157 done  
Copying config 5e30789bdf done  
Writing manifest to image destination
Storing signatures
Copying config 702c087bf9 done  
Writing manifest to image destination
Storing signatures
2024/03/13 09:43:41  info unpack layer: sha256:c64513b741452f95d8a147b69c30f403f6289542dd7b2b51dd8ba0cb35d0e08b
2024/03/13 09:43:41  warn rootless{dev/full} creating empty file in place of device 1:7
2024/03/13 09:43:41  warn rootless{dev/null} creating empty file in place of device 1:3
2024/03/13 09:43:41  warn rootless{dev/ptmx} creating empty file in place of device 5:2
2024/03/13 09:43:41  warn rootless{dev/random} creating empty file in place of device 1:8
2024/03/13 09:43:41  warn rootless{dev/tty} creating empty file in place of device 5:0
2024/03/13 09:43:41  warn rootless{dev/urandom} creating empty file in place of device 1:9
2024/03/13 09:43:41  warn rootless{dev/zero} creating empty file in place of device 1:5
FATAL:   While making image from oci registry: error fetching image to cache: while building SIF from layers: conveyor failed to get: while getting config: no descriptor found for reference "3cd894bdd2b6bb24dda16d1f7c71cd6c8a927589686a8c7927368aa3e3d7296c"
mv: cannot stat 'cowsay': No such file or directory
FATAL:   While checking container encryption: could not open image /corces/home/shared/tools/singularity/simages/nsheff/cowsay: failed to retrieve path for /corces/home/shared/tools/singularity/simages/nsheff/cowsay: lstat /corces/home/shared/tools/singularity/simages/nsheff/cowsay: no such file or directory
2024/03/13 09:43:42  info unpack layer: sha256:01b8b12bad90b51d9f15dd4b63103ea6221b339ac3b3e75807c963e678f28624
2024/03/13 09:43:42  info unpack layer: sha256:c5d85cf7a05fec99bb829db84dc5a21cc0aca569253f45d1ea10ca9e8a03fa9a
2024/03/13 09:43:42  info unpack layer: sha256:b6b268720157210d21bbe49f6112f815774e6d2a6144b14911749fadfdb034f0
2024/03/13 09:43:42  info unpack layer: sha256:e12192999ff18f01315563c63333d7c1059cd8e64dffe75fffe504b95eeb093c
2024/03/13 09:43:42  info unpack layer: sha256:f6761dbba4453fc878ed718984ebcd878db14933d8e4cdd5f78f34834c40cea7
INFO:    Creating SIF file...
FATAL:   "fortune": executable file not found in $PATH

I tried pulling cowsay and fortune individually and putting them in the right location and then running fortune | cowsay but I get the same executable file not found error

nsheff commented 8 months ago

It looks like it may be a file system issue... it says:

FATAL: While making image from oci registry: error fetching image to cache: while building SIF from layers: conveyor failed to get: while getting config: no descriptor found for reference "3cd894bdd2b6bb24dda16d1f7c71cd6c8a927589686a8c7927368aa3e3d7296c"

And: what are these warnings?

2024/03/13 09:50:15  warn rootless{dev/full} creating empty file in place of device 1:7
2024/03/13 09:50:15  warn rootless{dev/null} creating empty file in place of device 1:3
2024/03/13 09:50:15  warn rootless{dev/ptmx} creating empty file in place of device 5:2
2024/03/13 09:50:15  warn rootless{dev/random} creating empty file in place of device 1:8
2024/03/13 09:50:15  warn rootless{dev/tty} creating empty file in place of device 5:0
2024/03/13 09:50:15  warn rootless{dev/urandom} creating empty file in place of device 1:9
2024/03/13 09:50:15  warn rootless{dev/zero} creating empty file in place of device 1:5

And this: lstat /corces/home/shared/tools/singularity/simages/nsheff/cowsay: no such file or directory

Is there some singularity setting that is creating the SIF files somewhere else?

so if you do this, what happens?

apptainer pull cowsay docker://nsheff/cowsay
ls cowsay  # did it build the 'cowsay' image as a file in the pwd?
./cowsay cowsay hello  # you should be able to run it directly

This test is independent of bulker.

rcorces commented 8 months ago

I bet you're right. I'll try to figure out whats going on and get that minimal example to work independent of bulker. It currently doesnt work, despite apptainer creating a cowsay file

(p3.6.8) [rcorces@pelayo temp]$ apptainer pull cowsay docker://nsheff/cowsay
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Getting image source signatures
Copying blob 834a54f7272b done  
Copying blob c5d85cf7a05f done  
Copying blob c64513b74145 done  
Copying blob b6b268720157 done  
Copying blob 01b8b12bad90 done  
Copying blob e12192999ff1 done  
Copying config 5e30789bdf done  
Writing manifest to image destination
Storing signatures
2024/03/13 10:22:02  info unpack layer: sha256:c64513b741452f95d8a147b69c30f403f6289542dd7b2b51dd8ba0cb35d0e08b
2024/03/13 10:22:03  warn rootless{dev/full} creating empty file in place of device 1:7
2024/03/13 10:22:03  warn rootless{dev/null} creating empty file in place of device 1:3
2024/03/13 10:22:03  warn rootless{dev/ptmx} creating empty file in place of device 5:2
2024/03/13 10:22:03  warn rootless{dev/random} creating empty file in place of device 1:8
2024/03/13 10:22:03  warn rootless{dev/tty} creating empty file in place of device 5:0
2024/03/13 10:22:03  warn rootless{dev/urandom} creating empty file in place of device 1:9
2024/03/13 10:22:03  warn rootless{dev/zero} creating empty file in place of device 1:5
2024/03/13 10:22:03  info unpack layer: sha256:01b8b12bad90b51d9f15dd4b63103ea6221b339ac3b3e75807c963e678f28624
2024/03/13 10:22:03  info unpack layer: sha256:c5d85cf7a05fec99bb829db84dc5a21cc0aca569253f45d1ea10ca9e8a03fa9a
2024/03/13 10:22:03  info unpack layer: sha256:b6b268720157210d21bbe49f6112f815774e6d2a6144b14911749fadfdb034f0
2024/03/13 10:22:03  info unpack layer: sha256:e12192999ff18f01315563c63333d7c1059cd8e64dffe75fffe504b95eeb093c
2024/03/13 10:22:03  info unpack layer: sha256:834a54f7272b02b4924affec1dfae6d380640225961d10b63c2ac1832fe53918
INFO:    Creating SIF file...
(p3.6.8) [rcorces@pelayo temp]$ ls cowsay
cowsay
(p3.6.8) [rcorces@pelayo temp]$ ./cowsay cowsay hello
FATAL:   "cowsay": executable file not found in $PATH
(p3.6.8) [rcorces@pelayo temp]$
rcorces commented 8 months ago

Well. Problem solved. Turns out this was all caused by an environment variable I had created for some custom scripts.

export TMPDIR=~/tmp

This was presumably breaking some part of the apptainer caching system. I can confirm that the proposed solution of aliasing singularity to apptainer works. I can also confirm that simply changing "singularity" to "apptainer" in the bulker template files also seems to work.

Thanks for helping with this!

For anyone stumbling upon this in the future, these warnings dont seem to be an issue:

2024/03/13 09:50:15  warn rootless{dev/full} creating empty file in place of device 1:7
2024/03/13 09:50:15  warn rootless{dev/null} creating empty file in place of device 1:3
2024/03/13 09:50:15  warn rootless{dev/ptmx} creating empty file in place of device 5:2
2024/03/13 09:50:15  warn rootless{dev/random} creating empty file in place of device 1:8
2024/03/13 09:50:15  warn rootless{dev/tty} creating empty file in place of device 5:0
2024/03/13 09:50:15  warn rootless{dev/urandom} creating empty file in place of device 1:9
2024/03/13 09:50:15  warn rootless{dev/zero} creating empty file in place of device 1:5