/tmp dependency with singularity

[shahzebsiddiqui]

Version of Singularity:

[siddis14@amrndhl1157 tmp]$ singularity --version
2.4-dist

Recently running into an issue where /tmp gets 100% full because of doing singularity build or pulls. not sure what is causing this. I am doing periodic builds and pulls and storing the images somewhere else but /tmp is being written. If you can identify how this happening that would be great.

[siddis14@amrndhl1157 tmp]$ ls tmp07z1aymu/
bin   cublas_api.h.patch  environment  home  lib64  mnt  proc  run   singularity  sys  usr  workspace
boot  dev                 etc          lib   media  opt  root  sbin  srv          tmp  var
[siddis14@amrndhl1157 tmp]$ ls tmp07z1aymu/bin/
bash          bzmore  dnsdomainname  journalctl  mkdir          pwd         su                              tar           zcmp
bunzip2       cat     domainname     kill        mknod          rbash       sync                            tempfile      zdiff
bzcat         chgrp   echo           less        mktemp         readlink    systemctl                       touch         zegrep
bzcmp         chmod   egrep          lessecho    more           rm          systemd                         true          zfgrep
bzdiff        chown   false          lessfile    mount          rmdir       systemd-ask-password            umount        zforce
bzegrep       cp      fgrep          lesskey     mountpoint     rnano       systemd-escape                  uname         zgrep
bzexe         dash    findmnt        lesspipe    mv             run-parts   systemd-inhibit                 uncompress    zless
bzfgrep       date    grep           ln          nano           sed         systemd-machine-id-setup        vdir          zmore
bzgrep        dd      gunzip         login       networkctl     sh          systemd-notify                  wdctl         znew
bzip2         df      gzexe          loginctl    nisdomainname  sh.distrib  systemd-tmpfiles                which
bzip2recover  dir     gzip           ls          pidof          sleep       systemd-tty-ask-password-agent  ypdomainname
bzless        dmesg   hostname       lsblk       ps             stty        tailf                           zcat

[vsoch]

@shahzebsiddiqui I will try a fix in sregistry to the add/move branch that you are reviewing.

[vsoch]

okay all fixed! Please test. https://github.com/singularityhub/sregistry-cli/pull/67

[dtrudg]

Hi @shahzebsiddiqui - did the fix .@vsoch mentioned fix this? It looks like that was related to the sregistry CLI rather than Singularity itself. If you think the space-filling is due to singularity itself please let us know. Thanks!

[shahzebsiddiqui]

@dctrud @vsoch So we are still running version 2.4-dist and I was building squashfs images which according to singularity writes some files in /tmp.

According to documentation http://singularity.lbl.gov/build-environment

Singularity also uses some temporary directories to build the squashfs filesystem, so this temp space needs to be large enough to hold the entire resulting Singularity image. By default this happens in /tmp but can be overridden by setting SINGULARITY_TMPDIR to the full path where you want the squashfs temp files to be stored. Since images are typically built as root, be sure to set this variable in root’s environment.

Sorry I dont think this is an issue related with sregistry but rather how singularity works. Not sure if this behavior is still present in latest release but we ran out of space in /tmp because we only have few GBs allocated for this.

I suppose I can set SINGULARITY_TMPDIR system-wide to something like lustre to avoid singularity build filling up compute nodes /tmp.

Granted that right now root is limited to HPC users, I will be building bunch of containers and I don't want my container builds to fail due to space. I think it is worth. I think we can close this issue but worth mentioning to singularity team.

[dtrudg]

@shahzebsiddiqui - thank you for your reply. Closing as you suggest, since SINGULARITY_TMPDIR will fix for you. /tmp is still used in the current release for the build process.