Unable to source singularity_lib.sh

[cabrillo]

Good morning, Every few days 57, we observe that some executed jobs are not able to source the singulatity_lib.sh library. All WN has a cron where 2 times per day is checked, the integrity of the FS (cvmfs_fsck -p), but it seems that this is not solving the problem, in the end we have to do a cvmfs_config wipecache on all WN. Is there any way to keep the fs stable and without corrunctions? The OS is Centos7, and the package installed are: cvmfs-2.10.1-1.el7.x86_64 cvmfs-config-egi-2.6-1.7.obs.el7.noarch MOunted FS are: cvmfs2 25G 1.9G 23G 8% /cvmfs/config-egi.egi.eu cvmfs2 25G 1.9G 23G 8% /cvmfs/cms.cern.ch cvmfs2 25G 1.9G 23G 8% /cvmfs/grid.cern.ch cvmfs2 25G 1.9G 23G 8% /cvmfs/phys-ibergrid.egi.eu cvmfs2 25G 1.9G 23G 8% /cvmfs/ilc.desy.de cvmfs2 25G 1.9G 23G 8% /cvmfs/biomed.egi.eu cvmfs2 25G 1.9G 23G 8% /cvmfs/auger.egi.eu cvmfs2 25G 1.9G 23G 8% /cvmfs/singularity.opensciencegrid.org cvmfs2 25G 1.9G 23G 8% /cvmfs/oasis.opensciencegrid.org cvmfs2 25G 1.9G 23G 8% /cvmfs/wenmr.egi.eu cvmfs2 25G 1.9G 23G 8% /cvmfs/dirac.egi.eu cvmfs2 25G 1.9G 23G 8% /cvmfs/cms-ib.cern.ch cvmfs2 25G 1.9G 23G 8% /cvmfs/unpacked.cern.ch

default options: CVMFS_CACHE_BASE=/tmp/cvmfs2/ CVMFS_QUOTA_LIMIT=25000 CVMFS_DEFAULT_DOMAIN=cern.ch CVMFS_TIMEOUT=5 CVMFS_TIMEOUT_DIRECT=10 CVMFS_STRICT_MOUNT=no CVMFS_CHECK_PERMISSIONS=yes CVMFS_IGNORE_SIGNATURE=no CVMFS_NFILES=65536 CVMFS_SHARED_CACHE=yes CVMFS_NFS_SOURCE=no CVMFS_MOUNT_RW=no

and local settings:

CVMFS_DEBUGLOG=/tmp/cvmfs.log

CVMFS_REPOSITORIES=cms.cern.ch,grid.cern.ch CVMFS_CACHE_BASE=/tmp/cvmfs2/ CVMFS_QUOTA_LIMIT=25000 CVMFS_HTTP_PROXY="http://squid01.ifca.es:3128/|http://squid02.ifca.es:3128/"

Any advise is welcomed. Regards, I

[vvolkl]

One possible issue here is setting CVMFS_CACHE_BASE=/tmp/cvmfs2/ which could interfere with the cache as files are deleted from the tmp dir. Can you try removing this line from the configuration and use the default cache directory of /var/lib/cvmfs?

[cabrillo]

Hi vvolkl, I just change the directory to /var/lib/cvmfs on WNs, I will check the behaviour and give you some feedback in acouple of days. Thanks in advance, I

[vvolkl]

@cabrillo I'm assuming moving the cache away from /tmp fixed this issue, feel free to reopen otherwise.