Sporadic Input/output Errors Issues with singcvmfs

Hello all, I've been trying to use singcvmfs for running jobs on an HPC through SLURM and running into random but frequent Input/output error and Transport endpoint is not connected errors. Example error is below:

Traceback (most recent call last):
  File "import.py", line 2, in <module>
    import matplotlib.pyplot as plt
  File "/cvmfs/cybergis.illinois.edu/software/conda/cjw/wrfhydro-2022-03/lib/python3.7/site-packages/matplotlib/__init__.py", line 109, in <module>
  File "/cvmfs/cybergis.illinois.edu/software/conda/cjw/wrfhydro-2022-03/lib/python3.7/site-packages/matplotlib/rcsetup.py", line 28, in <module>
  File "/cvmfs/cybergis.illinois.edu/software/conda/cjw/wrfhydro-2022-03/lib/python3.7/site-packages/matplotlib/fontconfig_pattern.py", line 15, in <module>
  File "/cvmfs/cybergis.illinois.edu/software/conda/cjw/wrfhydro-2022-03/lib/python3.7/site-packages/pyparsing/__init__.py", line 138, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 724, in exec_module
  File "<frozen importlib._bootstrap_external>", line 859, in get_code
  File "<frozen importlib._bootstrap_external>", line 916, in get_data
OSError: [Errno 5] Input/output error: '/cvmfs/cybergis.illinois.edu/software/conda/cjw/wrfhydro-2022-03/lib/python3.7/site-packages/pyparsing/exceptions.py'

This error was created by a Python file that just tried to import two Python packages. I'm able to bring up the container, inspect the CVMFS repo with ls/cat, and even run simple scripts (for example calculating Pi with Monte Carlo), but when I try to import large Python packages like matplotlib or geopandas, I get errors about half the time.

I'm hoping to get feedback or guidance on how to reduce/eliminate these errors. I'm guessing there might be a small issue with my configuration. I also played around with setting up an alien cache which the README says is possible, but couldn't find documentation on how to execute that. Any solution that can reduce these errors would be appreciated!

My setup process is detailed here: https://github.com/cybergis/singcvmfs-on-keeling, but I'll provide some snippets of files below for convenience:

default.local

CVMFS_HTTP_PROXY="http://cvmfsqu1.cigi.illinois.edu:3128|http://cvmfsqu2.cigi.illinois.edu:3128;DIRECT"
CVMFS_CACHE_BASE="/tmp/cvmfs"
CVMFS_QUOTA_LIMIT=20000

cybergis.illinois.edu.conf

CVMFS_SERVER_URL="http://cvmfstr1.cigi.illinois.edu/cvmfs/@fqrn@"
CVMFS_KEYS_DIR="/etc/cvmfs/keys/illinois.edu"

submit_job.sh

#!/bin/bash
BASE=$(pwd)
export SINGCVMFS_REPOSITORIES="grid.cern.ch,cybergis.illinois.edu"
tmp_path="/tmp/cvmfs-$(openssl rand -hex 32)"
mkdir $tmp_path
source ~/.bashrc

singcvmfs -s exec -B $tmp_path:/tmp/cvmfs,$BASE/script:/script,$BASE/output:/output -cip ~/simages/cvmfs.sif bash -c "cd /script && /cvmfs/cybergis.illinois.edu/software/conda/cjw/wrfhydro-2022-03/bin/python import.py"

import.py

import geopandas as gpd
import matplotlib.pyplot as plt

The HPC I'm running on (Keeling at UIUC) is Centos 7 and our container image is basically Centos7 plus Lmod:

https://hub.docker.com/repository/docker/cybergisx/compute-cvmfs/general

Dockerfile:

FROM centos:7

# update and install essentials and lua
RUN yum -y update
RUN yum -y install epel-release && \
  yum repolist
RUN yum -y install bc \
  gcc \
  git \
  tcl-devel \
  Lmod \
  lua \
  lua5.1-bitop \
  lua-json \
  lua-libs \
  lua-lpeg \
  lua-posix \
  lua-term \
  make \
  procps \
  wget \
  which

CMD ["source", "/etc/profile.d/z00_lmod.sh"]

Our specific use-case is integrating singcvmfs into our CyberGIS-Compute software so that users have the same software environment as our CyberGIS-Jupyter which utilizes a software environment in a CVMFS repo I made. CVMFS has been working perfectly for our JupyterHub so I don't think the issue is our repo, only ran into issues when we started playing with singcvmfs.

Happy to provide any other info required, thank you for your time!

Wow, it looks like you have taken this further than anyone else has.

My first piece of advice is, if you have unprivileged user namespaces available, please consider using cvmfsexec mode 3 instead of singcvmfs. It's a lot more foolproof.

Debugging with singcvmfs is very challenging. Transport endpoint is not connected indicates that the cvmfs2 process has exited for some reason. Setting SINGCVMFS_LEVEL=debug and looking in the debug log for messages related to a cvmfs2 exit can help.

Had you found the general cvmfs alien cache documentation?

My first piece of advice is, if you have unprivileged user namespaces available, please consider using cvmfsexec mode 3 instead of singcvmfs. It's a lot more foolproof.

Looking through the documentation I couldn't figure out how to accomplish this. I cloned the repo, set the keys/config files like with singcvmfs and created an executable called mycvmfsexec. However, looking the documentation over, I can't quite figure out the syntax. I tried a few variations on the below:

>./mycvmfsexec cybergis.illinois.edu -- ls
unshare: unshare failed: Invalid argument

How would I run simple commands with this executable?

Debugging with singcvmfs is very challenging. Transport endpoint is not connected indicates that the cvmfs2 process has exited for some reason. Setting SINGCVMFS_LEVEL=debug and looking in the debug log for messages related to a cvmfs2 exit can help.

Thank you for this, the logs have yielded some valuable information like our open files ulimit being set too low on the cluster and issues with our squid proxies (forgot to allow the HPC nodes). I fixed the proxies which seems to have drastically reduced the frequency of errors and have reached out to our IT about the ulimits. The errors look like:

Thu May  4 13:21:31 2023 (cybergis.illinois.edu) decompressing /data/72/e053748b3c7d9a789ef978474ddc25f61ac077, local IO error
Thu May  4 13:21:31 2023 (cybergis.illinois.edu) failed to fetch /software/conda/cybergisx/python3-0.9.0/lib/libicuuc.so.68.2 (hash: 72e053748b3c7d9a789ef978474ddc25f61ac077, error 1 [local I/O failure])
Thu May  4 13:21:31 2023 (cybergis.illinois.edu) failed to open inode: 64686, CAS key 72e053748b3c7d9a789ef978474ddc25f61ac077, error code 2
Thu May  4 13:21:31 2023 (cybergis.illinois.edu) decompressing /data/72/e053748b3c7d9a789ef978474ddc25f61ac077, local IO error
Thu May  4 13:21:31 2023 (cybergis.illinois.edu) failed to fetch /software/conda/cybergisx/python3-0.9.0/lib/libicuuc.so.68.2 (hash: 72e053748b3c7d9a789ef978474ddc25f61ac077, error 1 [local I/O failure])
Thu May  4 13:21:31 2023 (cybergis.illinois.edu) failed to open inode: 64686, CAS key 72e053748b3c7d9a789ef978474ddc25f61ac077, error code 2
Thu May  4 13:21:31 2023 (cybergis.illinois.edu) decompressing /data/72/e053748b3c7d9a789ef978474ddc25f61ac077, local IO error
Thu May  4 13:21:31 2023 (cybergis.illinois.edu) failed to fetch /software/conda/cybergisx/python3-0.9.0/lib/libicuuc.so.68.2 (hash: 72e053748b3c7d9a789ef978474ddc25f61ac077, error 1 [local I/O failure])
Thu May  4 13:21:31 2023 (cybergis.illinois.edu) failed to open inode: 64686, CAS key 72e053748b3c7d9a789ef978474ddc25f61ac077, error code 2
Thu May  4 13:21:32 2023 (cybergis.illinois.edu) decompressing /data/05/dca7cae7f3fd0dd37e8ebbe3bc61309deef66f, local IO error

Had you found the general cvmfs alien cache documentation?

I have. I downloaded the repo onto the HPC using the cvmfs_preload command, but didn't quite understand how to set the alien cache with singcvmfs. I was also unsure as to how I could install the cvmfs2 binary on the HPC to create the skeleton of the alien cache.

Thank you for your hard work on this project and for taking the time to reply!

My first piece of advice is, if you have unprivileged user namespaces available, please consider using cvmfsexec mode 3 instead of singcvmfs. It's a lot more foolproof.

Looking through the documentation I couldn't figure out how to accomplish this. I cloned the repo, set the keys/config files like with singcvmfs and created an executable called mycvmfsexec. However, looking the documentation over, I can't quite figure out the syntax. I tried a few variations on the below:
>./mycvmfsexec cybergis.illinois.edu -- ls
unshare: unshare failed: Invalid argument

That indicates that unprivileged user namespaces are not enabled on your machine. On el7, they are not enabled by default. The apptainer documentation shows how to enable them.

How would I run simple commands with this executable?

You had the syntax right.

Debugging with singcvmfs is very challenging. Transport endpoint is not connected indicates that the cvmfs2 process has exited for some reason. Setting SINGCVMFS_LEVEL=debug and looking in the debug log for messages related to a cvmfs2 exit can help.

Thu May  4 13:21:31 2023 (cybergis.illinois.edu) decompressing /data/72/e053748b3c7d9a789ef978474ddc25f61ac077, local IO error
Thu May  4 13:21:31 2023 (cybergis.illinois.edu) failed to fetch /software/conda/cybergisx/python3-0.9.0/lib/libicuuc.so.68.2 (hash: 72e053748b3c7d9a789ef978474ddc25f61ac077, error 1 [local I/O failure])
Thu May  4 13:21:31 2023 (cybergis.illinois.edu) failed to open inode: 64686, CAS key 72e053748b3c7d9a789ef978474ddc25f61ac077, error code 2
Thu May  4 13:21:31 2023 (cybergis.illinois.edu) decompressing /data/72/e053748b3c7d9a789ef978474ddc25f61ac077, local IO error
Thu May  4 13:21:31 2023 (cybergis.illinois.edu) failed to fetch /software/conda/cybergisx/python3-0.9.0/lib/libicuuc.so.68.2 (hash: 72e053748b3c7d9a789ef978474ddc25f61ac077, error 1 [local I/O failure])
Thu May  4 13:21:31 2023 (cybergis.illinois.edu) failed to open inode: 64686, CAS key 72e053748b3c7d9a789ef978474ddc25f61ac077, error code 2
Thu May  4 13:21:31 2023 (cybergis.illinois.edu) decompressing /data/72/e053748b3c7d9a789ef978474ddc25f61ac077, local IO error
Thu May  4 13:21:31 2023 (cybergis.illinois.edu) failed to fetch /software/conda/cybergisx/python3-0.9.0/lib/libicuuc.so.68.2 (hash: 72e053748b3c7d9a789ef978474ddc25f61ac077, error 1 [local I/O failure])
Thu May  4 13:21:31 2023 (cybergis.illinois.edu) failed to open inode: 64686, CAS key 72e053748b3c7d9a789ef978474ddc25f61ac077, error code 2
Thu May  4 13:21:32 2023 (cybergis.illinois.edu) decompressing /data/05/dca7cae7f3fd0dd37e8ebbe3bc61309deef66f, local IO error

What kind of filesystem is /data? It looks like it has having errors writing to it. My strong recommendation is to use a local disk if available, or if not the next best thing is a loopback-mounted filesystem where the filesystem is a single file on a remote disk server. singcvmfs can bind such an ext3 filesystem in with SINGCVMFS_CACHEIMAGE (although without unprivileged user namespaces that's taking advantage of a feature that apptainer-1.1.8 just disabled by default for security reasons). Some people have also gotten away with RAM disks.

Had you found the general cvmfs alien cache documentation?

I have. I downloaded the repo onto the HPC using the cvmfs_preload command, but didn't quite understand how to set the alien cache with singcvmfs. I was also unsure as to how I could install the cvmfs2 binary on the HPC to create the skeleton of the alien cache.

You would need to edit dist/etc/cvmfs/default.local to set CVMFS_ALIEN_CACHE. I don't recommend it, though. Putting the cvmfs cache on a shared networked filesystem usually is a bad idea, because the metadata server of the filesystem becomes a bottleneck when lots of nodes start up and access the same files around the same time. A separate cache per node works better, even if it has to be a loopback-mounted cache filesystem.

What kind of filesystem is /data? It looks like it has having errors writing to it.

No idea, the path "/data/72" doesn't seem to exist in the container or the HPC. /data exists on the HPC, but I don't have write permissions to it.

The solution I'm currently using is to create a random folder in /tmp on the HPC node and using that as the cache. The code looks like:

tmp_path="/tmp/cvmfs-$(openssl rand -hex 32)"
mkdir $tmp_path
export SINGCVMFS_CACHEDIR=$tmp_path

singcvmfs -s exec -B $tmp_path:/tmp/cvmfs -cip ~/simages/cvmfs.sif [commands]

rm -r $tmp_path

From some experimentation though, it only seems to come up if there are multiple SLURM jobs using CVMFS on the same node.

My strong recommendation is to use a local disk if available, or if not the next best thing is a loopback-mounted filesystem where the filesystem is a single file on a remote disk server. singcvmfs can bind such an ext3 filesystem in with SINGCVMFS_CACHEIMAGE (although without unprivileged user namespaces that's taking advantage of a feature that apptainer-1.1.8 just disabled by default for security reasons). Some people have also gotten away with RAM disks.

I'll spend a bit more time looking into these solutions and work with the HPC IT here to see if I can get one if these working, thank you!

cvmfs / cvmfsexec

Sporadic Input/output Errors Issues with singcvmfs #69