ROBUSTNESS: Assert that `catfs` processes are terminated if a job terminates

I've verified that an SGE job on Wynton that forgets to do easycatfs unmount --all does not leave stray catfs processes behind. I've tested by submitting the following script to the same host multiple times using:

qsub -cwd -j yes -l hostname=qb3-ih20 ex-skip-unmount.sh

$ cat ex-skip-unmount.sh
#! /usr/bin/env bash

module load CBI easycatfs

echo "Hostname: $(hostname)"

echo "Existing job-specific /scratch folders:"
dirs=($(find /scratch -maxdepth 1 -user "${USER}" -type d))
echo " - count: ${#dirs[@]}"
for dir in "${dirs[@]}"; do
    du -s --bytes "${dir}"
done

echo "Pre-existing mounts:"
easycatfs mounts --full

echo "Pre-existing catfs processes:"
pids=($(pgrep catfs))
echo " - count: ${#pids[@]}"
for pid in "${pids[@]}"; do
    echo "catfs (pid $pid):"
    pstree -a -p -l "$pid"
done

echo "Mounting current folder:"
L_PWD=$(easycatfs mount "$PWD")
easycatfs mounts --full

md5sum "${L_PWD}"/data/1024MiB.bin

easycatfs cache-size --all --full

## Force kill current process
echo "Terminating current process ..."
kill -TERM "$$"
echo "This line should never be reached"

The result is always:

Pre-existing catfs processes:
 - count: 0

HenrikBengtsson / easycatfs

ROBUSTNESS: Assert that `catfs` processes are terminated if a job terminates #15