Configure singularity for NIH Biowulf compatibility

[cmarkello]

@adamnovak Ready for review.

[adamnovak]

Hey @cmarkello, this looks good, but I've hacked on it a bit to give it at least a chance of working on Kubernetes. Can you test it to see if it still works on Biowulf?

[cmarkello]

@adamnovak That appears to work on my system. Go ahead and merge if you think this works for you too.

[adamnovak]

OK, I just ran this on Kubernetes also and had it successfully run Singularity with user-mode namespaces. I think we can merge this.

Script, for reference:

kubectl --namespace toil-workflow delete job adamnovak-make-small ; kubectl --namespace toil-workflow apply -f - <<'EOF'
apiVersion: batch/v1
kind: Job
metadata:
  name: adamnovak-make-small
spec:
  ttlSecondsAfterFinished: 3600
  template:
    spec:
      containers:
      - name: main
        imagePullPolicy: Always
        image: quay.io/adamnovak/toil:3.22.0a1-7e8b68c5fe1b31e5f43b7b23a3a49a3cc76a9d9b
        command:
        - /bin/bash
        - -c
        - |
          set -e
          mkdir /tmp/work
          cd /tmp/work
          echo "Making graphs"
          apt-get update -qq -y && apt-get install -qq -y awscli
          virtualenv --system-site-packages --python python2.7 venv2
          . venv2/bin/activate
          pip install setuptools==44.0.0 git+https://github.com/vgteam/toil-vg.git@f50911a3faef0050cc5c60e2479085d1da31dc6f#egg=toil-vg
          toil clean aws:us-west-2:adamnovak-toiltestvg || true
          toil-vg construct \
            aws:us-west-2:adamnovak-toiltestvg \
            aws:us-west-2:vg-k8s/users/adamnovak/trash/small \
            --fasta https://github.com/vgteam/vg/raw/master/test/small/x.fa \
            --vcf https://github.com/vgteam/vg/raw/master/test/small/x.vcf.gz \
            --out_name small \
            --flat_alts \
            --xg_index \
            --pangenome \
            --normalize \
            --fasta_regions \
            --merge_graphs \
            --keep_vcfs \
            --realTimeLogging \
            --logInfo \
            --batchSystem kubernetes \
            --clean always \
            '--container' Singularity \
            --vg_docker quay.io/vgteam/vg:v1.22.0
        lifecycle:
          preStop:
            exec:
              command:
              - /bin/bash
              - -c
              - |
                pgrep -P 1 | xargs kill -2
        volumeMounts:
        - mountPath: /tmp
          name: scratch-volume
        - mountPath: /root/.aws
          name: s3-credentials
        resources:
          limits:
            cpu: 2
            memory: "4Gi"
            ephemeral-storage: "10Gi"
        env:
        - name: TOIL_KUBERNETES_OWNER
          value: adamnovak
        - name: TOIL_AWS_SECRET_NAME
          value: shared-s3-credentials
        - name: DEBIAN_FRONTEND
          value: noninteractive
      restartPolicy: Never
      volumes:
      - name: scratch-volume
        emptyDir: {}
      - name: s3-credentials
        secret:
          secretName: shared-s3-credentials
      serviceAccountName: toil-workflow-svc
  backoffLimit: 0
EOF