minio client behaving inconsistly in pods (storage benchmarking automation)

[dystewart]

Here is the repo housing all the applicable yaml: https://github.com/OCP-on-NERC/fio-openshift-storage-performance

Essentially what the job files are intended to do is:

1. run kubestr and fio tests
2. use mc to send output files to object storage

Here is an example of the inconsistency I'm running into Take a look at pod.yaml specifically:

       - image: docker.io/minio/mc
         command: ["mc"]
         args: ["alias", "set", "store", "$(S3Endpoint)", "$(AWS_ACCESS_KEY_ID)", "$(AWS_SECRET_ACCESS_KEY)"]

Deploying this pod, (with the associated obc.yaml and target-pvc.yaml) results in the output: Addedstoresuccessfully.

Take a look at benchmark-pv.yaml, specifically:

        - image: docker.io/minio/mc
          imagePullPolicy: IfNotPresent
          name: mc-results-to-bucket
          workingDir:
          command: 
            - /mc/mc-results-to-bucket.sh

Here is mc-results-to-bucket.sh:

apiVersion: v1
kind: ConfigMap
metadata:
  name: mc-results-to-bucket
data:
  mc-results-to-bucket.sh: |
    #!/bin/sh
    set -ue
    echo "Setting up mc connection to S3 Bucket:"
    mc alias set store ${S3Endpoint} ${AWS_ACCESS_KEY_ID} ${AWS_SECRET_ACCESS_KEY} --api S3v4

When this job runs however the output is: mc: <ERROR> Unable to save new mc config. mkdir /.mc: permission denied.

I'm using the same base image: docker.io/minio/mc so I'm not sure why the behavior would be different. Is there something I'm missing about deploying a pod via a job vs deploying a pod directly?

Any ideas?

[dystewart]

@larsks

[larsks]

OpenShift generally runs pods with a random UID. We can't assume write access anywhere in volumes that we've mounted on the pod. Mounting emptyDir volume for the config and pointing mc at it resolved this problem.