[BUG] Minio OpenShift Documentation missing.

[kaovilai]

Describe the bug A clear and concise description of what the bug is. https://docs.min.io/minio/k8s/openshift/deploy-minio-on-openshift.html used to exist. Now it redirects to https://min.io/docs/minio/kubernetes/upstream/operations/installation.html?ref=docs-redirect which has insufficient instructions on OpenShift.

This is the resulting status conditions of the console deployment kubectl minio init on OpenShift.

        pods "console-64cc47fc49-" is forbidden: unable to validate against any
        security context constraint: [provider "anyuid": Forbidden: not usable
        by user or serviceaccount, provider "containerized-data-importer":
        Forbidden: not usable by user or serviceaccount,
        spec.containers[0].securityContext.runAsUser: Invalid value: 1000: must
        be in the ranges: [1000660000, 1000669999], provider "nonroot":
        Forbidden: not usable by user or serviceaccount, provider "noobaa":
        Forbidden: not usable by user or serviceaccount, provider
        "noobaa-endpoint": Forbidden: not usable by user or serviceaccount,
        provider "volsync-mover": Forbidden: not usable by user or
        serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by
        user or serviceaccount, provider "kubevirt-controller": Forbidden: not
        usable by user or serviceaccount, provider "bridge-marker": Forbidden:
        not usable by user or serviceaccount, provider
        "machine-api-termination-handler": Forbidden: not usable by user or
        serviceaccount, provider "k10-prometheus-server": Forbidden: not usable
        by user or serviceaccount, provider "k10-grafana": Forbidden: not usable
        by user or serviceaccount, provider "hostnetwork": Forbidden: not usable
        by user or serviceaccount, provider "hostaccess": Forbidden: not usable
        by user or serviceaccount, provider "linux-bridge": Forbidden: not
        usable by user or serviceaccount, provider "nmstate": Forbidden: not
        usable by user or serviceaccount, provider "kubevirt-handler":
        Forbidden: not usable by user or serviceaccount, provider "rook-ceph":
        Forbidden: not usable by user or serviceaccount, provider
        "node-exporter": Forbidden: not usable by user or serviceaccount,
        provider "rook-ceph-csi": Forbidden: not usable by user or
        serviceaccount, provider "privileged": Forbidden: not usable by user or
        serviceaccount, provider "velero-privileged": Forbidden: not usable by
        user or serviceaccount]

Expected behavior Previously well documented docs exists.

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context Add any other context about the problem here.

[ravindk89]

We are working on creating an OpenShift specific platform page. Should be done this week.

https://github.com/minio/docs-k8s/blob/main/source/openshift/deploy-minio-on-openshift.rst <- you can find the original instructions here in the meantime.

Although it is odd - @dvaldivia @Alevsk should we support simple kubectl minio init on OpenShift? I can't think of anything that should prevent this, unless OS operates in a very restricted security environment.

[ravindk89]

We used a 302 here on purpose as well - this has been a massive migration. The OpenShift (and in the near future Rancher) docs will be restored and have some proper URLs attributed to them.

Hang in there :)

[ravindk89]

@kaovilai if you use oc minio init instead of kubectl minio init does that help? In internal discussions, it looks like it might be that using oc and logging in to fully authenticate your connection to the cluster might help.

https://docs.openshift.com/container-platform/3.11/cli_reference/differences_oc_kubectl.html <- they should otherwise be similar, but so far auth/security context issues seem to resolve once you dip fully into the OpenShift ecosystem's specific tooling.

[kaovilai]

I am authenticated. Both oc and kubectl are using the same kubeconfig

[kaovilai]

tldr if you require deep system permissions, SCC needs to be declared. https://docs.openshift.com/container-platform/4.11/authentication/managing-security-context-constraints.html

[kaovilai]

logging in to fully authenticate your connection

Not sure what you mean by this.. I am either logged in and can create pods or I am not.

My error clearly shows the pod has been created.

[kaovilai]

The error specifically indicate that minio init command uses

Forbidden: not usable by user or serviceaccount,
        spec.containers[0].securityContext.runAsUser: Invalid value: 1000

If you insist on using this user, you need to declare a security context constraint that is not default.

[kaovilai]

Here is a pod when installing minio operator using operator from OperatorHub on OpenShift.

This pod is valid because runAsUser is within the acceptable range.

kind: Pod
apiVersion: v1
metadata:
  generateName: minio-operator-6d7bc4c996-
  annotations:
    openshift.io/scc: restricted
    operators.operatorframework.io/builder: operator-sdk-v1.23.0
    operators.operatorframework.io/project_layout: unknown
    olm.targetNamespaces: ''
    operatorframework.io/properties: >-
      {"properties":[{"type":"olm.package","value":{"packageName":"minio-operator","version":"4.5.0"}},{"type":"olm.gvk","value":{"group":"minio.min.io","kind":"Tenant","version":"v1"}},{"type":"olm.gvk","value":{"group":"minio.min.io","kind":"Tenant","version":"v2"}}]}
    repository: 'https://github.com/minio/operator'
    k8sMinVersion: '1.18'
    alm-examples: >-
      [{"apiVersion":"minio.min.io/v2","kind":"Tenant","metadata":{"annotations":{"prometheus.io/path":"/minio/v2/metrics/cluster","prometheus.io/port":"9000","prometheus.io/scrape":"true"},"labels":{"app":"minio"},"name":"storage-lite","namespace":"tenant-lite"},"spec":{"certConfig":{},"configuration":{"name":"storage-configuration"},"env":[],"externalCaCertSecret":[],"externalCertSecret":[],"externalClientCertSecrets":[],"features":{"bucketDNS":false,"domains":{}},"image":"quay.io/minio/minio@sha256:3f9b739b3903d6022f8c5309991b199d7006b802a31eef42eeec615005eb765f","imagePullSecret":{},"log":{"affinity":{"nodeAffinity":{},"podAffinity":{},"podAntiAffinity":{}},"annotations":{},"audit":{"diskCapacityGB":1},"db":{"affinity":{"nodeAffinity":{},"podAffinity":{},"podAntiAffinity":{}},"annotations":{},"env":[],"image":"","initimage":"","labels":{},"nodeSelector":{},"resources":{},"securityContext":{"fsGroup":999,"runAsGroup":999,"runAsNonRoot":true,"runAsUser":999},"serviceAccountName":"","tolerations":[],"volumeClaimTemplate":{"metadata":{},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"1Gi"}},"storageClassName":"standard"}}},"env":[],"image":"","labels":{},"nodeSelector":{},"resources":{},"securityContext":{"fsGroup":1000,"runAsGroup":1000,"runAsNonRoot":true,"runAsUser":1000},"serviceAccountName":"","tolerations":[]},"mountPath":"/export","podManagementPolicy":"Parallel","pools":[{"name":"pool-0","servers":4,"volumeClaimTemplate":{"metadata":{"name":"data"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"2Gi"}}}},"volumesPerServer":2}],"priorityClassName":"","prometheus":{"affinity":{"nodeAffinity":{},"podAffinity":{},"podAntiAffinity":{}},"annotations":{},"diskCapacityGB":1,"env":[],"image":"","initimage":"","labels":{},"nodeSelector":{},"resources":{},"securityContext":{"fsGroup":1000,"runAsGroup":1000,"runAsNonRoot":true,"runAsUser":1000},"serviceAccountName":"","sidecarimage":"","storageClassName":"standard"},"requestAutoCert":true,"serviceAccountName":"","serviceMetadata":{"consoleServiceAnnotations":{},"consoleServiceLabels":{},"minioServiceAnnotations":{},"minioServiceLabels":{}},"subPath":"","users":[{"name":"storage-user"}]}}]
    k8s.v1.cni.cncf.io/network-status: |-
      [{
          "name": "openshift-sdn",
          "interface": "eth0",
          "ips": [
              "10.131.0.19"
          ],
          "default": true,
          "dns": {}
      }]
    k8s.v1.cni.cncf.io/networks-status: |-
      [{
          "name": "openshift-sdn",
          "interface": "eth0",
          "ips": [
              "10.131.0.19"
          ],
          "default": true,
          "dns": {}
      }]
    capabilities: Full Lifecycle
    olm.operatorNamespace: openshift-operators
    containerImage: >-
      quay.io/minio/operator@sha256:ff5f7e8cc02f904b1b676e9e2165c1121d9ec20ae08b425f0989c200471cbc89
    olmcahash: fae2bab4b33fe459f1419daedda8dd1f86664ccc4050266b3359dc52b55e16f8
    categories: 'AI/Machine Learning, Big Data, Cloud Provider, Storage'
    description: |-
      MinIO is a Kubernetes-native high performance object store with an
      S3-compatible API. The MinIO Operator supports deploying MinIO Tenants
      onto any Kubernetes.
    olm.operatorGroup: global-operators
  resourceVersion: '381562035'
  name: minio-operator-6d7bc4c996-rrszs
  uid: 69516ab6-edd5-44a6-92e7-aa209d1ae1aa
  creationTimestamp: '2022-09-21T20:05:39Z'
  managedFields:
    - manager: kube-controller-manager
      operation: Update
      apiVersion: v1
      time: '2022-09-21T20:05:39Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            'f:olm.operatorNamespace': {}
            'f:olmcahash': {}
            'f:alm-examples': {}
            'f:description': {}
            'f:olm.operatorGroup': {}
            'f:capabilities': {}
            .: {}
            'f:containerImage': {}
            'f:categories': {}
            'f:operators.operatorframework.io/project_layout': {}
            'f:operatorframework.io/properties': {}
            'f:k8sMinVersion': {}
            'f:operators.operatorframework.io/builder': {}
            'f:olm.targetNamespaces': {}
            'f:repository': {}
          'f:generateName': {}
          'f:labels':
            .: {}
            'f:name': {}
            'f:pod-template-hash': {}
          'f:ownerReferences':
            .: {}
            'k:{"uid":"5cd4e279-1db4-46d6-b4ed-2a0843087397"}': {}
        'f:spec':
          'f:volumes':
            .: {}
            'k:{"name":"apiservice-cert"}':
              .: {}
              'f:name': {}
              'f:secret':
                .: {}
                'f:defaultMode': {}
                'f:items': {}
                'f:secretName': {}
            'k:{"name":"webhook-cert"}':
              .: {}
              'f:name': {}
              'f:secret':
                .: {}
                'f:defaultMode': {}
                'f:items': {}
                'f:secretName': {}
          'f:containers':
            'k:{"name":"minio-operator"}':
              'f:image': {}
              'f:volumeMounts':
                .: {}
                'k:{"mountPath":"/apiserver.local.config/certificates"}':
                  .: {}
                  'f:mountPath': {}
                  'f:name': {}
                'k:{"mountPath":"/tmp/k8s-webhook-server/serving-certs"}':
                  .: {}
                  'f:mountPath': {}
                  'f:name': {}
              'f:terminationMessagePolicy': {}
              .: {}
              'f:resources':
                .: {}
                'f:requests':
                  .: {}
                  'f:cpu': {}
                  'f:ephemeral-storage': {}
                  'f:memory': {}
              'f:env':
                .: {}
                'k:{"name":"OPERATOR_CONDITION_NAME"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
              'f:securityContext':
                .: {}
                'f:runAsNonRoot': {}
              'f:terminationMessagePath': {}
              'f:imagePullPolicy': {}
              'f:name': {}
          'f:dnsPolicy': {}
          'f:serviceAccount': {}
          'f:restartPolicy': {}
          'f:schedulerName': {}
          'f:terminationGracePeriodSeconds': {}
          'f:serviceAccountName': {}
          'f:enableServiceLinks': {}
          'f:securityContext': {}
          'f:affinity':
            .: {}
            'f:podAntiAffinity':
              .: {}
              'f:requiredDuringSchedulingIgnoredDuringExecution': {}
    - manager: multus
      operation: Update
      apiVersion: v1
      time: '2022-09-21T20:05:41Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            'f:k8s.v1.cni.cncf.io/network-status': {}
            'f:k8s.v1.cni.cncf.io/networks-status': {}
      subresource: status
    - manager: Go-http-client
      operation: Update
      apiVersion: v1
      time: '2022-09-21T20:05:43Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:status':
          'f:conditions':
            'k:{"type":"ContainersReady"}':
              .: {}
              'f:lastProbeTime': {}
              'f:lastTransitionTime': {}
              'f:status': {}
              'f:type': {}
            'k:{"type":"Initialized"}':
              .: {}
              'f:lastProbeTime': {}
              'f:lastTransitionTime': {}
              'f:status': {}
              'f:type': {}
            'k:{"type":"Ready"}':
              .: {}
              'f:lastProbeTime': {}
              'f:lastTransitionTime': {}
              'f:status': {}
              'f:type': {}
          'f:containerStatuses': {}
          'f:hostIP': {}
          'f:phase': {}
          'f:podIP': {}
          'f:podIPs':
            .: {}
            'k:{"ip":"10.131.0.19"}':
              .: {}
              'f:ip': {}
          'f:startTime': {}
      subresource: status
    - manager: minio-operator
      operation: Update
      apiVersion: v1
      time: '2022-09-21T20:06:47Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:labels':
            'f:operator': {}
  namespace: openshift-operators
  ownerReferences:
    - apiVersion: apps/v1
      kind: ReplicaSet
      name: minio-operator-6d7bc4c996
      uid: 5cd4e279-1db4-46d6-b4ed-2a0843087397
      controller: true
      blockOwnerDeletion: true
  labels:
    name: minio-operator
    operator: leader
    pod-template-hash: 6d7bc4c996
spec:
  restartPolicy: Always
  serviceAccountName: minio-operator
  imagePullSecrets:
    - name: minio-operator-dockercfg-bhzcs
  priority: 0
  schedulerName: default-scheduler
  enableServiceLinks: true
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
              - key: name
                operator: In
                values:
                  - minio-operator
          topologyKey: kubernetes.io/hostname
  terminationGracePeriodSeconds: 30
  preemptionPolicy: PreemptLowerPriority
  nodeName: ip-10-0-173-131.ec2.internal
  securityContext:
    seLinuxOptions:
      level: 's0:c20,c10'
    fsGroup: 1000400000
  containers:
    - resources:
        requests:
          cpu: 200m
          ephemeral-storage: 500Mi
          memory: 256Mi
      terminationMessagePath: /dev/termination-log
      name: minio-operator
      env:
        - name: OPERATOR_CONDITION_NAME
          value: minio-operator.v4.5.0
      securityContext:
        capabilities:
          drop:
            - KILL
            - MKNOD
            - SETGID
            - SETUID
        runAsUser: 1000400000
        runAsNonRoot: true
      imagePullPolicy: IfNotPresent
      volumeMounts:
        - name: apiservice-cert
          mountPath: /apiserver.local.config/certificates
        - name: webhook-cert
          mountPath: /tmp/k8s-webhook-server/serving-certs
        - name: kube-api-access-59wsx
          readOnly: true
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      terminationMessagePolicy: File
      image: >-
        quay.io/minio/operator@sha256:ff5f7e8cc02f904b1b676e9e2165c1121d9ec20ae08b425f0989c200471cbc89
  serviceAccount: minio-operator
  volumes:
    - name: apiservice-cert
      secret:
        secretName: minio-operator-service-cert
        items:
          - key: tls.crt
            path: apiserver.crt
          - key: tls.key
            path: apiserver.key
        defaultMode: 420
    - name: webhook-cert
      secret:
        secretName: minio-operator-service-cert
        items:
          - key: tls.crt
            path: tls.crt
          - key: tls.key
            path: tls.key
        defaultMode: 420
    - name: kube-api-access-59wsx
      projected:
        sources:
          - serviceAccountToken:
              expirationSeconds: 3607
              path: token
          - configMap:
              name: kube-root-ca.crt
              items:
                - key: ca.crt
                  path: ca.crt
          - downwardAPI:
              items:
                - path: namespace
                  fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.namespace
          - configMap:
              name: openshift-service-ca.crt
              items:
                - key: service-ca.crt
                  path: service-ca.crt
        defaultMode: 420
  dnsPolicy: ClusterFirst
  tolerations:
    - key: node.kubernetes.io/not-ready
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
    - key: node.kubernetes.io/unreachable
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
    - key: node.kubernetes.io/memory-pressure
      operator: Exists
      effect: NoSchedule
status:
  phase: Running
  conditions:
    - type: Initialized
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2022-09-21T20:05:39Z'
    - type: Ready
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2022-09-21T20:05:43Z'
    - type: ContainersReady
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2022-09-21T20:05:43Z'
    - type: PodScheduled
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2022-09-21T20:05:39Z'
  hostIP: 10.0.173.131
  podIP: 10.131.0.19
  podIPs:
    - ip: 10.131.0.19
  startTime: '2022-09-21T20:05:39Z'
  containerStatuses:
    - restartCount: 0
      started: true
      ready: true
      name: minio-operator
      state:
        running:
          startedAt: '2022-09-21T20:05:42Z'
      imageID: >-
        quay.io/minio/operator@sha256:436b6afa74dc91664123bf6f79e9f9c227b6312f682c6f091a946a11d7e72583
      image: >-
        quay.io/minio/operator@sha256:ff5f7e8cc02f904b1b676e9e2165c1121d9ec20ae08b425f0989c200471cbc89
      lastState: {}
      containerID: 'cri-o://70a6f063ff371113a2fb5744fdedf0a6464e182d5a7c501893a3dff0a4dddce8'
  qosClass: Burstable

[kaovilai]

tldr: do this for kubectl minio init pods on OpenShift.

      securityContext:
        capabilities:
          drop:
            - KILL
            - MKNOD
            - SETGID
            - SETUID
        runAsUser: 1000400000
        runAsNonRoot: true

[kaovilai]

I just ran the diff between the two deployments. One created by OperatorHub operator and one by kubectl minio init

The difference thing that made minio deployment from kubectl minio init work after looking at the diff is to do the following.

          securityContext:
-            runAsUser: 1000
-            runAsGroup: 1000

This has been documented in https://github.com/minio/operator/blob/7dfb6adcd5b7befe915b35af91c48368c503599d/olm.sh#L74-L77

Clearing prior outdated comments. @ravindk89

[ravindk89]

Thanks for investigating @kaovilai - mostly my comment around auth is we've seen some weird behavior there before. SecurityContexts as well, for that matter.

I'm not sure we will keep the kubectl|oc minio init path in our official docs for OpenShift. We've seen substantial confusion around setting SCC from a number of users, so anything we can do to avoid making that part of the critical path of setup is of interest.

That is to say, once we publish the new platform docs for OpenShift, Operator installation will be either via MarketPlace or OperatorHub.

[kaovilai]

Is there a reason runAsUser is specified or required in the kubectl minio init case? if not, I would suggest removing those two lines from all the pods created by kubectl minio so at least it has higher compatibility.

[ravindk89]

There is likely - @Alevsk or @dvaldivia might know the specifics. As noted, SecurityContexts have been a consistent item we've had to address and re-evaluate repeatedly over multiple releases.

[ravindk89]

Fixed by #577

https://min.io/docs/minio/kubernetes/openshift/index.html

Redirects should be working as well.

Further work will be required to completely customize the docs for OpenSHift (e.g. kubectl -> oc for CLI tutorials) but this is at least parity.

[ravindk89]

@cniackz is looking into further optimizing our default securityConstraints, w.r.t. your comment @kaovilai

[kaovilai]

@ravindk89 After reading more. https://cloud.redhat.com/blog/a-guide-to-openshift-and-uids and inspecting default ranges on openshift. I would suggest that minio cli to check for the valid ranges in the install namespace. If the annotation is found such as here, uid needs to be 1000000000.

The valid range do not overlap with another namespace so you will need to do check dynamically each time cli is run.

kind: Project
apiVersion: project.openshift.io/v1
metadata:
  name: default
  uid: 8ba1907f-7a8f-44e3-b969-74f855d6a9db
  resourceVersion: '322577690'
  creationTimestamp: '2022-04-25T14:57:26Z'
  labels:
    kubernetes.io/metadata.name: default
    olm.operatorgroup.uid/74361eb0-5ea9-4c1d-a28a-2c8e9d4aecb0: ''
  annotations:
    openshift.io/sa.scc.mcs: 's0:c1,c0'
    openshift.io/sa.scc.supplemental-groups: 1000000000/10000
    openshift.io/sa.scc.uid-range: 1000000000/10000