search

loft-sh / vcluster

vCluster - Create fully functional virtual Kubernetes clusters - Each vcluster runs inside a namespace of the underlying k8s cluster. It's cheaper than creating separate full-blown clusters and it offers better multi-tenancy and isolation than regular namespaces.
https://www.vcluster.com
Apache License 2.0
6.26k stars 398 forks source link

Bitnami Harbor registry pods sync fails #394

Closed olljanat closed 2 years ago

olljanat commented 2 years ago

What happened?

I'm trying to deploy https://github.com/bitnami/charts/tree/master/bitnami/harbor to vcluster but noticed that registry pods sync fails to errors like these (looks that it also variate a bit on every run:

$ kubectl -n one-harbor describe pods harbor-registry-...
Events:
  Type     Reason     Age                From        Message
  ----     ------     ----               ----        -------
  Warning  SyncError  18s                pod-syncer  Error syncing to physical cluster: Pod "harbor-registry-..." is invalid: spec.containers[1].name: Duplicate value: "registryctl"
  Warning  SyncError  17s (x2 over 18s)  pod-syncer  Error syncing to physical cluster: Internal error occurred: error in remove for path: '/spec/containers/0/volumeMounts/2': Unable to access invalid index: 2: invalid index referenced
  Warning  SyncError  6s (x9 over 18s)   pod-syncer  Error syncing to physical cluster: Internal error occurred: error in remove for path: '/spec/containers/0/ports/0': Unable to access invalid index: 0: invalid index referenced

Events:
  Type     Reason                  Age                From                     Message
  ----     ------                  ----               ----                     -------
  Warning  SyncError               83s                pod-syncer               Error syncing to physical cluster: Internal error occurred: error in remove for path: '/spec/containers/0/volumeMounts/2': Unable to access invalid index: 2: invalid index referenced

Events:
  Type     Reason                  Age                From                     Message
  ----     ------                  ----               ----                     -------
  Warning  SyncError               54s                pod-syncer               Error syncing to physical cluster: Internal error occurred: error in remove for path: '/spec/containers/0/ports/0': Unable to access invalid index: 0: invalid index referenced
  Warning  SyncError               54s                pod-syncer               Error syncing to physical cluster: Internal error occurred: error in remove for path: '/spec/containers/0/volumeMounts/2': Unable to access invalid index: 2: invalid index referenced  

Events:
  Type     Reason     Age                From        Message
  ----     ------     ----               ----        -------
  Warning  SyncError  30s (x3 over 31s)  pod-syncer  Error syncing to physical cluster: Internal error occurred: error in remove for path: '/spec/containers/0/volumeMounts/2': Unable to access invalid index: 2: invalid index referenced
  Warning  SyncError  19s (x9 over 31s)  pod-syncer  Error syncing to physical cluster: Internal error occurred: error in remove for path: '/spec/containers/0/ports/0': Unable to access invalid index: 0: invalid index referenced
  Warning  SyncError  8s                 pod-syncer  Error syncing to physical cluster: Pod "harbor-registry-..." is invalid: spec.containers[1].name: Duplicate value: "registry"

What did you expect to happen?

Sync should works

How can we reproduce it (as minimally and precisely as possible)?

Here is exported deployment file which contains only problematic registry part:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    meta.helm.sh/release-name: harbor
    meta.helm.sh/release-namespace: harbor
  labels:
    app.kubernetes.io/component: registry
    app.kubernetes.io/instance: harbor
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: harbor
    helm.sh/chart: harbor-11.2.4
  name: harbor-registry
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/component: registry
      app.kubernetes.io/instance: harbor
      app.kubernetes.io/name: harbor
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        checksum/configmap: 2ba0fb15273f0f3a2a5ede7ec4b76257a0af061c9016c128ed06d92274c12076
        checksum/secret: 8099af1972adde29df27088a65816a8d7e7cea619f287b072bbe9ac6c086e6cb
        checksum/secret-core: e57abb7c9179bdd83c3d11c03d636f563dbbd17fd03dc72732aaf33197d7d3df
        checksum/secret-jobservice: 94889bed02e230e7da2c303bf7901140020fd2154a03de431130ecfc809370e2
      creationTimestamp: null
      labels:
        app.kubernetes.io/component: registry
        app.kubernetes.io/instance: harbor
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: harbor
        helm.sh/chart: harbor-11.2.4
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchLabels:
                  app.kubernetes.io/component: registry
                  app.kubernetes.io/instance: harbor
                  app.kubernetes.io/name: harbor
              namespaces:
              - harbor
              topologyKey: kubernetes.io/hostname
            weight: 1
      automountServiceAccountToken: false
      containers:
      - env:
        - name: BITNAMI_DEBUG
          value: "false"
        envFrom:
        - secretRef:
            name: harbor-registry
        image: docker.io/bitnami/harbor-registry:2.4.1-debian-10-r58
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 6
          httpGet:
            path: /
            port: registry
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        name: registry
        ports:
        - containerPort: 5000
          name: registry
          protocol: TCP
        - containerPort: 5001
          name: debug
          protocol: TCP
        readinessProbe:
          failureThreshold: 6
          httpGet:
            path: /
            port: registry
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        resources: {}
        securityContext:
          runAsNonRoot: true
          runAsUser: 1001
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /storage
          name: registry-data
        - mountPath: /etc/registry/root.crt
          name: registry-root-certificate
          subPath: tls.crt
        - mountPath: /etc/registry/passwd
          name: registry-htpasswd
          subPath: passwd
        - mountPath: /etc/registry/config.yml
          name: registry-config
          subPath: config.yml
      - env:
        - name: BITNAMI_DEBUG
          value: "false"
        - name: CORE_SECRET
          valueFrom:
            secretKeyRef:
              key: secret
              name: harbor-core
        - name: JOBSERVICE_SECRET
          valueFrom:
            secretKeyRef:
              key: secret
              name: harbor-jobservice
        envFrom:
        - secretRef:
            name: harbor-registry
        image: docker.io/bitnami/harbor-registryctl:2.4.1-debian-10-r59
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 6
          httpGet:
            path: /api/health
            port: registryctl
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        name: registryctl
        ports:
        - containerPort: 8080
          name: registryctl
          protocol: TCP
        readinessProbe:
          failureThreshold: 6
          httpGet:
            path: /api/health
            port: registryctl
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        resources: {}
        securityContext:
          runAsNonRoot: true
          runAsUser: 1001
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /storage
          name: registry-data
        - mountPath: /etc/registry/config.yml
          name: registry-config
          subPath: config.yml
        - mountPath: /etc/registryctl/config.yml
          name: registry-config
          subPath: ctl-config.yml
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 1001
      terminationGracePeriodSeconds: 30
      volumes:
      - name: registry-htpasswd
        secret:
          defaultMode: 420
          items:
          - key: REGISTRY_HTPASSWD
            path: passwd
          secretName: harbor-registry
      - name: registry-root-certificate
        secret:
          defaultMode: 420
          secretName: harbor-core
      - configMap:
          defaultMode: 420
          name: harbor-registry
        name: registry-config
      - name: registry-data
        persistentVolumeClaim:
          claimName: harbor-registry

Anything else we need to know?

No response

Host cluster Kubernetes version

```console $ kubectl version Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.0", GitCommit:"ab69524f795c42094a6630298ff53f3c3ebab7f4", GitTreeState:"clean", BuildDate:"2021-12-07T18:16:20Z", GoVersion:"go1.17.3", Compiler:"gc", Platform:"windows/amd64"} Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.6", GitCommit:"f59f5c2fda36e4036b49ec027e556a15456108f0", GitTreeState:"clean", BuildDate:"2022-01-19T17:26:47Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"linux/amd64"} ```

Host cluster Kubernetes distribution

``` v1.22.7+rke2r1 ```

vlcuster version

```console $ vcluster --version vcluster version 0.6.0 ```

Vcluster Kubernetes distribution(k3s(default)), k8s, k0s)

``` k8s ```

OS and Arch

``` OS: Ubuntu 20.04 Arch: amd64 ```
FabianKramm commented 2 years ago

@olljanat thanks for creating this issue! The provided deployment works for us, it looks like the error originates from an admission controller that tries to modify the synced pod, are you using any mutating admission controllers in the host cluster?

olljanat commented 2 years ago

Yes we are using mutating admission controllers but not on that area. However looks that you are rights and this is more likely bug on Kyverno which is not able to handle that case.

Thanks for you help.