VictoriaMetrics / operator

Kubernetes operator for Victoria Metrics
Apache License 2.0
436 stars 146 forks source link

vmsingle cannot remove pvc #1158

Open whc9527 opened 1 week ago

whc9527 commented 1 week ago

install command: helm upgrade --install vmks vm/victoria-metrics-k8s-stack -f values.yaml -n monitoring

values.yaml

... ...
# Configures vmsingle params
vmsingle:
  # -- VMSingle annotations
  annotations: {}
  # -- Create VMSingle CR
  enabled: true
  # -- Full spec for VMSingle CRD. Allowed values describe [here](https://docs.victoriametrics.com/operator/api#vmsinglespec)
  spec:
    port: "8429"
    # -- Data retention period. Possible units character: h(ours), d(ays), w(eeks), y(ears), if no unit character specified - month. The minimum retention period is 24h. See these [docs](https://docs.victoriametrics.com/single-server-victoriametrics/#retention)
    retentionPeriod: "1h"
    replicaCount: 1
    extraArgs: {}
    nodeSelector: 
      business_type: prometheus
    resources: {}
    storage: {}
    storageDataPath: "/vmdata"
... ...

Combined with the code below operator, vmsingle should not have PVC dependencies after installation

    if cr.Spec.Storage != nil && cr.Spec.StorageDataPath == "" {
        if err := createVMSingleStorage(ctx, cr, rclient); err != nil {
            return fmt.Errorf("cannot create storage: %w", err)
        }
    }

https://github.com/VictoriaMetrics/operator/blob/v0.49.0/internal/controller/operator/factory/vmsingle/vmsingle.go#L72

But after the installation, it was found that vmsingle has always been PVC dependent, which is why?

get the pod yaml of vmsingle k -n monitoring get pod vmsingle-vmks-victoria-metrics-k8s-stack-6584f64f97-qp6qs -o yaml

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2024-11-14T10:40:40Z"
  generateName: vmsingle-vmks-victoria-metrics-k8s-stack-6584f64f97-
  labels:
    app.kubernetes.io/component: monitoring
    app.kubernetes.io/instance: vmks-victoria-metrics-k8s-stack
    app.kubernetes.io/name: vmsingle
    managed-by: vm-operator
    pod-template-hash: 6584f64f97
  name: vmsingle-vmks-victoria-metrics-k8s-stack-6584f64f97-qp6qs
  namespace: monitoring
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: vmsingle-vmks-victoria-metrics-k8s-stack-6584f64f97
    uid: e2b6ed73-d36f-493f-88be-3ea08293d0f0
  resourceVersion: "2327881791"
  uid: f73c7f52-7f66-4722-bafa-4c7b60cdb8be
spec:
  containers:
  - args:
    - -httpListenAddr=:8429
    - -retentionPeriod=1h
    - -storageDataPath=/victoria-metrics-data
    image: victoriametrics/victoria-metrics:v1.106.0
    imagePullPolicy: IfNotPresent
    name: vmsingle
    ports:
    - containerPort: 8429
      name: http
      protocol: TCP
    readinessProbe:
      failureThreshold: 10
      httpGet:
        path: /health
        port: 8429
        scheme: HTTP
      periodSeconds: 5
      successThreshold: 1
      timeoutSeconds: 5
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: FallbackToLogsOnError
    volumeMounts:
    - mountPath: /victoria-metrics-data
      name: data
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-sxgjz
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeSelector:
    business_type: prometheus
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: vmsingle-vmks-victoria-metrics-k8s-stack
  serviceAccountName: vmsingle-vmks-victoria-metrics-k8s-stack
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: vmsingle-vmks-victoria-metrics-k8s-stack
  - name: kube-api-access-sxgjz
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2024-11-14T10:40:40Z"
    message: '0/1026 nodes are available: pod has unbound immediate PersistentVolumeClaims.
      , preemption: 0/1026 nodes are available: 1026 No preemption victims found for
      incoming pod..'
    reason: Unschedulable
    status: "False"
    type: PodScheduled
  phase: Pending
  qosClass: BestEffort
f41gh7 commented 3 days ago

Hello, thanks for reporting. Looks like bug to me.

f41gh7 commented 2 days ago

Issue will be fixed at the next release. But provided configuration example is not valid and operator will reject it with validation error.

Proper configuration must have defined volumes and volumeMounts for defined storageDataPath:

# Configures vmsingle params
vmsingle:
  enabled: true
  spec:
    retentionPeriod: "1h"
    replicaCount: 1
    nodeSelector: 
      business_type: prometheus
    storageDataPath: "/vmdata"
    volumes:
    - name: storage
      # replace with actual volume source
      emptyDir: {}
    volumeMounts:
   - name: storage
     mountPath: "/vmdata"