[victoria-metrics-k8s-stack] argocd - OutOfSync (This resource is not present in the application's source. It will be deleted from Kubernetes if the prune option is enabled during sync.)

[AurimasNav]

As per picture below, pvc, pod and controllerrevision are being deleted and re-created in an infinite loop.

argocd version: 2.12.2 k3s version: v1.30.4+k3s1

helm via kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources: []
namespace: victoriametrics

helmGlobals:
  chartHome: charts
helmCharts:
- name: victoria-metrics-k8s-stack
  version: 0.25.3
  repo: https://victoriametrics.github.io/helm-charts
  releaseName: victoria-metrics-k8s-stack
  includeCRDs: true
  namespace: victoriametrics
  valuesFile: values.yaml

application.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: victoria-metrics-k8s-stack
  namespace: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    server: https://kubernetes.default.svc
    namespace: victoriametrics
  project: default
  source:
    path: gitops/victoria-metrics-k8s-stack
    repoURL: https://github.com/AurimasNav/k8s-at-home.git
    targetRevision: main
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true
    - RespectIgnoreDifferences=true
  ignoreDifferences:
    - group: ""
      kind: Secret
      name: victoria-metrics-k8s-stack-victoria-metrics-operator-validation
      namespace: kube-system
      jsonPointers:
        - /data
    - group: admissionregistration.k8s.io
      kind: ValidatingWebhookConfiguration
      name: victoria-metrics-k8s-stack-victoria-metrics-operator-admission
      jqPathExpressions:
      - '.webhooks[]?.clientConfig.caBundle'

[AndrewChubatiuk]

hey @AurimasNav ControllerRevision for which service is it?

[AurimasNav]

hey @AurimasNav ControllerRevision for which service is it?

it is named vmalertmanager-victoria-metrics-k8s-stack-c9f494cf7, so I guess for vmalert manager?

I should have also mentioned that I am using default values for install.

[AndrewChubatiuk]

looks like it's an operator issue, fix is expected to be released today

[AurimasNav]

I have patched operator deployment with image 0.47.2 it seems to have fixed pod and controllerrevision, but the pvc is still affected.

[AndrewChubatiuk]

alertmanager with default configuration in k8s stack uses emptydir storage. does this happen when you recreate alertmanager?

[f41gh7]

Looks like an issue still exists for kubernetes clusters < 1.27. There will be patch release soon

[f41gh7]

Also, I was not able to reproduce any problem with PVC

[AurimasNav]

alertmanager with default configuration in k8s stack uses emptydir storage. does this happen when you recreate alertmanager?

I have completely deleted my application and redeployed it with latest version 0.25.5, the pvc is still being created (by vm-operator?) and argocd tries to delete it as it is not in source.

these are the exact files I am using to deploy this: argocd application - https://github.com/AurimasNav/k8s-at-home/tree/main/argocd-apps/base/apps/victoria-metrics-k8s-stack helm via kustomize - https://github.com/AurimasNav/k8s-at-home/tree/main/gitops/victoria-metrics-k8s-stack

[AndrewChubatiuk]

hey @AurimasNav was able to reproduce this issue and transfered it to operator repo, as it's related to how it manages pvc, which is managed by VMSingle

[f41gh7]

Fix will be included to the next release.