search

fluxcd / flux2-monitoring-example

Prometheus monitoring for the Flux control plane
https://fluxcd.io/flux/monitoring/
Apache License 2.0
52 stars 145 forks source link

HelmRepository with OCI urls have no status #29

Closed defenestration closed 3 months ago

defenestration commented 8 months ago

We updated flux 2.1.x to flux 2.2.3. Some of the kube-state-metrics alerts we had started triggering. I was able to make some changes for notification resources but we have some OCI helm charts in use that have no status reported in the kubernetes object, giving kube-state-metrics nothing to scrape a status from apparently.

Other helm repositories using https are fine, and have a fully defined status: field.

❯ flux get sources helm -A
NAMESPACE           NAME                                REVISION        SUSPENDED   READY   MESSAGE
...
flux-system         azure-oci                                           False       True    Helm repository is Ready
flux-system         ww-gitops                                           False       True    Helm repository is Ready
ingress-nginx       ingress-nginx                       sha256:9c0f8ad3 False       True    stored artifact: revision 'sha256:9c0f8ad3'

We use pretty much the same kube-metrics-config as here https://github.com/fluxcd/flux2-monitoring-example/blob/main/monitoring/controllers/kube-prometheus-stack/kube-state-metrics-config.yaml

Heres a helm repo we use for the weaveworks dashboard as a public example. Note that status field is empty which seems to be the problem. 

❯ k get helmrepository -n flux-system ww-gitops -o yaml
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
  annotations:
    metadata.weave.works/description: This is the source location for the Weave GitOps
      Dashboard's helm chart.
  creationTimestamp: "2023-06-22T13:12:06Z"
  generation: 1
  labels:
    app.kubernetes.io/component: ui
    app.kubernetes.io/created-by: weave-gitops-cli
    app.kubernetes.io/name: weave-gitops-dashboard
    app.kubernetes.io/part-of: weave-gitops
    kustomize.toolkit.fluxcd.io/name: infrastructure
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  name: ww-gitops
  namespace: flux-system
  resourceVersion: "473650341"
  uid: 2be26781-703e-4ef2-afe7-bec1072cba83
spec:
  interval: 1h0m0s
  provider: generic
  timeout: 60s
  type: oci
  url: oci://ghcr.io/weaveworks/charts
status: {}

Would appreciate some advise here. Probably going to be disabling the HelmRepository monitoring for now as a workaround.

darkowlzz commented 8 months ago

Hi, objects that don't have any status are assumed to be ready as they have nothing to do. Their existence can be assumed as they are ready. In this case, HelmRepository of type OCI are static objects. They are not reconciled any more. https://github.com/fluxcd/flux2-monitoring-example/pull/9 may be relevant here, at least the PR description. I believe that the corresponding grafana panel for readiness of objects is configured to show not ready only when the status says Ready=False explicitly. Also, refer https://github.com/fluxcd/source-controller/issues/1249 for more details about this change.