search

weaveworks / weave-gitops-enterprise

This repo provides the enterprise level features for the weave-gitops product, including CAPI cluster creation and team workspaces.
https://docs.gitops.weave.works/
Apache License 2.0
160 stars 29 forks source link

Implement data collection and storage #2442

Closed jpellizzari closed 1 year ago

jpellizzari commented 1 year ago

From this part of the rfc: https://github.com/weaveworks/weave-gitops-private/tree/main/docs/rfcs/0004-query-scaling#data-collection

To collect data from each of the leaf clusters, a scheduled process (known as the Collector module) will poll each of the clusters via the Kubernetes API. This process will use the credentials specified in the GitOpsCluster object on the management cluster to list namespaces and objects on the leaf cluster and populate the central store.

The RFC specifies Redis as a good candidate, so lets start there.

Other notes:

enekofb commented 1 year ago

made progress, can watch helm releases

Feature: can watch kubernetes cluster resource by kind
  In order to index kubernetes resources by kind
  As collector
  I want to watch kubernetes resources by kind
=== CONT  TestFeatures
    store.go:123: "level"=0 "msg"="db path" "/var/folders/9b/bkrspzws5xgd7x_ldtc880pr0000gn/T/db1065723458/charts.db"="<no-value>"
    store.go:132: "level"=0 "msg"="db created"
    store.go:138: "level"=0 "msg"="schema created"
    main_test.go:88: "level"=0 "msg"="created inmemory store"
    main_test.go:97: "level"=0 "msg"="created clusters watcher"

  Background:
    Given a clusters watcher                               # main_test.go:80 -> github.com/enekofb/collector.aClustersWatcher
    main_test.go:104: "level"=0 "msg"="to watch:" "helm.toolkit.fluxcd.io/v2beta1"="<no-value>"

  Scenario: can watch helm release                         # features/can-watch-kind.feature:9
    Given a kind "helm.toolkit.fluxcd.io/v2beta1" to watch # main_test.go:103 -> github.com/enekofb/collector.aKindToWatch
    main_test.go:231: "level"=0 "msg"="environment started"
    main_test.go:121: "level"=0 "msg"="kube environment created:" "https://127.0.0.1:55263"="<no-value>"
    main_test.go:141: "level"=0 "msg"="kube client created"
    main_test.go:149: "level"=0 "msg"="number of resources found:" "<non-string-key: 1>"="<no-value>"
    And a kubernetes cluster with resources of that kind   # main_test.go:114 -> github.com/enekofb/collector.aKubernetesClusterWithResourcesOfThatKind
    cluster.go:83: "level"=0 "msg"="adding cluster" "<non-string-key: \"default/https:/>"="<no-value>"
    watcher.go:97: "level"=0 "msg"="using default manager function"
    cluster.go:89: "level"=0 "msg"="watcher created"
    main_test.go:169: "level"=0 "msg"="watcher added to cluster:" "<non-string-key: \"default/https:/>"="<no-value>"
    main_test.go:177: "level"=0 "msg"="waiting for started status:" "stopped"="<no-value>"
    watcher.go:81: watcher: "level"=0 "msg"="created default controller manager"
    watcher.go:159: watcher: "level"=0 "msg"="watcher manager created"
    helm_release.go:34: watcher: "level"=0 "msg"="creating helm reconciler"
    watcher.go:177: watcher: "level"=0 "msg"="watcher with helm reconciler started"
    cluster.go:95: "level"=0 "msg"="watcher started"
    controller.go:186: watcher: "level"=0 "msg"="Starting EventSource" "controller"="helmrelease" "controllerGroup"="helm.toolkit.fluxcd.io" "controllerKind"="HelmRelease" "source"="kind source: *v2beta1.HelmRelease"
    controller.go:194: watcher: "level"=0 "msg"="Starting Controller" "controller"="helmrelease" "controllerGroup"="helm.toolkit.fluxcd.io" "controllerKind"="HelmRelease"
    controller.go:228: watcher: "level"=0 "msg"="Starting workers" "controller"="helmrelease" "controllerGroup"="helm.toolkit.fluxcd.io" "controllerKind"="HelmRelease" "worker count"=1
    helm_release.go:106: watcher: "level"=0 "msg"="stored helm release document with id" "controller"="helmrelease" "controllerGroup"="helm.toolkit.fluxcd.io" "controllerKind"="HelmRelease" "HelmRelease"={"name":"podinfo","namespace":"default"} "namespace"="default" "name"="podinfo" "reconcileID"="84c08111-46f3-409d-a6c5-0ed10d6dc83e" "helmRelease"={"kind":"HelmRelease","apiVersion":"helm.toolkit.fluxcd.io/v2beta1","metadata":{"name":"podinfo","namespace":"default","uid":"0fe6feb1-14fa-4567-8a1b-5e47ca762576","resourceVersion":"702","generation":1,"creationTimestamp":"2023-03-06 12:36:07 +0000 GMT","annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"helm.toolkit.fluxcd.io/v2beta1\",\"kind\":\"HelmRelease\",\"metadata\":{\"annotations\":{},\"name\":\"podinfo\",\"namespace\":\"default\"},\"spec\":{\"chart\":{\"spec\":{\"chart\":\"podinfo\",\"sourceRef\":{\"apiVersion\":\"source.toolkit.fluxcd.io/v1beta2\",\"kind\":\"HelmRepository\",\"name\":\"app-charts\",\"namespace\":\"default\"},\"version\":\"0.0.15\"}},\"interval\":\"10m0s\",\"values\":{}}}\n"},"finalizers":["finalizers.fluxcd.io"],"managedFields":[{"manager":"helm-controller","operation":"Update","apiVersion":"helm.toolkit.fluxcd.io/v2beta1","time":"2023-03-06 12:36:07 +0000 GMT","fieldsType":"FieldsV1","fieldsV1":"{\"f:metadata\":{\"f:finalizers\":{\".\":{},\"v:\\\"finalizers.fluxcd.io\\\"\":{}}}}"},{"manager":"kubectl-client-side-apply","operation":"Update","apiVersion":"helm.toolkit.fluxcd.io/v2beta1","time":"2023-03-06 12:36:07 +0000 GMT","fieldsType":"FieldsV1","fieldsV1":"{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:kubectl.kubernetes.io/last-applied-configuration\":{}}},\"f:spec\":{\".\":{},\"f:chart\":{\".\":{},\"f:spec\":{\".\":{},\"f:chart\":{},\"f:reconcileStrategy\":{},\"f:sourceRef\":{\".\":{},\"f:apiVersion\":{},\"f:kind\":{},\"f:name\":{},\"f:namespace\":{}},\"f:version\":{}}},\"f:interval\":{},\"f:values\":{}}}"},{"manager":"helm-controller","operation":"Update","apiVersion":"helm.toolkit.fluxcd.io/v2beta1","time":"2023-03-06 12:36:09 +0000 GMT","fieldsType":"FieldsV1","fieldsV1":"{\"f:status\":{\"f:conditions\":{},\"f:helmChart\":{},\"f:lastAppliedRevision\":{},\"f:lastAttemptedRevision\":{},\"f:lastAttemptedValuesChecksum\":{},\"f:lastReleaseRevision\":{},\"f:observedGeneration\":{}}}","subresource":"status"}]},"spec":{"chart":{"spec":{"chart":"podinfo","version":"0.0.15","sourceRef":{"apiVersion":"source.toolkit.fluxcd.io/v1beta2","kind":"HelmRepository","name":"app-charts","namespace":"default"},"reconcileStrategy":"ChartVersion"}},"interval":{"Duration":"10m0s"},"values":"&JSON{Raw:*[123 125],}"},"status":{"observedGeneration":1,,"conditions":[{"type":"Ready","status":"True","lastTransitionTime":"2023-03-06 12:36:09 +0000 GMT","reason":"ReconciliationSucceeded","message":"Release reconciliation succeeded"},{"type":"Released","status":"True","lastTransitionTime":"2023-03-06 12:36:09 +0000 GMT","reason":"InstallSucceeded","message":"Helm install succeeded"}],"lastAppliedRevision":"0.0.15","lastAttemptedRevision":"0.0.15","lastAttemptedValuesChecksum":"da39a3ee5e6b4b0d3255bfef95601890afd80709","lastReleaseRevision":1,"helmChart":"default/default-podinfo"}} "<non-string-key: 1>"="<no-value>"
    main_test.go:177: "level"=0 "msg"="waiting for started status:" "started"="<no-value>"
    main_test.go:186: "level"=0 "msg"="watcher has started"
    When watched the kind in the cluster                   # main_test.go:158 -> github.com/enekofb/collector.watchedTheKindInTheCluster
    main_test.go:195: "level"=0 "msg"="expected num docs: '1'"
    main_test.go:203: "level"=0 "msg"="found num docs: '1' "
    Then i got all the results                             # main_test.go:192 -> github.com/enekofb/collector.iGotAllTheResults

1 scenarios (1 passed)
5 steps (5 passed)
2.838522s
--- PASS: TestFeatures (2.84s)
=== RUN   TestFeatures/can_watch_helm_release
    --- PASS: TestFeatures/can_watch_helm_release (2.84s)
PASS

Process finished with the exit code 0
enekofb commented 1 year ago

next is extend to kustomizations and to integrate with wge

enekofb commented 1 year ago 
2 scenarios (2 passed)
12 steps (12 passed)
5.688976375s
--- PASS: TestFeatures (5.69s)
    --- PASS: TestFeatures/can_watch_helm_release (2.85s)
    --- PASS: TestFeatures/can_watch_kustomizations (2.84s)
PASS
ok      github.com/enekofb/collector    5.997s

can watch kustomizations

 controller.go:228: watcher: "level"=0 "msg"="Starting workers" "controller"="kustomization" "controllerGroup"="kustomize.toolkit.fluxcd.io" "controllerKind"="Kustomization" "worker count"=1
    controller.go:228: watcher: "level"=0 "msg"="Starting workers" "controller"="helmrelease" "controllerGroup"="helm.toolkit.fluxcd.io" "controllerKind"="HelmRelease" "worker count"=1
    kustomize.go:69: watcher: "level"=0 "msg"="add or update operation requested" "controller"="kustomization" "controllerGroup"="kustomize.toolkit.fluxcd.io" "controllerKind"="Kustomization" "Kustomization"={"name":"podinfo","namespace":"default"} "namespace"="default" "name"="podinfo" "reconcileID"="ba96273a-65b4-4520-913a-58fc36c9bb4a" "kustomization"="default/podinfo"
    helm_release.go:106: watcher: "level"=0 "msg"="stored helm release document with id" "controller"="helmrelease" "controllerGroup"="helm.toolkit.fluxcd.io" "controllerKind"="HelmRelease" "HelmRelease"={"name":"podinfo","namespace":"default"} "namespace"="default" "name"="podinfo" "reconcileID"="7530d7e3-004e-427b-8828-e66e3dba2b24" "helmRelease"={"kind":"HelmRelease","apiVersion":"helm.toolkit.fluxcd.io/v2beta1","metadata":{"name":"podinfo","namespace":"default","uid":"0fe6feb1-14fa-4567-8a1b-5e47ca762576","resourceVersion":"702","generation":1,"creationTimestamp":"2023-03-06 12:36:07 +0000 GMT","annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"helm.toolkit.fluxcd.io/v2beta1\",\"kind\":\"HelmRelease\",\"metadata\":{\"annotations\":{},\"name\":\"podinfo\",\"namespace\":\"default\"},\"spec\":{\"chart\":{\"spec\":{\"chart\":\"podinfo\",\"sourceRef\":{\"apiVersion\":\"source.toolkit.fluxcd.io/v1beta2\",\"kind\":\"HelmRepository\",\"name\":\"app-charts\",\"namespace\":\"default\"},\"version\":\"0.0.15\"}},\"interval\":\"10m0s\",\"values\":{}}}\n"},"finalizers":["finalizers.fluxcd.io"],"managedFields":[{"manager":"helm-controller","operation":"Update","apiVersion":"helm.toolkit.fluxcd.io/v2beta1","time":"2023-03-06 12:36:07 +0000 GMT","fieldsType":"FieldsV1","fieldsV1":"{\"f:metadata\":{\"f:finalizers\":{\".\":{},\"v:\\\"finalizers.fluxcd.io\\\"\":{}}}}"},{"manager":"kubectl-client-side-apply","operation":"Update","apiVersion":"helm.toolkit.fluxcd.io/v2beta1","time":"2023-03-06 12:36:07 +0000 GMT","fieldsType":"FieldsV1","fieldsV1":"{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:kubectl.kubernetes.io/last-applied-configuration\":{}}},\"f:spec\":{\".\":{},\"f:chart\":{\".\":{},\"f:spec\":{\".\":{},\"f:chart\":{},\"f:reconcileStrategy\":{},\"f:sourceRef\":{\".\":{},\"f:apiVersion\":{},\"f:kind\":{},\"f:name\":{},\"f:namespace\":{}},\"f:version\":{}}},\"f:interval\":{},\"f:values\":{}}}"},{"manager":"helm-controller","operation":"Update","apiVersion":"helm.toolkit.fluxcd.io/v2beta1","time":"2023-03-06 12:36:09 +0000 GMT","fieldsType":"FieldsV1","fieldsV1":"{\"f:status\":{\"f:conditions\":{},\"f:helmChart\":{},\"f:lastAppliedRevision\":{},\"f:lastAttemptedRevision\":{},\"f:lastAttemptedValuesChecksum\":{},\"f:lastReleaseRevision\":{},\"f:observedGeneration\":{}}}","subresource":"status"}]},"spec":{"chart":{"spec":{"chart":"podinfo","version":"0.0.15","sourceRef":{"apiVersion":"source.toolkit.fluxcd.io/v1beta2","kind":"HelmRepository","name":"app-charts","namespace":"default"},"reconcileStrategy":"ChartVersion"}},"interval":{"Duration":"10m0s"},"values":"&JSON{Raw:*[123 125],}"},"status":{"observedGeneration":1,,"conditions":[{"type":"Ready","status":"True","lastTransitionTime":"2023-03-06 12:36:09 +0000 GMT","reason":"ReconciliationSucceeded","message":"Release reconciliation succeeded"},{"type":"Released","status":"True","lastTransitionTime":"2023-03-06 12:36:09 +0000 GMT","reason":"InstallSucceeded","message":"Helm install succeeded"}],"lastAppliedRevision":"0.0.15","lastAttemptedRevision":"0.0.15","lastAttemptedValuesChecksum":"da39a3ee5e6b4b0d3255bfef95601890afd80709","lastReleaseRevision":1,"helmChart":"default/default-podinfo"}} "<non-string-key: 1>"="<no-value>"
    kustomize.go:107: watcher: "level"=0 "msg"="stored document with id" "controller"="kustomization" "controllerGroup"="kustomize.toolkit.fluxcd.io" "controllerKind"="Kustomization" "Kustomization"={"name":"podinfo","namespace":"default"} "namespace"="default" "name"="podinfo" "reconcileID"="ba96273a-65b4-4520-913a-58fc36c9bb4a" "kustomization"={"kind":"Kustomization","apiVersion":"kustomize.toolkit.fluxcd.io/v1beta2","metadata":{"name":"podinfo","namespace":"default","uid":"9dcb8b62-7524-4524-a6ae-436ffa9d65f6","resourceVersion":"334333","generation":1,"creationTimestamp":"2023-03-09 16:16:29 +0000 GMT","annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"kustomize.toolkit.fluxcd.io/v1beta2\",\"kind\":\"Kustomization\",\"metadata\":{\"annotations\":{},\"name\":\"podinfo\",\"namespace\":\"default\"},\"spec\":{\"interval\":\"10m\",\"path\":\"./kustomize\",\"prune\":true,\"sourceRef\":{\"kind\":\"GitRepository\",\"name\":\"podinfo\"},\"targetNamespace\":\"default\"}}\n"},"finalizers":["finalizers.fluxcd.io"],"managedFields":[{"manager":"gotk-kustomize-controller","operation":"Update","apiVersion":"kustomize.toolkit.fluxcd.io/v1beta2","time":"2023-03-09 16:16:29 +0000 GMT","fieldsType":"FieldsV1","fieldsV1":"{\"f:metadata\":{\"f:finalizers\":{\".\":{},\"v:\\\"finalizers.fluxcd.io\\\"\":{}}}}"},{"manager":"kubectl-client-side-apply","operation":"Update","apiVersion":"kustomize.toolkit.fluxcd.io/v1beta2","time":"2023-03-09 16:16:29 +0000 GMT","fieldsType":"FieldsV1","fieldsV1":"{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:kubectl.kubernetes.io/last-applied-configuration\":{}}},\"f:spec\":{\".\":{},\"f:force\":{},\"f:interval\":{},\"f:path\":{},\"f:prune\":{},\"f:sourceRef\":{\".\":{},\"f:kind\":{},\"f:name\":{}},\"f:targetNamespace\":{}}}"},{"manager":"gotk-kustomize-controller","operation":"Update","apiVersion":"kustomize.toolkit.fluxcd.io/v1beta2","time":"2023-03-09 16:16:31 +0000 GMT","fieldsType":"FieldsV1","fieldsV1":"{\"f:status\":{\"f:conditions\":{},\"f:lastAttemptedRevision\":{}}}","subresource":"status"}]},"spec":{,"interval":{"Duration":"10m0s"},"path":"./kustomize","prune":true,"sourceRef":{,"kind":"GitRepository","name":"podinfo"},"targetNamespace":"default"},"status":{,"observedGeneration":-1,"conditions":[{"type":"Reconciling","status":"True","observedGeneration":1,"lastTransitionTime":"2023-03-10 11:16:26 +0000 GMT","reason":"ProgressingWithRetry","message":"Detecting drift for revision master/67e2c98a60dc92283531412a9e604dd4bae005a9 with a timeout of 9m30s"},{"type":"Ready","status":"False","observedGeneration":1,"lastTransitionTime":"2023-03-10 11:16:26 +0000 GMT","reason":"ReconciliationFailed","message":"Service/default/podinfo dry-run failed, reason: Invalid, error: Service \"podinfo\" is invalid: spec.ports[0].name: Required value\n"}],"lastAttemptedRevision":"master/67e2c98a60dc92283531412a9e604dd4bae005a9"}} "<non-string-key: 2>"="<no-value>"
    kustomize.go:69: watcher: "level"=0 "msg"="add or update operation requested" "controller"="kustomization" "controllerGroup"="kustomize.toolkit.fluxcd.io" "controllerKind"="Kustomization" "Kustomization"={"name":"podinfo2","namespace":"default"} "namespace"="default" "name"="podinfo2" "reconcileID"="99cddf38-ca18-4940-aa68-c9c3371aa1f3" "kustomization"="default/podinfo2"
    kustomize.go:107: watcher: "level"=0 "msg"="stored document with id" "controller"="kustomization" "controllerGroup"="kustomize.toolkit.fluxcd.io" "controllerKind"="Kustomization" "Kustomization"={"name":"podinfo2","namespace":"default"} "namespace"="default" "name"="podinfo2" "reconcileID"="99cddf38-ca18-4940-aa68-c9c3371aa1f3" "kustomization"={"kind":"Kustomization","apiVersion":"kustomize.toolkit.fluxcd.io/v1beta2","metadata":{"name":"podinfo2","namespace":"default","uid":"0cae9608-7523-40f7-8ef5-540f629daacc","resourceVersion":"335182","generation":1,"creationTimestamp":"2023-03-10 09:02:00 +0000 GMT","annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"kustomize.toolkit.fluxcd.io/v1beta2\",\"kind\":\"Kustomization\",\"metadata\":{\"annotations\":{},\"name\":\"podinfo2\",\"namespace\":\"default\"},\"spec\":{\"interval\":\"10m\",\"path\":\"./kustomize\",\"prune\":true,\"sourceRef\":{\"kind\":\"GitRepository\",\"name\":\"podinfo\"},\"targetNamespace\":\"default\"}}\n"},"finalizers":["finalizers.fluxcd.io"],"managedFields":[{"manager":"gotk-kustomize-controller","operation":"Update","apiVersion":"kustomize.toolkit.fluxcd.io/v1beta2","time":"2023-03-10 09:02:00 +0000 GMT","fieldsType":"FieldsV1","fieldsV1":"{\"f:metadata\":{\"f:finalizers\":{\".\":{},\"v:\\\"finalizers.fluxcd.io\\\"\":{}}}}"},{"manager":"gotk-kustomize-controller","operation":"Update","apiVersion":"kustomize.toolkit.fluxcd.io/v1beta2","time":"2023-03-10 09:02:00 +0000 GMT","fieldsType":"FieldsV1","fieldsV1":"{\"f:status\":{\"f:conditions\":{},\"f:lastAttemptedRevision\":{}}}","subresource":"status"},{"manager":"kubectl-client-side-apply","operation":"Update","apiVersion":"kustomize.toolkit.fluxcd.io/v1beta2","time":"2023-03-10 09:02:00 +0000 GMT","fieldsType":"FieldsV1","fieldsV1":"{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:kubectl.kubernetes.io/last-applied-configuration\":{}}},\"f:spec\":{\".\":{},\"f:force\":{},\"f:interval\":{},\"f:path\":{},\"f:prune\":{},\"f:sourceRef\":{\".\":{},\"f:kind\":{},\"f:name\":{}},\"f:targetNamespace\":{}}}"}]},"spec":{,"interval":{"Duration":"10m0s"},"path":"./kustomize","prune":true,"sourceRef":{,"kind":"GitRepository","name":"podinfo"},"targetNamespace":"default"},"status":{,"observedGeneration":-1,"conditions":[{"type":"Reconciling","status":"True","observedGeneration":1,"lastTransitionTime":"2023-03-10 11:23:26 +0000 GMT","reason":"ProgressingWithRetry","message":"Detecting drift for revision master/67e2c98a60dc92283531412a9e604dd4bae005a9 with a timeout of 9m30s"},{"type":"Ready","status":"False","observedGeneration":1,"lastTransitionTime":"2023-03-10 11:23:26 +0000 GMT","reason":"ReconciliationFailed","message":"Service/default/podinfo dry-run failed, reason: Invalid, error: Service \"podinfo\" is invalid: spec.ports[0].name: Required value\n"}],"lastAttemptedRevision":"master/67e2c98a60dc92283531412a9e604dd4bae005a9"}} "<non-string-key: 3>"="<no-value>"
    main_test.go:199: "level"=0 "msg"="waiting for started status:" "started"="<no-value>"
    main_test.go:208: "level"=0 "msg"="watcher has started"
enekofb commented 1 year ago

next is to integrate with wge

using this branch https://github.com/weaveworks/weave-gitops-enterprise/tree/scaling/add-collector

enekofb commented 1 year ago

integrating

this test as regression point https://github.com/weaveworks/weave-gitops-enterprise/commit/eb610f82a605713e3fe97192dcc72b7fce0db1f6#diff-097b90a9f3126043fde08a0309d9a50756fbb92f1dc95e08266592af2d66ebb5R36 

--- PASS: TestCollectorAcceptance (5.68s)
=== RUN   TestCollectorAcceptance/can_watch_helm_releases
    --- PASS: TestCollectorAcceptance/can_watch_helm_releases (2.84s)
=== RUN   TestCollectorAcceptance/can_watch_kustomizations
    --- PASS: TestCollectorAcceptance/can_watch_kustomizations (2.84s)
PASS

Process finished with the exit code 0