search

weaveworks / gitopssets-controller

Manages the GitOpsSet CRDs.
Apache License 2.0
9 stars 5 forks source link

Controller continuously crashLoop during WGE upgrade #122

Closed souleb closed 1 year ago

souleb commented 1 year ago

After upgrading Flux to v2.1.0 we have tried upgrading WGE, but the gitopssets controller keeps crash Looping.

We use the following HelmRlease:

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
  name: weave-gitops-enterprise-charts
  namespace: flux-system
spec:
  interval: 10m
  secretRef:
    name: xxx
  url: https://charts.dev.wkp.weave.works/releases/charts-v3
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: weave-gitops-enterprise
  namespace: flux-system
spec:
  install:
    crds: CreateReplace
  upgrade:
    crds: CreateReplace
  serviceAccountName: helm-controller
  chart:
    spec:
      chart: mccp
      sourceRef:
        kind: HelmRepository
        name: weave-gitops-enterprise-charts
        namespace: flux-system
      version: 0.30.0
  interval: 10m
  values:
    enableExplorer: true
    enableTerraformUI: true
    global:
      capiEnabled: false
    # -- Configure TLS settings if needed
    tls:
      enabled: false
    config:
      capi:
        repositoryURL: xxx
        # -- Can be changed depending on your git repo structure
        repositoryPath: ./clusters/empcentral-dv-aks/clusters
        # repositoryClustersPath: ./cluster
      git:
        type: github
        # -- Change if using on-prem github/gitlab
        # hostname: https://github.com
      oidc:
        enabled: true
        issuerURL: xxx
        redirectURL: xxx
        clientCredentialsSecret: xxx
      auth:
        userAccount:
          enabled: false
    ingress:
      enabled: true
      className: "nginx"
      annotations:
        external-dns.alpha.kubernetes.io/target: xxx
      hosts:
        - host: xxx
          paths:
            - path: /
              pathType: ImplementationSpecific
    gitopssets-controller:
      controllerManager:
        manager:
          args:
            - --health-probe-bind-address=:8081
            - --metrics-bind-address=127.0.0.1:8080
            - --leader-elect
            - --enabled-generators=GitRepository,Cluster,PullRequests,List,APIClient,ImagePolicy,Matrix,Config

We use Azure AKS with the following cluster info

{
  "major": "1",
  "minor": "25",
  "gitVersion": "v1.25.11",
  "gitCommit": "8cfcba0b15c343a8dc48567a74c29ec4844e0b9e",
  "gitTreeState": "clean",
  "buildDate": "2023-06-19T16:12:25Z",
  "goVersion": "go1.19.10",
  "compiler": "gc",
  "platform": "linux/amd64"
}

Flux version returns :

⋊> ~ flux version                                                                                                                                                                                   10:42:12
flux: v2.1.0
helm-controller: v0.36.0
image-automation-controller: v0.36.0
image-reflector-controller: v0.30.0
kustomize-controller: v1.1.0
notification-controller: v1.1.0
source-controller: v1.1.0
bigkevmcd commented 1 year ago

And there is nothing at all in the logs?

bigkevmcd commented 1 year ago

Even with --previous ?

souleb commented 1 year ago 
⋊> ~ k logs pod/gitopssets-controller-manager-74ddf9d874-k4tpt -n flux-system                                                                                                                       11:41:37
{"level":"info","ts":"2023-09-06T09:37:09.536Z","logger":"setup","msg":"configuring manager","version":"v0.15.3"}
{"level":"info","ts":"2023-09-06T09:37:09.536Z","logger":"setup","msg":"Enabled generators","generators":["GitRepository","Cluster","PullRequests","List","APIClient","ImagePolicy","Matrix","Config"]}
{"level":"info","ts":"2023-09-06T09:37:09.612Z","logger":"controller-runtime.metrics","msg":"Metrics server is starting to listen","addr":"127.0.0.1:8080"}
{"level":"info","ts":"2023-09-06T09:37:09.725Z","logger":"setup","msg":"starting manager"}
{"level":"info","ts":"2023-09-06T09:37:09.726Z","msg":"Starting server","kind":"health probe","addr":"[::]:8081"}
{"level":"info","ts":"2023-09-06T09:37:09.726Z","msg":"Starting server","path":"/metrics","kind":"metrics","addr":"127.0.0.1:8080"}
I0906 09:37:09.826745       1 leaderelection.go:248] attempting to acquire leader lease flux-system/539e4b66.weave.works...
I0906 09:37:26.932456       1 leaderelection.go:258] successfully acquired lease flux-system/539e4b66.weave.works
{"level":"info","ts":"2023-09-06T09:37:26.932Z","msg":"Starting EventSource","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet","source":"kind source: *v1alpha1.GitOpsSet"}
{"level":"info","ts":"2023-09-06T09:37:26.932Z","msg":"Starting EventSource","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet","source":"kind source: *v1beta2.GitRepository"}
{"level":"info","ts":"2023-09-06T09:37:26.932Z","msg":"Starting EventSource","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet","source":"kind source: *v1.Secret"}
{"level":"info","ts":"2023-09-06T09:37:26.932Z","msg":"Starting EventSource","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet","source":"kind source: *v1alpha1.GitopsCluster"}
{"level":"info","ts":"2023-09-06T09:37:26.932Z","msg":"Starting EventSource","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet","source":"kind source: *v1beta2.ImagePolicy"}
{"level":"info","ts":"2023-09-06T09:37:26.932Z","msg":"Starting Controller","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet"}
W0906 09:37:26.935676       1 warnings.go:70] v1beta2 GitRepository is deprecated, upgrade to v1
W0906 09:37:26.937683       1 warnings.go:70] v1beta2 GitRepository is deprecated, upgrade to v1
⋊> ~ k logs pod/gitopssets-controller-manager-74ddf9d874-k4tpt -n flux-system --previous                                                                                                            11:42:11
{"level":"info","ts":"2023-09-06T09:37:09.536Z","logger":"setup","msg":"configuring manager","version":"v0.15.3"}
{"level":"info","ts":"2023-09-06T09:37:09.536Z","logger":"setup","msg":"Enabled generators","generators":["GitRepository","Cluster","PullRequests","List","APIClient","ImagePolicy","Matrix","Config"]}
{"level":"info","ts":"2023-09-06T09:37:09.612Z","logger":"controller-runtime.metrics","msg":"Metrics server is starting to listen","addr":"127.0.0.1:8080"}
{"level":"info","ts":"2023-09-06T09:37:09.725Z","logger":"setup","msg":"starting manager"}
{"level":"info","ts":"2023-09-06T09:37:09.726Z","msg":"Starting server","kind":"health probe","addr":"[::]:8081"}
{"level":"info","ts":"2023-09-06T09:37:09.726Z","msg":"Starting server","path":"/metrics","kind":"metrics","addr":"127.0.0.1:8080"}
I0906 09:37:09.826745       1 leaderelection.go:248] attempting to acquire leader lease flux-system/539e4b66.weave.works...
I0906 09:37:26.932456       1 leaderelection.go:258] successfully acquired lease flux-system/539e4b66.weave.works
{"level":"info","ts":"2023-09-06T09:37:26.932Z","msg":"Starting EventSource","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet","source":"kind source: *v1alpha1.GitOpsSet"}
{"level":"info","ts":"2023-09-06T09:37:26.932Z","msg":"Starting EventSource","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet","source":"kind source: *v1beta2.GitRepository"}
{"level":"info","ts":"2023-09-06T09:37:26.932Z","msg":"Starting EventSource","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet","source":"kind source: *v1.Secret"}
{"level":"info","ts":"2023-09-06T09:37:26.932Z","msg":"Starting EventSource","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet","source":"kind source: *v1alpha1.GitopsCluster"}
{"level":"info","ts":"2023-09-06T09:37:26.932Z","msg":"Starting EventSource","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet","source":"kind source: *v1beta2.ImagePolicy"}
{"level":"info","ts":"2023-09-06T09:37:26.932Z","msg":"Starting Controller","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet"}
W0906 09:37:26.935676       1 warnings.go:70] v1beta2 GitRepository is deprecated, upgrade to v1
W0906 09:37:26.937683       1 warnings.go:70] v1beta2 GitRepository is deprecated, upgrade to v1
bigkevmcd commented 1 year ago 
$ k logs pod/gitopssets-controller-manager-f7668466c-bl5rn -n gitopssets-system
{"level":"info","ts":"2023-09-06T12:50:03.212Z","logger":"setup","msg":"configuring manager","version":"v0.16.1"}
{"level":"info","ts":"2023-09-06T12:50:03.212Z","logger":"setup","msg":"Enabled generators","generators":["GitRepository","OCIRepository","PullRequests","List","APIClient","Matrix","Config"]}
{"level":"info","ts":"2023-09-06T12:50:03.259Z","logger":"controller-runtime.metrics","msg":"Metrics server is starting to listen","addr":"127.0.0.1:8080"}
{"level":"info","ts":"2023-09-06T12:50:03.274Z","logger":"setup","msg":"starting manager"}
{"level":"info","ts":"2023-09-06T12:50:03.274Z","msg":"Starting server","path":"/metrics","kind":"metrics","addr":"127.0.0.1:8080"}
{"level":"info","ts":"2023-09-06T12:50:03.274Z","msg":"Starting server","kind":"health probe","addr":"[::]:8081"}
I0906 12:50:03.375541       1 leaderelection.go:248] attempting to acquire leader lease gitopssets-system/539e4b66.weave.works...
I0906 12:50:03.392731       1 leaderelection.go:258] successfully acquired lease gitopssets-system/539e4b66.weave.works
{"level":"info","ts":"2023-09-06T12:50:03.393Z","msg":"Starting EventSource","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet","source":"kind source: *v1alpha1.GitOpsSet"}
{"level":"info","ts":"2023-09-06T12:50:03.393Z","msg":"Starting EventSource","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet","source":"kind source: *v1beta2.GitRepository"}
{"level":"info","ts":"2023-09-06T12:50:03.393Z","msg":"Starting EventSource","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet","source":"kind source: *v1.Secret"}
{"level":"info","ts":"2023-09-06T12:50:03.393Z","msg":"Starting EventSource","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet","source":"kind source: *v1beta2.OCIRepository"}
{"level":"info","ts":"2023-09-06T12:50:03.393Z","msg":"Starting Controller","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet"}
W0906 12:50:03.396320       1 warnings.go:70] v1beta2 GitRepository is deprecated, upgrade to v1
W0906 12:50:03.400415       1 warnings.go:70] v1beta2 GitRepository is deprecated, upgrade to v1
{"level":"info","ts":"2023-09-06T12:50:03.494Z","msg":"Starting workers","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet","worker count":1}
$ kubectl get pods --all-namespaces -o jsonpath="{.items[*].spec.containers[*].image}" |\
> tr -s '[[:space:]]' '\n' |\
> sort |\
> uniq -c
   1 docker.io/kindest/kindnetd:v20230511-dc714da8
   1 docker.io/kindest/local-path-provisioner:v20230511-dc714da8
   1 gcr.io/kubebuilder/kube-rbac-proxy:v0.14.1
   1 ghcr.io/fluxcd/kustomize-controller:v1.1.0
   1 ghcr.io/fluxcd/source-controller:v1.1.0
   1 ghcr.io/weaveworks/gitopssets-controller:v0.16.1
   2 registry.k8s.io/coredns/coredns:v1.9.3
   1 registry.k8s.io/etcd:3.5.6-0
   1 registry.k8s.io/kube-apiserver:v1.25.9
   1 registry.k8s.io/kube-controller-manager:v1.25.9
   1 registry.k8s.io/kube-proxy:v1.25.9
   1 registry.k8s.io/kube-scheduler:v1.25.9
bigkevmcd commented 1 year ago

Closing this as it was an OOM issue