search

fluxcd / helm-operator

Successor: https://github.com/fluxcd/helm-controller — The Flux Helm Operator, once upon a time a solution for declarative Helming.
https://docs.fluxcd.io/projects/helm-operator/
Apache License 2.0
649 stars 262 forks source link

Helm Operator crashes with cannot patch resource "events" in API group error #175

Closed demisx closed 4 years ago

demisx commented 4 years ago

Describe the bug I am upgrading helm operator from v0.15.0. I've deleted the old flux chart and installed the latest 1.0.0 with helm. I then installed the latest Helm Operator v0.4.0 with helm chart. However, the operator crashes with cannot patch resource "events" in API group "" in the namespace "flux" error after running for a couple of mins.

To Reproduce Install with:

helm install fluxcd/helm-operator \
  --name helm-operator \
  --namespace flux \
  --set createCRD=true \
  --set git.ssh.secretName=flux-git-deploy \
  --set chartsSyncInterval=2m \
  --version 0.4.0 \
  --atomic

Expected behavior A clear and concise description of what you expected to happen.

Logs

W0106 21:26:17.036972       7 client_config.go:541] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
ts=2020-01-06T21:26:17.064452665Z caller=helm.go:70 component=helm version=v2 info="connected to Tiller" version="sem_ver:\"v2.15.2\" git_commit:\"8dce272473e5f2a7bf58ce79bb5c3691db54c96b\" git_tree_state:\"clean\" " host=tiller-deploy.kube-system:44134 options="{Host: Port: Namespace:kube-system TLSVerify:false TLSEnable:false TLSKey:/etc/fluxd/helm/tls.key TLSCert:/etc/fluxd/helm/tls.crt TLSCACert: TLSHostname:}"
ts=2020-01-06T21:26:17.064831768Z caller=operator.go:99 component=operator info="setting up event handlers"
ts=2020-01-06T21:26:17.064865058Z caller=operator.go:119 component=operator info="event handlers set up"
ts=2020-01-06T21:26:17.064885759Z caller=main.go:279 component=helm-operator info="waiting for informer caches to sync"
ts=2020-01-06T21:26:17.165057648Z caller=main.go:284 component=helm-operator info="informer caches synced"
ts=2020-01-06T21:26:17.165299211Z caller=git.go:94 component=gitchartsync info="starting sync of git chart sources"
ts=2020-01-06T21:26:17.175356351Z caller=operator.go:131 component=operator info="starting operator"
ts=2020-01-06T21:26:17.176395112Z caller=operator.go:133 component=operator info="starting workers"
ts=2020-01-06T21:26:17.18408072Z caller=server.go:42 component=daemonhttp info="starting HTTP server on :3030"
ts=2020-01-06T21:26:17.534050425Z caller=checkpoint.go:24 component=checkpoint msg="up to date" latest=0.10.1

ts=2020-01-06T21:29:08.092923628Z caller=release.go:142 component=release release=elasticsearch targetNamespace=default resource=default:helmrelease/elasticsearch helmVersion=v2 error="chart unavailable: no cached repo found. (try 'helm repo update'). open /var/fluxd/helm/repository/cache/stable-index.yaml: no such file or directory"
ts=2020-01-06T21:29:08.77262632Z caller=release.go:142 component=release release=cert-manager targetNamespace=cert-manager resource=cert-manager:helmrelease/cert-manager helmVersion=v2 error="chart unavailable: no cached repo found. (try 'helm repo update'). open /var/fluxd/helm/repository/cache/stable-index.yaml: no such file or directory"
ts=2020-01-06T21:29:08.973334924Z caller=release.go:142 component=release release=flux targetNamespace=flux resource=flux:helmrelease/flux helmVersion=v2 error="chart unavailable: no cached repo found. (try 'helm repo update'). open /var/fluxd/helm/repository/cache/stable-index.yaml: no such file or directory"
ts=2020-01-06T21:29:09.179209821Z caller=release.go:142 component=release release=helm-operator targetNamespace=flux resource=flux:helmrelease/helm-operator helmVersion=v2 error="chart unavailable: no cached repo found. (try 'helm repo update'). open /var/fluxd/helm/repository/cache/stable-index.yaml: no such file or directory"
ts=2020-01-06T21:29:20.583292399Z caller=release.go:254 component=release release=external-dns targetNamespace=default resource=default:helmrelease/external-dns helmVersion=v2 info="Helm release sync succeeded" revision=2.9.3
ts=2020-01-06T21:29:22.581301388Z caller=release.go:254 component=release release=k8s-spot-termination-handler targetNamespace=kube-system resource=kube-system:helmrelease/k8s-spot-termination-handler helmVersion=v2 info="Helm release sync succeeded" revision=1.4.3
ts=2020-01-06T21:29:32.502399118Z caller=release.go:254 component=release release=nginx-ingress targetNamespace=default resource=default:helmrelease/nginx-ingress helmVersion=v2 info="Helm release sync succeeded" revision=1.24.3
ts=2020-01-06T21:29:40.984223742Z caller=release.go:254 component=release release=sealed-secrets-controller targetNamespace=kube-system resource=kube-system:helmrelease/sealed-secrets-controller helmVersion=v2 info="Helm release sync succeeded" revision=1.4.0

ts=2020-01-06T21:29:50.639202345Z caller=release.go:254 component=release release=stolon targetNamespace=default resource=default:helmrelease/stolon helmVersion=v2 info="Helm release sync succeeded" revision=1.1.2
ts=2020-01-06T21:30:17.082535926Z caller=operator.go:309 component=operator info="enqueuing release" resource=cert-manager:helmrelease/cert-manager
ts=2020-01-06T21:30:17.082782679Z caller=operator.go:309 component=operator info="enqueuing release" resource=default:helmrelease/elasticsearch
ts=2020-01-06T21:30:17.082969231Z caller=operator.go:309 component=operator info="enqueuing release" resource=flux:helmrelease/flux
ts=2020-01-06T21:30:17.083414455Z caller=operator.go:309 component=operator info="enqueuing release" resource=monitoring:helmrelease/prometheus-operator
ts=2020-01-06T21:30:17.083695158Z caller=operator.go:309 component=operator info="enqueuing release" resource=default:helmrelease/external-dns
ts=2020-01-06T21:30:17.08387862Z caller=operator.go:309 component=operator info="enqueuing release" resource=flux:helmrelease/helm-operator
ts=2020-01-06T21:30:17.084077481Z caller=operator.go:309 component=operator info="enqueuing release" resource=kube-system:helmrelease/k8s-spot-termination-handler
ts=2020-01-06T21:30:17.084306654Z caller=operator.go:309 component=operator info="enqueuing release" resource=default:helmrelease/nginx-ingress
ts=2020-01-06T21:30:17.084492476Z caller=operator.go:309 component=operator info="enqueuing release" resource=kube-system:helmrelease/sealed-secrets-controller
ts=2020-01-06T21:30:17.084678698Z caller=operator.go:309 component=operator info="enqueuing release" resource=default:helmrelease/stolon
ts=2020-01-06T21:30:19.565257709Z caller=release.go:142 component=release release=cert-manager targetNamespace=cert-manager resource=cert-manager:helmrelease/cert-manager helmVersion=v2 error="chart unavailable: no cached repo found. (try 'helm repo update'). open /var/fluxd/helm/repository/cache/flagger-index.yaml: no such file or directory"
ts=2020-01-06T21:30:21.573208924Z caller=release.go:142 component=release release=elasticsearch targetNamespace=default resource=default:helmrelease/elasticsearch helmVersion=v2 error="chart unavailable: no cached repo found. (try 'helm repo update'). open /var/fluxd/helm/repository/cache/flagger-index.yaml: no such file or directory"
ts=2020-01-06T21:30:24.297758829Z caller=release.go:254 component=release release=flux targetNamespace=flux resource=flux:helmrelease/flux helmVersion=v2 info="Helm release sync succeeded" revision=1.0.0
ts=2020-01-06T21:30:29.058104692Z caller=release.go:254 component=release release=helm-operator targetNamespace=flux resource=flux:helmrelease/helm-operator helmVersion=v2 info="Helm release sync succeeded" revision=0.4.0
ts=2020-01-06T21:30:29.242897034Z caller=annotator.go:81 component=release release=helm-operator targetNamespace=flux resource=flux:helmrelease/helm-operator helmVersion=v2 output="Error from server (Forbidden): deployments.extensions \"helm-operator-flux-memcached\" is forbidden: User \"system:serviceaccount:flux:helm-operator\" cannot get resource \"deployments\" in API group \"extensions\" in the namespace \"flux\"\nError from server (Forbidden): secrets \"helm-operator-flux-git-deploy\" is forbidden: User \"system:serviceaccount:flux:helm-operator\" cannot get resource \"secrets\" in API group \"\" in the namespace \"flux\"\nError from server (Forbidden): configmaps \"helm-operator-flux-ssh-config\" is forbidden: User \"system:serviceaccount:flux:helm-operator\" cannot get resource \"configmaps\" in API group \"\" in the namespace \"flux\"\nError from server (Forbidden): clusterroles.rbac.authorization.k8s.io \"helm-operator-flux\" is forbidden: User \"system:serviceaccount:flux:helm-operator\" cannot get resource \"clusterroles\" in API group \"rbac.authorization.k8s.io\" at the cluster scope\nError from server (Forbidden): services \"helm-operator-flux\" is forbidden: User \"system:serviceaccount:flux:helm-operator\" cannot get resource \"services\" in API group \"\" in the namespace \"flux\"\nError from server (Forbidden): deployments.extensions \"helm-operator-flux\" is forbidden: User \"system:serviceaccount:flux:helm-operator\" cannot get resource \"deployments\" in API group \"extensions\" in the namespace \"flux\"\nError from server (Forbidden): configmaps \"helm-operator-flux-kube-config\" is forbidden: User \"system:serviceaccount:flux:helm-operator\" cannot get resource \"configmaps\" in API group \"\" in the namespace \"flux\"\nError from server (Forbidden): serviceaccounts \"helm-operator-flux\" is forbidden: User \"system:serviceaccount:flux:helm-operator\" cannot get resource \"serviceaccounts\" in API group \"\" in the namespace \"flux\"\nError from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io \"helm-operator-flux\" is forbidden: User \"system:serviceaccount:flux:helm-operator\" cannot get resource \"clusterrolebindings\" in API group \"rbac.authorization.k8s.io\" at the cluster scope\nError from server (Forbidden): services \"helm-operator-flux-memcached\" is forbidden: User \"system:serviceaccount:flux:helm-operator\" cannot get resource \"services\" in API group \"\" in the namespace \"flux\"\n" err="exit status 1"
E0106 21:30:29.250408       7 event.go:237] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"helm-operator.15e76975d8f37756", GenerateName:"", Namespace:"flux", SelfLink:"", UID:"", ResourceVersion:"2660", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"HelmRelease", Namespace:"flux", Name:"helm-operator", UID:"bb1d0e00-b5fb-4ad3-a747-4bed3d2e3527", APIVersion:"helm.fluxcd.io/v1", ResourceVersion:"66045", FieldPath:""}, Reason:"ChartSynced", Message:"Chart managed by HelmRelease processed", Source:v1.EventSource{Component:"helm-operator", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63713942949, loc:(*time.Location)(0x29bea00)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbf7d079d4eb45a8a, ext:252262648909, loc:(*time.Location)(0x29bea00)}}, Count:2, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "helm-operator.15e76975d8f37756" is forbidden: User "system:serviceaccount:flux:helm-operator" cannot patch resource "events" in API group "" in the namespace "flux"' (will not retry!)
ts=2020-01-06T21:30:29.541977061Z caller=main.go:303 exiting...=terminated
ts=2020-01-06T21:30:29.542053641Z caller=status.go:86 component=statusupdater loop=stopping err=null
ts=2020-01-06T21:30:29.542131772Z caller=operator.go:143 component=operator info="stopping workers"
ts=2020-01-06T21:30:29.542185183Z caller=server.go:59 component=daemonhttp info="HTTP server stopped"
ts=2020-01-06T21:30:29.542203563Z caller=git.go:138 component=gitchartsync info="stopping sync of git chart sources"
rpc error: code = Unknown desc = Error: No such container: 04f70ed238cfdab299a3066df25acf73fa7e9f28fc04cb2ea4b2f8e1dc9124c0

Additional context Add any other context about the problem here, e.g

stefanprodan commented 4 years ago

Based on cannot get resource deployments in API group extensions I would say you are using an old chart, we've moved to apps/v1 for some months now. Try deleting the local repo and add it with helm repo add fluxcd https://charts.fluxcd.io

demisx commented 4 years ago

@stefanprodan Thank you very much for your prompt response. This is what I've tried again with a brand new cluster:

  1. Remove existing fluxcd repo and re-add a new one (though, they appear to be the same)

    $ helm repo list
---
fluxcd      https://charts.fluxcd.io 
...
$ helm repo remove fluxcd
$  helm repo add fluxcd https://charts.fluxcd.io
$ helm repo list
---
fluxcd      https://charts.fluxcd.io

  2. Bring up a brand new Kops 1.15 cluster in AWS

  3. Install flux and helm operator with:

    helm repo add fluxcd https://charts.fluxcd.io
helm repo update
ssh-keyscan -p [port] [git.host] > /tmp/flux_known_hosts
helm install fluxcd/flux \
--name flux \
--set git.ciSkip=true \
--set git.url=$git_url \
--set git.branch=$flux_git_branch \
--set git.path=$flux_git_path \
--set git.pollInterval=3m \
--set git.label=flux-sync-$env \
--set-file ssh.known_hosts=/tmp/flux_known_hosts \
--namespace fluxcd \ # <- changed this from flux to fluxcd
--version 1.0.0 \
--atomic
rm /tmp/flux_known_hosts
...
helm install fluxcd/helm-operator \
--name helm-operator \
--namespace fluxcd \ # <- changed this from flux to fluxcd
--set createCRD=true \
--set git.ssh.secretName=flux-git-deploy \
--set chartsSyncInterval=2m \
--version 0.4.0 \
--atomic

  4. I see the following issues: 4.1 Flux periodically reports this error in the log (possible #1948?):

    ts=2020-01-07T16:14:20.369096153Z caller=warming.go:180 component=warmer canonical_name=quay.io/weaveworks/flux auth={map[]} err="requesting tags: Get https://quay.io/v2/weaveworks/flux/tags/list: denied: Namespace weaveworks has been disabled. Please contact a system administrator."

4.2 Helm operator still crashes with similar error. Here is the full log:

dmoore [stg] (stg)[1]> k logs -n fluxcd -f deploy/helm-operator
W0107 16:05:23.738911       6 client_config.go:541] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
ts=2020-01-07T16:05:23.75774262Z caller=helm.go:70 component=helm version=v2 info="connected to Tiller" version="sem_ver:\"v2.15.2\" git_commit:\"8dce272473e5f2a7bf58ce79bb5c3691db54c96b\" git_tree_state:\"clean\" " host=tiller-deploy.kube-system:44134 options="{Host: Port: Namespace:kube-system TLSVerify:false TLSEnable:false TLSKey:/etc/fluxd/helm/tls.key TLSCert:/etc/fluxd/helm/tls.crt TLSCACert: TLSHostname:}"
ts=2020-01-07T16:05:23.758192625Z caller=operator.go:99 component=operator info="setting up event handlers"
ts=2020-01-07T16:05:23.758230936Z caller=operator.go:119 component=operator info="event handlers set up"
ts=2020-01-07T16:05:23.758310486Z caller=main.go:279 component=helm-operator info="waiting for informer caches to sync"
ts=2020-01-07T16:05:23.864134169Z caller=main.go:284 component=helm-operator info="informer caches synced"
ts=2020-01-07T16:05:23.864795935Z caller=git.go:94 component=gitchartsync info="starting sync of git chart sources"
ts=2020-01-07T16:05:23.865577403Z caller=operator.go:131 component=operator info="starting operator"
ts=2020-01-07T16:05:23.865683214Z caller=operator.go:133 component=operator info="starting workers"
ts=2020-01-07T16:05:23.865902886Z caller=server.go:42 component=daemonhttp info="starting HTTP server on :3030"
ts=2020-01-07T16:05:24.147486811Z caller=checkpoint.go:24 component=checkpoint msg="up to date" latest=0.10.1
ts=2020-01-07T16:05:31.72605474Z caller=release.go:142 component=release release=elasticsearch targetNamespace=default resource=default:helmrelease/elasticsearch helmVersion=v2 error="chart unavailable: no cached repo found. (try 'helm repo update'). open /var/fluxd/helm/repository/cache/stable-index.yaml: no such file or directory"
ts=2020-01-07T16:05:32.094896824Z caller=release.go:142 component=release release=cert-manager targetNamespace=cert-manager resource=cert-manager:helmrelease/cert-manager helmVersion=v2 error="chart unavailable: no cached repo found. (try 'helm repo update'). open /var/fluxd/helm/repository/cache/stable-index.yaml: no such file or directory"
ts=2020-01-07T16:05:32.338339381Z caller=release.go:142 component=release release=flux targetNamespace=fluxcd resource=fluxcd:helmrelease/flux helmVersion=v2 error="chart unavailable: no cached repo found. (try 'helm repo update'). open /var/fluxd/helm/repository/cache/stable-index.yaml: no such file or directory"
ts=2020-01-07T16:05:32.527347888Z caller=release.go:142 component=release release=helm-operator targetNamespace=fluxcd resource=fluxcd:helmrelease/helm-operator helmVersion=v2 error="chart unavailable: no cached repo found. (try 'helm repo update'). open /var/fluxd/helm/repository/cache/stable-index.yaml: no such file or directory"
ts=2020-01-07T16:05:42.575867011Z caller=release.go:254 component=release release=external-dns targetNamespace=default resource=default:helmrelease/external-dns helmVersion=v2 info="Helm release sync succeeded" revision=2.9.3
ts=2020-01-07T16:05:43.825114388Z caller=release.go:254 component=release release=k8s-spot-termination-handler targetNamespace=kube-system resource=kube-system:helmrelease/k8s-spot-termination-handler helmVersion=v2 info="Helm release sync succeeded" revision=1.4.3
ts=2020-01-07T16:05:53.469081504Z caller=release.go:254 component=release release=nginx-ingress targetNamespace=default resource=default:helmrelease/nginx-ingress helmVersion=v2 info="Helm release sync succeeded" revision=1.24.7
ts=2020-01-07T16:06:02.106397471Z caller=release.go:254 component=release release=sealed-secrets-controller targetNamespace=kube-system resource=kube-system:helmrelease/sealed-secrets-controller helmVersion=v2 info="Helm release sync succeeded" revision=1.4.0
ts=2020-01-07T16:06:11.273049339Z caller=release.go:254 component=release release=stolon targetNamespace=default resource=default:helmrelease/stolon helmVersion=v2 info="Helm release sync succeeded" revision=1.1.2
ts=2020-01-07T16:07:02.258191067Z caller=release.go:254 component=release release=prometheus-operator targetNamespace=monitoring resource=monitoring:helmrelease/prometheus-operator helmVersion=v2 info="Helm release sync succeeded" revision=8.0.0
ts=2020-01-07T16:07:02.413186259Z caller=annotator.go:100 component=release release=prometheus-operator targetNamespace=monitoring resource=monitoring:helmrelease/prometheus-operator helmVersion=v2 err="Object 'Kind' is missing in 'null'"
ts=2020-01-07T16:07:02.413245729Z caller=annotator.go:100 component=release release=prometheus-operator targetNamespace=monitoring resource=monitoring:helmrelease/prometheus-operator helmVersion=v2 err="Object 'Kind' is missing in 'null'"
ts=2020-01-07T16:07:02.421969457Z caller=annotator.go:100 component=release release=prometheus-operator targetNamespace=monitoring resource=monitoring:helmrelease/prometheus-operator helmVersion=v2 err="Object 'Kind' is missing in 'null'"
ts=2020-01-07T16:07:02.422034508Z caller=annotator.go:100 component=release release=prometheus-operator targetNamespace=monitoring resource=monitoring:helmrelease/prometheus-operator helmVersion=v2 err="Object 'Kind' is missing in 'null'"
ts=2020-01-07T16:07:02.422068108Z caller=annotator.go:100 component=release release=prometheus-operator targetNamespace=monitoring resource=monitoring:helmrelease/prometheus-operator helmVersion=v2 err="Object 'Kind' is missing in 'null'"
ts=2020-01-07T16:07:02.422110108Z caller=annotator.go:100 component=release release=prometheus-operator targetNamespace=monitoring resource=monitoring:helmrelease/prometheus-operator helmVersion=v2 err="Object 'Kind' is missing in 'null'"
ts=2020-01-07T16:07:02.425466375Z caller=annotator.go:100 component=release release=prometheus-operator targetNamespace=monitoring resource=monitoring:helmrelease/prometheus-operator helmVersion=v2 err="Object 'Kind' is missing in 'null'"
ts=2020-01-07T16:07:02.428220136Z caller=annotator.go:100 component=release release=prometheus-operator targetNamespace=monitoring resource=monitoring:helmrelease/prometheus-operator helmVersion=v2 err="Object 'Kind' is missing in 'null'"
ts=2020-01-07T16:07:02.430292722Z caller=annotator.go:100 component=release release=prometheus-operator targetNamespace=monitoring resource=monitoring:helmrelease/prometheus-operator helmVersion=v2 err="Object 'Kind' is missing in 'null'"
ts=2020-01-07T16:07:02.430330492Z caller=annotator.go:100 component=release release=prometheus-operator targetNamespace=monitoring resource=monitoring:helmrelease/prometheus-operator helmVersion=v2 err="Object 'Kind' is missing in 'null'"
ts=2020-01-07T16:07:02.430636964Z caller=annotator.go:100 component=release release=prometheus-operator targetNamespace=monitoring resource=monitoring:helmrelease/prometheus-operator helmVersion=v2 err="Object 'Kind' is missing in 'null'"
ts=2020-01-07T16:07:02.430684255Z caller=annotator.go:100 component=release release=prometheus-operator targetNamespace=monitoring resource=monitoring:helmrelease/prometheus-operator helmVersion=v2 err="Object 'Kind' is missing in 'null'"
ts=2020-01-07T16:07:02.4326394Z caller=annotator.go:100 component=release release=prometheus-operator targetNamespace=monitoring resource=monitoring:helmrelease/prometheus-operator helmVersion=v2 err="Object 'Kind' is missing in 'null'"
ts=2020-01-07T16:07:02.440035187Z caller=annotator.go:100 component=release release=prometheus-operator targetNamespace=monitoring resource=monitoring:helmrelease/prometheus-operator helmVersion=v2 err="Object 'Kind' is missing in 'null'"
ts=2020-01-07T16:07:02.440085668Z caller=annotator.go:100 component=release release=prometheus-operator targetNamespace=monitoring resource=monitoring:helmrelease/prometheus-operator helmVersion=v2 err="Object 'Kind' is missing in 'null'"
ts=2020-01-07T16:07:02.451190404Z caller=annotator.go:100 component=release release=prometheus-operator targetNamespace=monitoring resource=monitoring:helmrelease/prometheus-operator helmVersion=v2 err="Object 'Kind' is missing in 'null'"
ts=2020-01-07T16:07:02.467314329Z caller=annotator.go:100 component=release release=prometheus-operator targetNamespace=monitoring resource=monitoring:helmrelease/prometheus-operator helmVersion=v2 err="Object 'Kind' is missing in 'null'"
ts=2020-01-07T16:07:02.478129553Z caller=annotator.go:100 component=release release=prometheus-operator targetNamespace=monitoring resource=monitoring:helmrelease/prometheus-operator helmVersion=v2 err="Object 'Kind' is missing in 'null'"
ts=2020-01-07T16:07:02.479371352Z caller=annotator.go:100 component=release release=prometheus-operator targetNamespace=monitoring resource=monitoring:helmrelease/prometheus-operator helmVersion=v2 err="Object 'Kind' is missing in 'null'"
ts=2020-01-07T16:07:02.483277302Z caller=annotator.go:100 component=release release=prometheus-operator targetNamespace=monitoring resource=monitoring:helmrelease/prometheus-operator helmVersion=v2 err="Object 'Kind' is missing in 'null'"
ts=2020-01-07T16:07:02.484537532Z caller=annotator.go:100 component=release release=prometheus-operator targetNamespace=monitoring resource=monitoring:helmrelease/prometheus-operator helmVersion=v2 err="Object 'Kind' is missing in 'null'"
ts=2020-01-07T16:07:02.484588153Z caller=annotator.go:100 component=release release=prometheus-operator targetNamespace=monitoring resource=monitoring:helmrelease/prometheus-operator helmVersion=v2 err="Object 'Kind' is missing in 'null'"
ts=2020-01-07T16:07:02.48545914Z caller=annotator.go:100 component=release release=prometheus-operator targetNamespace=monitoring resource=monitoring:helmrelease/prometheus-operator helmVersion=v2 err="Object 'Kind' is missing in 'null'"
ts=2020-01-07T16:07:02.48549709Z caller=annotator.go:100 component=release release=prometheus-operator targetNamespace=monitoring resource=monitoring:helmrelease/prometheus-operator helmVersion=v2 err="Object 'Kind' is missing in 'null'"
ts=2020-01-07T16:07:02.491042473Z caller=annotator.go:100 component=release release=prometheus-operator targetNamespace=monitoring resource=monitoring:helmrelease/prometheus-operator helmVersion=v2 err="Object 'Kind' is missing in 'null'"
ts=2020-01-07T16:07:02.491104723Z caller=annotator.go:100 component=release release=prometheus-operator targetNamespace=monitoring resource=monitoring:helmrelease/prometheus-operator helmVersion=v2 err="Object 'Kind' is missing in 'null'"
ts=2020-01-07T16:07:02.496769627Z caller=annotator.go:100 component=release release=prometheus-operator targetNamespace=monitoring resource=monitoring:helmrelease/prometheus-operator helmVersion=v2 err="Object 'Kind' is missing in 'null'"
ts=2020-01-07T16:07:02.496832728Z caller=annotator.go:100 component=release release=prometheus-operator targetNamespace=monitoring resource=monitoring:helmrelease/prometheus-operator helmVersion=v2 err="Object 'Kind' is missing in 'null'"
ts=2020-01-07T16:07:02.499911582Z caller=annotator.go:100 component=release release=prometheus-operator targetNamespace=monitoring resource=monitoring:helmrelease/prometheus-operator helmVersion=v2 err="Object 'Kind' is missing in 'null'"
ts=2020-01-07T16:07:02.499958102Z caller=annotator.go:100 component=release release=prometheus-operator targetNamespace=monitoring resource=monitoring:helmrelease/prometheus-operator helmVersion=v2 err="Object 'Kind' is missing in 'null'"
ts=2020-01-07T16:07:02.503336449Z caller=annotator.go:100 component=release release=prometheus-operator targetNamespace=monitoring resource=monitoring:helmrelease/prometheus-operator helmVersion=v2 err="Object 'Kind' is missing in 'null'"
ts=2020-01-07T16:07:02.504435558Z caller=annotator.go:100 component=release release=prometheus-operator targetNamespace=monitoring resource=monitoring:helmrelease/prometheus-operator helmVersion=v2 err="Object 'Kind' is missing in 'null'"
ts=2020-01-07T16:07:02.504493087Z caller=annotator.go:100 component=release release=prometheus-operator targetNamespace=monitoring resource=monitoring:helmrelease/prometheus-operator helmVersion=v2 err="Object 'Kind' is missing in 'null'"
ts=2020-01-07T16:07:02.506961667Z caller=annotator.go:100 component=release release=prometheus-operator targetNamespace=monitoring resource=monitoring:helmrelease/prometheus-operator helmVersion=v2 err="Object 'Kind' is missing in 'null'"
ts=2020-01-07T16:07:23.780124828Z caller=operator.go:309 component=operator info="enqueuing release" resource=cert-manager:helmrelease/cert-manager
ts=2020-01-07T16:07:23.78034867Z caller=operator.go:309 component=operator info="enqueuing release" resource=default:helmrelease/external-dns
ts=2020-01-07T16:07:23.780768363Z caller=operator.go:309 component=operator info="enqueuing release" resource=monitoring:helmrelease/prometheus-operator
ts=2020-01-07T16:07:23.780906594Z caller=operator.go:309 component=operator info="enqueuing release" resource=kube-system:helmrelease/sealed-secrets-controller
ts=2020-01-07T16:07:23.781054324Z caller=operator.go:309 component=operator info="enqueuing release" resource=default:helmrelease/stolon
ts=2020-01-07T16:07:23.781161575Z caller=operator.go:309 component=operator info="enqueuing release" resource=default:helmrelease/elasticsearch
ts=2020-01-07T16:07:23.781314237Z caller=operator.go:309 component=operator info="enqueuing release" resource=fluxcd:helmrelease/flux
ts=2020-01-07T16:07:23.781446357Z caller=operator.go:309 component=operator info="enqueuing release" resource=fluxcd:helmrelease/helm-operator
ts=2020-01-07T16:07:23.781608709Z caller=operator.go:309 component=operator info="enqueuing release" resource=kube-system:helmrelease/k8s-spot-termination-handler
ts=2020-01-07T16:07:23.781818481Z caller=operator.go:309 component=operator info="enqueuing release" resource=default:helmrelease/nginx-ingress
ts=2020-01-07T16:07:27.077414872Z caller=release.go:142 component=release release=cert-manager targetNamespace=cert-manager resource=cert-manager:helmrelease/cert-manager helmVersion=v2 error="chart unavailable: no cached repo found. (try 'helm repo update'). open /var/fluxd/helm/repository/cache/flagger-index.yaml: no such file or directory"
ts=2020-01-07T16:07:31.437605671Z caller=release.go:142 component=release release=elasticsearch targetNamespace=default resource=default:helmrelease/elasticsearch helmVersion=v2 error="chart unavailable: no cached repo found. (try 'helm repo update'). open /var/fluxd/helm/repository/cache/flagger-index.yaml: no such file or directory"
ts=2020-01-07T16:07:35.888387792Z caller=release.go:254 component=release release=flux targetNamespace=fluxcd resource=fluxcd:helmrelease/flux helmVersion=v2 info="Helm release sync succeeded" revision=1.0.0
ts=2020-01-07T16:07:39.079858651Z caller=release.go:254 component=release release=helm-operator targetNamespace=fluxcd resource=fluxcd:helmrelease/helm-operator helmVersion=v2 info="Helm release sync succeeded" revision=0.4.0
ts=2020-01-07T16:07:39.315477875Z caller=annotator.go:81 component=release release=helm-operator targetNamespace=fluxcd resource=fluxcd:helmrelease/helm-operator helmVersion=v2 output="error: You must be logged in to the server (Unauthorized)\nerror: You must be logged in to the server (Unauthorized)\nerror: You must be logged in to the server (Unauthorized)\nerror: You must be logged in to the server (Unauthorized)\nerror: You must be logged in to the server (Unauthorized)\nerror: You must be logged in to the server (Unauthorized)\nerror: You must be logged in to the server (Unauthorized)\nerror: You must be logged in to the server (Unauthorized)\nerror: You must be logged in to the server (Unauthorized)\nerror: You must be logged in to the server (Unauthorized)\n" err="exit status 1"
E0107 16:07:39.319466       6 event.go:237] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"helm-operator.15e7a661b107ae3a", GenerateName:"", Namespace:"fluxcd", SelfLink:"", UID:"", ResourceVersion:"1347", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"HelmRelease", Namespace:"fluxcd", Name:"helm-operator", UID:"acebc156-74ad-41c5-beac-e88288b996d9", APIVersion:"helm.fluxcd.io/v1", ResourceVersion:"11820", FieldPath:""}, Reason:"ChartSynced", Message:"Chart managed by HelmRelease processed", Source:v1.EventSource{Component:"helm-operator", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63714009932, loc:(*time.Location)(0x29bea00)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbf7d4912d2f3e2b5, ext:135629480489, loc:(*time.Location)(0x29bea00)}}, Count:2, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "helm-operator.15e7a661b107ae3a" is forbidden: User "system:serviceaccount:fluxcd:helm-operator" cannot patch resource "events" in API group "" in the namespace "fluxcd"' (will not retry!)
ts=2020-01-07T16:07:39.674950609Z caller=main.go:303 exiting...=terminated
ts=2020-01-07T16:07:39.675093649Z caller=status.go:86 component=statusupdater loop=stopping err=null
ts=2020-01-07T16:07:39.67514324Z caller=server.go:59 component=daemonhttp info="HTTP server stopped"
ts=2020-01-07T16:07:39.6751613Z caller=operator.go:143 component=operator info="stopping workers"
ts=2020-01-07T16:07:39.67517901Z caller=git.go:138 component=gitchartsync info="stopping sync of git chart sources"

  1. Then this output:

    $ kubectl logs -n fluxcd -f deploy/helm-operator
---
Error from server (NotFound): deployments.extensions "helm-operator" not found
$ helm ls -a
---
NAME                            REVISION    UPDATED                     STATUS      CHART                               APP VERSION NAMESPACE   
...
flux                            2           Tue Jan  7 08:07:35 2020    DEPLOYED    flux-1.0.0                          1.17.0      fluxcd      
helm-operator                   2           Tue Jan  7 08:07:38 2020    DEPLOYED    flux-0.4.0                          1.8.0       fluxcd

  2. This is what I get if I try to deploy previous chart version v0.3.0

    kubectl delete crd/helmreleases.helm.fluxcd.io
helm install fluxcd/helm-operator \
--name helm-operator \
--namespace fluxcd \
--set createCRD=true \
--set git.ssh.secretName=flux-git-deploy \
--set chartsSyncInterval=2m \
--version 0.3.0 \
--atomic
W0107 16:23:57.909586       6 client_config.go:549] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
ts=2020-01-07T16:23:57.935741932Z caller=helm.go:88 component=helm info="connected to Tiller" version="sem_ver:\"v2.15.2\" git_commit:\"8dce272473e5f2a7bf58ce79bb5c3691db54c96b\" git_tree_state:\"clean\" " host=tiller-deploy.kube-system:44134 options="{Host: Port: Namespace:kube-system TLSVerify:false TLSEnable:false TLSKey:/etc/fluxd/helm/tls.key TLSCert:/etc/fluxd/helm/tls.crt TLSCACert: TLSHostname:}"
ts=2020-01-07T16:23:57.936047655Z caller=operator.go:95 component=operator info="setting up event handlers"
ts=2020-01-07T16:23:57.936073815Z caller=operator.go:115 component=operator info="event handlers set up"
ts=2020-01-07T16:23:57.936088275Z caller=main.go:224 component=helm-operator info="waiting for informer caches to sync"
ts=2020-01-07T16:23:58.038453381Z caller=main.go:229 component=helm-operator info="informer caches synced"
ts=2020-01-07T16:23:58.043244259Z caller=chartsync.go:152 component=chartsync info="starting git chart sync loop"
ts=2020-01-07T16:23:58.043335881Z caller=operator.go:127 component=operator info="starting operator"
ts=2020-01-07T16:23:58.043357571Z caller=operator.go:129 component=operator info="starting workers"
ts=2020-01-07T16:23:58.070187237Z caller=server.go:42 component=daemonhttp info="starting HTTP server on :3030"
ts=2020-01-07T16:23:58.522638626Z caller=checkpoint.go:24 component=checkpoint msg="up to date" latest=0.10.1

ts=2020-01-07T16:26:37.759007813Z caller=release.go:184 component=release info="processing release elasticsearch (as 5a62959e-4929-4b50-9b6f-c9934421079f)" action=CREATE options="{DryRun:true ReuseName:false}" timeout=300s
ts=2020-01-07T16:26:38.691069159Z caller=release.go:184 component=release info="processing release cert-manager (as 8bec9c76-d4b6-43fe-9b2f-4543ae305c11)" action=CREATE options="{DryRun:true ReuseName:false}" timeout=300s
ts=2020-01-07T16:26:40.541762763Z caller=release.go:184 component=release info="processing release flux (as 9f4dd09a-5b77-4706-b4de-b6e786db18d3)" action=CREATE options="{DryRun:true ReuseName:false}" timeout=300s
ts=2020-01-07T16:26:42.26749586Z caller=release.go:184 component=release info="processing release helm-operator (as 3689128a-5594-4c71-979e-ba8d6d42ae7b)" action=CREATE options="{DryRun:true ReuseName:false}" timeout=300s
ts=2020-01-07T16:26:42.336079774Z caller=release.go:184 component=release info="processing release helm-operator (as helm-operator)" action=UPDATE options="{DryRun:false ReuseName:false}" timeout=300s
ts=2020-01-07T16:26:43.131833547Z caller=release.go:368 component=release output="Error from server (Forbidden): configmaps \"helm-operator-flux-ssh-config\" is forbidden: User \"system:serviceaccount:fluxcd:helm-operator\" cannot get resource \"configmaps\" in API group \"\" in the namespace \"fluxcd\"\nError from server (Forbidden): clusterroles.rbac.authorization.k8s.io \"helm-operator-flux\" is forbidden: User \"system:serviceaccount:fluxcd:helm-operator\" cannot get resource \"clusterroles\" in API group \"rbac.authorization.k8s.io\" at the cluster scope\nError from server (Forbidden): services \"helm-operator-flux\" is forbidden: User \"system:serviceaccount:fluxcd:helm-operator\" cannot get resource \"services\" in API group \"\" in the namespace \"fluxcd\"\nError from server (Forbidden): deployments.extensions \"helm-operator-flux-memcached\" is forbidden: User \"system:serviceaccount:fluxcd:helm-operator\" cannot get resource \"deployments\" in API group \"extensions\" in the namespace \"fluxcd\"\nError from server (Forbidden): secrets \"helm-operator-flux-git-deploy\" is forbidden: User \"system:serviceaccount:fluxcd:helm-operator\" cannot get resource \"secrets\" in API group \"\" in the namespace \"fluxcd\"\nError from server (Forbidden): serviceaccounts \"helm-operator-flux\" is forbidden: User \"system:serviceaccount:fluxcd:helm-operator\" cannot get resource \"serviceaccounts\" in API group \"\" in the namespace \"fluxcd\"\nError from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io \"helm-operator-flux\" is forbidden: User \"system:serviceaccount:fluxcd:helm-operator\" cannot get resource \"clusterrolebindings\" in API group \"rbac.authorization.k8s.io\" at the cluster scope\nError from server (Forbidden): services \"helm-operator-flux-memcached\" is forbidden: User \"system:serviceaccount:fluxcd:helm-operator\" cannot get resource \"services\" in API group \"\" in the namespace \"fluxcd\"\nError from server (Forbidden): deployments.extensions \"helm-operator-flux\" is forbidden: User \"system:serviceaccount:fluxcd:helm-operator\" cannot get resource \"deployments\" in API group \"extensions\" in the namespace \"fluxcd\"\n" err="exit status 1"
ts=2020-01-07T16:26:43.134020646Z caller=chartsync.go:405 component=chartsync warning="could not update the release revision" resource=fluxcd:helmrelease/helm-operator err="helmreleases.helm.fluxcd.io \"helm-operator\" is forbidden: User \"system:serviceaccount:fluxcd:helm-operator\" cannot get resource \"helmreleases\" in API group \"helm.fluxcd.io\" in the namespace \"fluxcd\""
ts=2020-01-07T16:26:43.134881608Z caller=chartsync.go:408 component=chartsync warning="could not update the values checksum" namespace=fluxcd resource=helm-operator err="helmreleases.helm.fluxcd.io \"helm-operator\" is forbidden: User \"system:serviceaccount:fluxcd:helm-operator\" cannot get resource \"helmreleases\" in API group \"helm.fluxcd.io\" in the namespace \"fluxcd\""
E0107 16:26:43.136782       6 event.go:191] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"helm-operator.15e7a7896f42a01e", GenerateName:"", Namespace:"fluxcd", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"HelmRelease", Namespace:"fluxcd", Name:"helm-operator", UID:"3689128a-5594-4c71-979e-ba8d6d42ae7b", APIVersion:"helm.fluxcd.io/v1", ResourceVersion:"18334", FieldPath:""}, Reason:"ChartSynced", Message:"Chart managed by HelmRelease processed", Source:v1.EventSource{Component:"helm-operator", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbf7d4a30c815c21e, ext:165251868443, loc:(*time.Location)(0x26dec00)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbf7d4a30c815c21e, ext:165251868443, loc:(*time.Location)(0x26dec00)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events is forbidden: User "system:serviceaccount:fluxcd:helm-operator" cannot create resource "events" in API group "" in the namespace "fluxcd"' (will not retry!)
ts=2020-01-07T16:26:43.347194222Z caller=status.go:72 component=statusupdater namespace=monitoring resource=prometheus-operator err="helmreleases.helm.fluxcd.io \"prometheus-operator\" is forbidden: User \"system:serviceaccount:fluxcd:helm-operator\" cannot get resource \"helmreleases\" in API group \"helm.fluxcd.io\" in the namespace \"monitoring\""
ts=2020-01-07T16:26:43.72749931Z caller=main.go:248 exiting...=terminated
ts=2020-01-07T16:26:43.73946846Z caller=status.go:79 component=statusupdater loop=stopping err=null
ts=2020-01-07T16:26:43.739679212Z caller=server.go:59 component=daemonhttp info="HTTP server stopped"
ts=2020-01-07T16:26:43.739706733Z caller=operator.go:139 component=operator info="stopping workers"
ts=2020-01-07T16:26:43.739728553Z caller=chartsync.go:258 component=chartsync stopping=true
  1. And similar output with v0.3.0 chart: 
    $ kubectl logs -n fluxcd -f deploy/helm-operator
---
Error from server (NotFound): deployments.extensions "helm-operator" not found
$ kubectl logs -n fluxcd -f deploy/helm-operator
---
Error from server (NotFound): deployments.extensions "helm-operator" not found
$ helm ls -a
---
NAME                            REVISION    UPDATED                     STATUS      CHART                               APP VERSION NAMESPACE   
...
NAME                            REVISION    UPDATED                     STATUS      CHART                               APP VERSION NAMESPACE   
flux                            2           Tue Jan  7 08:07:35 2020    DEPLOYED    flux-1.0.0                          1.17.0      fluxcd      
helm-operator                   2           Tue Jan  7 08:26:42 2020    DEPLOYED    flux-0.3.0                          1.5.0       fluxcd  
...
demisx commented 4 years ago

Sorry, I found the problem. I had a typo in the helm release manifest file for the helm-operator where the chart name was chart.name: flux instead of chart.name: helm-operator, which caused an older flux 0.4.0 being deployed alongside with the new 1.0.0. Thank you @stefanprodan @hiddeco. You are great and spot on as always!