search

mongodb / mongodb-atlas-kubernetes

MongoDB Atlas Kubernetes Operator - Manage your MongoDB Atlas clusters from Kubernetes
http://www.mongodb.com/cloud/atlas
Apache License 2.0
146 stars 75 forks source link

The ServiceAccount for atlas-operator cannot create Event in the user's namespace. #348

Closed denist-huma closed 2 years ago

denist-huma commented 2 years ago

What did you do to encounter the bug? Steps to reproduce the behavior:

While creating AtlasDatabaseUser "operator-sandbox/todo-app". User "system:serviceaccount:atlas-operator:mongodb-atlas-operator" cannot create resource "events" in API group "" in the namespace "operator-sandbox"

2021-11-25T10:21:52.394Z        INFO    controllers.AtlasDatabaseUser   -> Starting AtlasDatabaseUser reconciliation    {"atlasdatabaseuser": "operator-sandbox/todo-app", "spec": {"projectRef":{"name":"hu-sandbox","namespace":""},"databaseName":"admin","roles":[{"roleName":"readWrite","databaseName":"todo-app"}],"scopes":[{"name":"hu-uk-compose-operator","type":"CLUSTER"}],"passwordSecretRef":{"name":"hu-sandbox-todo-app-password"},"username":"todo-app"}, "status": {"conditions":[{"type":"Ready","status":"False","lastTransitionTime":"2021-11-25T10:20:46Z"},{"type":"DatabaseUserReady","status":"False","lastTransitionTime":"2021-11-25T10:20:46Z","reason":"ClustersAppliedDatabaseUsersChanges","message":"0 out of 1 clusters have applied database user changes"}],"observedGeneration":1,"passwordVersion":"7879809"}}
2021-11-25T10:21:52.394Z        DEBUG   controllers.AtlasDatabaseUser   AtlasProject connection Secret is not specified - using the Operator one: atlas-operator/mongodb-atlas-operator-api-key {"atlasdatabaseuser": "operator-sandbox/todo-app"}
2021-11-25T10:21:52.705Z        DEBUG   controllers.AtlasDatabaseUser   HTTP Request (GET) https://cloud.mongodb.com/api/atlas/v1.0/groups/5f1313011621fe3c7268a8b1/clusters/hu-uk-compose-operator [time (ms): 305, status: 200]     {"atlasdatabaseuser": "operator-sandbox/todo-app"}
2021-11-25T10:21:52.900Z        DEBUG   controllers.AtlasDatabaseUser   HTTP Request (GET) https://cloud.mongodb.com/api/atlas/v1.0/groups/5f1313011621fe3c7268a8b1/databaseUsers/admin/todo-app [time (ms): 195, status: 200]        {"atlasdatabaseuser": "operator-sandbox/todo-app"}
2021-11-25T10:21:53.096Z        DEBUG   controllers.AtlasDatabaseUser   HTTP Request (GET) https://cloud.mongodb.com/api/atlas/v1.0/groups/5f1313011621fe3c7268a8b1/clusters [time (ms): 195, status: 200]   {"atlasdatabaseuser": "operator-sandbox/todo-app"}
2021-11-25T10:21:53.296Z        DEBUG   controllers.AtlasDatabaseUser   HTTP Request (GET) https://cloud.mongodb.com/api/atlas/v1.0/groups/5f1313011621fe3c7268a8b1/clusters/hu-uk-compose-operator/status [time (ms): 198, status: 200]      {"atlasdatabaseuser": "operator-sandbox/todo-app"}
2021-11-25T10:21:53.296Z        DEBUG   controllers.AtlasDatabaseUser   1 out of 1 clusters have applied database user changes  {"atlasdatabaseuser": "operator-sandbox/todo-app"}
2021-11-25T10:21:53.523Z        DEBUG   controllers.AtlasDatabaseUser   HTTP Request (GET) https://cloud.mongodb.com/api/atlas/v1.0/groups/5f1313011621fe3c7268a8b1/clusters [time (ms): 226, status: 200]   {"atlasdatabaseuser": "operator-sandbox/todo-app"}
2021-11-25T10:21:53.535Z        DEBUG   controllers.AtlasDatabaseUser   Ensured connection Secret up-to-date    {"atlasdatabaseuser": "operator-sandbox/todo-app", "secretname": "hu-sandbox-hu-uk-compose-operator-todo-app"}
2021-11-25T10:21:53.535Z        INFO    controllers.AtlasDatabaseUser   Status update   {"atlasdatabaseuser": "operator-sandbox/todo-app", "lastCondition": {"type":"Ready","status":"True","lastTransitionTime":null}}
2021-11-25T10:21:53.535Z        DEBUG   controller-runtime.manager.events       Normal  {"object": {"kind":"AtlasDatabaseUser","namespace":"operator-sandbox","name":"todo-app","uid":"d1b55647-16d0-42fd-8b08-4452e85db715","apiVersion":"atlas.mongodb.com/v1","resourceVersion":"7880233"}, "reason": "ConnectionSecretsEnsured", "message": "Connection Secrets were created/updated: hu-sandbox-hu-uk-compose-operator-todo-app"}
2021-11-25T10:21:53.535Z        DEBUG   controller-runtime.manager.events       Normal  {"object": {"kind":"AtlasDatabaseUser","namespace":"operator-sandbox","name":"todo-app","uid":"d1b55647-16d0-42fd-8b08-4452e85db715","apiVersion":"atlas.mongodb.com/v1","resourceVersion":"7880233"}, "reason": "Ready", "message": ""}
E1125 10:21:53.542031       1 event.go:264] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"todo-app.16bac2e761b5a5c6", GenerateName:"", Namespace:"operator-sandbox", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"AtlasDatabaseUser", Namespace:"operator-sandbox", Name:"todo-app", UID:"d1b55647-16d0-42fd-8b08-4452e85db715", APIVersion:"atlas.mongodb.com/v1", ResourceVersion:"7880233", FieldPath:""}, Reason:"ConnectionSecretsEnsured", Message:"Connection Secrets were created/updated: hu-sandbox-hu-uk-compose-operator-todo-app", Source:v1.EventSource{Component:"AtlasDatabaseUser", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xc05ff6d05fe55bc6, ext:54773328760176, loc:(*time.Location)(0x22a9380)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xc05ff6d05fe55bc6, ext:54773328760176, loc:(*time.Location)(0x22a9380)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events is forbidden: User "system:serviceaccount:atlas-operator:mongodb-atlas-operator" cannot create resource "events" in API group "" in the namespace "operator-sandbox"' (will not retry!)
E1125 10:21:53.544811       1 event.go:264] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"todo-app.16bac2e761b9fb81", GenerateName:"", Namespace:"operator-sandbox", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"AtlasDatabaseUser", Namespace:"operator-sandbox", Name:"todo-app", UID:"d1b55647-16d0-42fd-8b08-4452e85db715", APIVersion:"atlas.mongodb.com/v1", ResourceVersion:"7880233", FieldPath:""}, Reason:"Ready", Message:"", Source:v1.EventSource{Component:"AtlasDatabaseUser", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xc05ff6d05fe9b181, ext:54773329044317, loc:(*time.Location)(0x22a9380)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xc05ff6d05fe9b181, ext:54773329044317, loc:(*time.Location)(0x22a9380)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events is forbidden: User "system:serviceaccount:atlas-operator:mongodb-atlas-operator" cannot create resource "events" in API group "" in the namespace "operator-sandbox"' (will not retry!)

What did you expect? A clear and concise description of what you expected to happen.

Put an event I guess.

What happened instead? A clear and concise description of what happened instead

nothing

Screenshots If applicable, add screenshots to help explain your problem.

Operator Information

Kubernetes Cluster Information

Additional context Add any other context about the problem here.

If possible, please include:

leo-ri commented 2 years ago

hi, @denist-huma may I ask quay.io/denistrofimov/mongodb-atlas-kubernetes:v0.6.1-dt - was this image built from the master branch? ( = what's the difference between versions with 0.6.1? can we get the commitID it was built on?)

Is this a cluster wide or multinamespaced configuration (was watched_namespace used)? was the configuration from deploy/... directory? It would really help if you can provide operator deployment yaml configuration, cluster role/service account/binding yaml what namespaces have atlasproject/cluster/user? Thank you

denist-huma commented 2 years ago

hi, @denist-huma may I ask quay.io/denistrofimov/mongodb-atlas-kubernetes:v0.6.1-dt - was this image built from the master branch? ( = what's the difference between versions with 0.6.1? can we get the commitID it was built on?)

You are welcome, @leo-ri The 1st URL you listed leads to https://quay.io/repository/denistrofimov/mongodb-atlas-kubernetes?tag=v0.6.1-dt&tab=tags The commit corresponds to the tag of my fork, https://github.com/denist-huma/mongodb-atlas-kubernetes/releases/tag/v0.6.1-dt

Is this a cluster wide or multinamespaced configuration (was watched_namespace used)? was the configuration from deploy/... directory? It would really help if you can provide operator deployment yaml configuration, cluster role/service account/binding yaml what namespaces have atlasproject/cluster/user? Thank you

The cluster-wide, watched_namespace is not used, I have no clue about it, how it can help me? That was related to the 0.2.2 version of the chart

$ helm list -n atlas-operator
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                           APP VERSION
atlas-operator  atlas-operator  6               2021-11-24 15:38:30.300030088 +0300 MSK deployed        mongodb-atlas-operator-0.2.2    0.5.0

$ helm get values -n atlas-operator atlas-operator
USER-SUPPLIED VALUES:
affinity: {}
atlasURI: https://cloud.mongodb.com
disableAtlasClusterReconciler: "true"
fullnameOverride: ""
image:
  pullPolicy: Always
  repository: quay.io/denistrofimov/mongodb-atlas-kubernetes
  tag: v0.6.1-dt
imagePullSecrets:
- name: regcred
mongodb-atlas-operator-crds:
  enabled: true
nameOverride: ""
nodeSelector: {}
podAnnotations: {}
podSecurityContext:
  runAsNonRoot: true
  runAsUser: 2000
resources:
  limits:
    cpu: 500m
    memory: 256Mi
  requests:
    cpu: 100m
    memory: 50Mi
securityContext:
  allowPrivilegeEscalation: false
serviceAccount:
  annotations: {}
  create: true
  name: ""
tolerations: []
watchNamespaces: ""

I searched in the output to help you.

helm get manifest -n atlas-operator atlas-operator > manifest.out.yaml

The problem I expect is in the role mongodb-atlas-operator-leader-election-role IMHO. That is the only one mention the "events" resource.

The root of the error is that role mongodb-atlas-operator-leader-election-role is in the namespace: atlas-operator But the AtlasDatabaseUser is in the namespace "operator-sandbox" as I said in the first message.

While creating AtlasDatabaseUser "operator-sandbox/todo-app". User "system:serviceaccount:atlas-operator:mongodb-atlas-operator" cannot create resource "events" in API group "" in the namespace "operator-sandbox"

That I took from a live deployment:

# $ kg -o yaml role mongodb-atlas-operator-leader-election-role -n atlas-operator
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  annotations:
    meta.helm.sh/release-name: atlas-operator
    meta.helm.sh/release-namespace: atlas-operator
  creationTimestamp: "2021-07-07T16:05:53Z"
  labels:
    app.kubernetes.io/managed-by: Helm
  name: mongodb-atlas-operator-leader-election-role
  namespace: atlas-operator
  resourceVersion: "178576170"
  selfLink: /apis/rbac.authorization.k8s.io/v1/namespaces/atlas-operator/roles/mongodb-atlas-operator-leader-election-role
  uid: 2b9b0b66-a08b-4bbe-a45e-c52168e549e5
rules:
- apiGroups:
  - ""
  - coordination.k8s.io
  resources:
  - configmaps
  - leases
  verbs:
  - get
  - list
  - watch
  - create
  - update
  - patch
  - delete
- apiGroups:
  - ""
  resources:
  - events
  verbs:
  - create
  - patch
leo-ri commented 2 years ago

@denist-huma, The version in this repo (0.6.0+) does not match the version of the operator chart 0.2.2, repo version requires the use of chart mongodb-atlas-operator-0.2.4 or higher (0.2.5). Reconciliation events were introduced later in operator 0.6.0 version and chart version 0.2.2 does not have the right permissions for the serviceaccount, permissions were added in 0.2.4-0.2.5 atlas-operator chart

denist-huma commented 2 years ago

@leo-ri hey thanks for the suggestion. I switched to my updated fork of the chart now https://github.com/denist-huma/helm-charts/tree/compose-operator-dep.