search

mongodb / mongodb-atlas-kubernetes

MongoDB Atlas Kubernetes Operator - Manage your MongoDB Atlas clusters from Kubernetes
http://www.mongodb.com/cloud/atlas
Apache License 2.0
146 stars 75 forks source link

Cross namespace watching for GlobalAPIKey doesn't work #600

Closed sunchill06 closed 1 year ago

sunchill06 commented 2 years ago

What did you do to encounter the bug? I can still see the same issue with GlobalAPIKey. I thought this was fixed in https://github.com/mongodb/mongodb-atlas-kubernetes/pull/282

{"level":"INFO","time":"2022-07-12T09:04:54.662Z","msg":"Status update","atlasproject":"ns2/test-project","lastCondition":{"type":"ProjectReady","status":"False","lastTransitionTime":null,"reason":"AtlasCredentialsNotProvided","message":"Secret \"mongodb-atlas-operator-api-key\" not found"}}

Am I missing something here? 😕

What did you expect? Expected it to work fine with GlobalAPIKey as per https://github.com/mongodb/mongodb-atlas-kubernetes/pull/282

What happened instead? The operator is unable to list/read secret from its own namespace if WATCH_NAMESPACE is defined and is a different namespace.

Screenshots N/A

Operator Information

Kubernetes Cluster Information

Additional context Add any other context about the problem here.

If possible, please include:

igor-karpukhin commented 2 years ago

Hello @sunchill06. Thanks for your issue. You can fix that by providing multiple namespaces, separating them by commas like this: WATCH_NAMESPACE=ns1,ns2,...nsN

igor-karpukhin commented 2 years ago

Hi @sunchill06 , is this still an issue? Did you try this approach by specifying what namespaces to watch?

sunchill06 commented 2 years ago

Hi @igor-karpukhin, sorry about the delayed response. It still doesn't work.

❯ helm upgrade --install atlas-operator mongodb/mongodb-atlas-operator --namespace=atlas-operator-ga --set watchNamespaces=atlas-operator-ga,platform  --set mongodb-atlas-operator-crds.enabled=false --version "1.1.0"  
Error: failed parsing --set data: key "platform" has no value

I tried following as well, but that also doesn't work.


❯ helm upgrade --install atlas-operator mongodb/mongodb-atlas-operator --namespace=atlas-operator-ga --set watchNamespaces="atlas-operator-ga\,platform" --set mongodb-atlas-operator-crds.enabled=false --version "1.1.0"
Error: UPGRADE FAILED: failed to create resource: namespaces "atlas-operator-ga,platform" not found```
igor-karpukhin commented 2 years ago

Hi, @sunchill06. I see the problem. The current way the operator is installed with helm charts, it tries to create Role and RoleBinding from the watchNamespace parameter, which will fail. What you can do is just install the operator into the namespace, and then edit the deployment. Modify the WATCH_NAMESPACE env variable to contain two comma-separated namespaces you'd like the operator to watch.

sunchill06 commented 2 years ago

Thanks @igor-karpukhin. Your suggestions works fine. But there are some downsides to it:

Do you think this can be fixed in the chart itself and we can simply pass multiple namespaces in against WATCH_NAMESPACE variable in values.yaml itself? I am unable to find any way to automatically update the deployment to include all the namespaces that I want to watch.

sunchill06 commented 2 years ago

@igor-karpukhin @fabritsius I would also like to draw your attention to following issue when using multiple WATCH_NAMESPACE(s)

{"level":"INFO","time":"2022-08-31T11:50:48.892Z","msg":"starting with configuration","config":{"AtlasDomain":"https://cloud.mongodb.com/","EnableLeaderElection":true,"MetricsAddr":":8080","Namespace":"","WatchedNamespaces":{"mongodb":true,"platform":true},"ProbeAddr":":8081","GlobalAPISecret":{"Namespace":"mongodb","Name":"mongodb-atlas-operator-api-key"},"LogLevel":"info","LogEncoder":"json"},"version":"v1.1.0"}
{"level":"INFO","time":"2022-08-31T11:50:48.892Z","msg":"MongoDB Atlas Operator version v1.1.0"}
I0831 11:50:49.944171       1 request.go:665] Waited for 1.040902034s due to client-side throttling, not priority and fairness, request: GET:https://x.x.x.x:443/apis/certificates.k8s.io/v1?timeout=32s
{"level":"INFO","time":"2022-08-31T11:50:51.348Z","msg":"Metrics server is starting to listen","addr":":8080"}
{"level":"INFO","time":"2022-08-31T11:50:51.348Z","msg":"starting manager"}
{"level":"INFO","time":"2022-08-31T11:50:51.348Z","msg":"Starting server","path":"/metrics","kind":"metrics","addr":"[::]:8080"}
{"level":"INFO","time":"2022-08-31T11:50:51.349Z","msg":"Starting server","kind":"health probe","addr":"[::]:8081"}
I0831 11:50:51.349062       1 leaderelection.go:248] attempting to acquire leader lease mongodb/06d035fb.mongodb.com...
I0831 11:51:07.758328       1 leaderelection.go:258] successfully acquired lease mongodb/06d035fb.mongodb.com
{"level":"INFO","time":"2022-08-31T11:51:07.758Z","msg":"Starting EventSource","source":"kind source: *v1.AtlasDeployment"}
{"level":"INFO","time":"2022-08-31T11:51:07.758Z","msg":"Starting EventSource","source":"kind source: *v1.AtlasBackupSchedule"}
{"level":"INFO","time":"2022-08-31T11:51:07.758Z","msg":"Starting EventSource","source":"kind source: *v1.AtlasBackupPolicy"}
{"level":"INFO","time":"2022-08-31T11:51:07.758Z","msg":"Starting Controller"}
{"level":"INFO","time":"2022-08-31T11:51:07.758Z","msg":"Starting EventSource","source":"kind source: *v1.AtlasProject"}
{"level":"INFO","time":"2022-08-31T11:51:07.758Z","msg":"Starting EventSource","source":"kind source: *v1.Secret"}
{"level":"INFO","time":"2022-08-31T11:51:07.758Z","msg":"Starting EventSource","source":"kind source: *v1.AtlasDatabaseUser"}
{"level":"INFO","time":"2022-08-31T11:51:07.758Z","msg":"Starting EventSource","source":"kind source: *v1.Secret"}
{"level":"INFO","time":"2022-08-31T11:51:07.758Z","msg":"Starting Controller"}
{"level":"INFO","time":"2022-08-31T11:51:07.758Z","msg":"Starting Controller"}
W0831 11:51:07.760855       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: failed to list *v1.AtlasBackupPolicy: atlasbackuppolicies.atlas.mongodb.com is forbidden: User "system:serviceaccount:mongodb:mongodb-atlas-operator" cannot list resource "atlasbackuppolicies" in API group "atlas.mongodb.com" at the cluster scope
E0831 11:51:07.760905       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: Failed to watch *v1.AtlasBackupPolicy: failed to list *v1.AtlasBackupPolicy: atlasbackuppolicies.atlas.mongodb.com is forbidden: User "system:serviceaccount:mongodb:mongodb-atlas-operator" cannot list resource "atlasbackuppolicies" in API group "atlas.mongodb.com" at the cluster scope
W0831 11:51:07.761343       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: failed to list *v1.AtlasDatabaseUser: atlasdatabaseusers.atlas.mongodb.com is forbidden: User "system:serviceaccount:mongodb:mongodb-atlas-operator" cannot list resource "atlasdatabaseusers" in API group "atlas.mongodb.com" at the cluster scope
E0831 11:51:07.761377       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: Failed to watch *v1.AtlasDatabaseUser: failed to list *v1.AtlasDatabaseUser: atlasdatabaseusers.atlas.mongodb.com is forbidden: User "system:serviceaccount:mongodb:mongodb-atlas-operator" cannot list resource "atlasdatabaseusers" in API group "atlas.mongodb.com" at the cluster scope
W0831 11:51:07.761458       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: failed to list *v1.AtlasBackupSchedule: atlasbackupschedules.atlas.mongodb.com is forbidden: User "system:serviceaccount:mongodb:mongodb-atlas-operator" cannot list resource "atlasbackupschedules" in API group "atlas.mongodb.com" at the cluster scope
E0831 11:51:07.761478       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: Failed to watch *v1.AtlasBackupSchedule: failed to list *v1.AtlasBackupSchedule: atlasbackupschedules.atlas.mongodb.com is forbidden: User "system:serviceaccount:mongodb:mongodb-atlas-operator" cannot list resource "atlasbackupschedules" in API group "atlas.mongodb.com" at the cluster scope
W0831 11:51:07.761489       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:mongodb:mongodb-atlas-operator" cannot list resource "secrets" in API group "" at the cluster scope

It seems like controller is looking to create cluster scoped cache and not multinamespacedcache.

igor-karpukhin commented 2 years ago

Hello @sunchill06 , sorry for the late response. Thanks for opening a PR, and your findings in multiple watched namespaces, we will take a look at it shortly!

igor-karpukhin commented 1 year ago

Hi @sunchill06. Sorry for late reply. We added multi-namespace support to our operator (https://github.com/mongodb/mongodb-atlas-kubernetes/pull/705). Your PR to helm-charts can be merged, but before that, please also modify examples in README.md

sunchill06 commented 1 year ago

Thanks @igor-karpukhin. Sorry about the late reply. I have taken care of this. https://github.com/mongodb/helm-charts/pull/167#issuecomment-1308982321. Hope its alright.

igor-karpukhin commented 1 year ago

Fixed by https://github.com/mongodb/helm-charts/pull/167