Closed vfouqueron closed 1 month ago
Hello!
Somewhere, at the beginning of the logs, the Operator should tell you the issue which blocks the main loop.
Can you share logs of the Operator from the beginning?
Best Regards, Adam.
Thank you for your quick answer.
Here are the logs :
2024-07-22T07:12:01Z INF nice to meet you operator-id=2qr99
2024-07-22T07:12:01Z INF Operator Feature agency-poll (deployment.feature.agency-poll) is disabled. operator-id=2qr99
2024-07-22T07:12:01Z INF Operator Feature async-backup-creation (deployment.feature.async-backup-creation) is enabled. operator-id=2qr99
2024-07-22T07:12:01Z INF Operator Feature backup-cleanup (deployment.feature.backup-cleanup) is disabled. operator-id=2qr99
2024-07-22T07:12:01Z INF Operator Feature deployment-spec-defaults-restore (deployment.feature.deployment-spec-defaults-restore) is enabled. operator-id=2qr99
2024-07-22T07:12:01Z INF Operator Feature encryption-rotation (deployment.feature.encryption-rotation) is disabled. operator-id=2qr99
2024-07-22T07:12:01Z INF Operator Feature enforced-resign-leadership (deployment.feature.enforced-resign-leadership) is enabled. operator-id=2qr99
2024-07-22T07:12:01Z INF Operator Feature ephemeral-volumes (deployment.feature.ephemeral-volumes) is disabled. operator-id=2qr99
2024-07-22T07:12:01Z INF Operator Feature failover-leadership (deployment.feature.failover-leadership) is disabled. operator-id=2qr99
2024-07-22T07:12:01Z INF Operator Feature force-rebuild-out-synced-shards (deployment.feature.force-rebuild-out-synced-shards) is disabled. operator-id=2qr99
2024-07-22T07:12:01Z INF Operator Feature graceful-shutdown (deployment.feature.graceful-shutdown) is enabled. operator-id=2qr99
2024-07-22T07:12:01Z INF Operator Feature init-containers-copy-resources (deployment.feature.init-containers-copy-resources) is enabled. operator-id=2qr99
2024-07-22T07:12:01Z INF Operator Feature init-containers-upscale-resources (deployment.feature.init-containers-upscale-resources) is enabled. operator-id=2qr99
2024-07-22T07:12:01Z INF Operator Feature jwt-rotation (deployment.feature.jwt-rotation) is enabled. operator-id=2qr99
2024-07-22T07:12:01Z INF Operator Feature local-storage.pass-reclaim-policy (deployment.feature.local-storage.pass-reclaim-policy) is disabled. operator-id=2qr99
2024-07-22T07:12:01Z INF Operator Feature local-volume-replacement-check (deployment.feature.local-volume-replacement-check) is disabled. operator-id=2qr99
2024-07-22T07:12:01Z INF Operator Feature maintenance (deployment.feature.maintenance) is enabled. operator-id=2qr99
2024-07-22T07:12:01Z INF Operator Feature metrics-exporter (deployment.feature.metrics-exporter) is enabled. operator-id=2qr99
2024-07-22T07:12:01.094462732Z (deployment.feature.optional-graceful-shutdown) is disabled. operator-id=2qr99
2024-07-22T07:12:01Z INF Operator Feature random-pod-names (deployment.feature.random-pod-names) is disabled. operator-id=2qr99
2024-07-22T07:12:01Z INF Operator Feature rebalancer-v2 (deployment.feature.rebalancer-v2) is disabled. operator-id=2qr99
2024-07-22T07:12:01Z INF Operator Feature restart-policy-always (deployment.feature.restart-policy-always) is disabled. operator-id=2qr99
2024-07-22T07:12:01Z INF Operator Feature secured-containers (deployment.feature.secured-containers) is disabled. operator-id=2qr99
2024-07-22T07:12:01Z INF Operator Feature sensitive-information-protection (deployment.feature.sensitive-information-protection) is disabled. operator-id=2qr99
2024-07-22T07:12:01Z INF Operator Feature short-pod-names (deployment.feature.short-pod-names) is disabled. operator-id=2qr99
2024-07-22T07:12:01Z INF Operator Feature timezone-management (deployment.feature.timezone-management) is disabled. operator-id=2qr99
2024-07-22T07:12:01Z INF Operator Feature tls-rotation (deployment.feature.tls-rotation) is enabled. operator-id=2qr99
2024-07-22T07:12:01Z INF Operator Feature tls-sni (deployment.feature.tls-sni) is enabled. operator-id=2qr99
2024-07-22T07:12:01Z INF Operator Feature upgrade-version-check (deployment.feature.upgrade-version-check) is enabled. operator-id=2qr99
2024-07-22T07:12:01Z INF Operator Feature upgrade-version-check-v2 (deployment.feature.upgrade-version-check-v2) is disabled. operator-id=2qr99
2024-07-22T07:12:01Z INF Operator Feature version.3-10 (deployment.feature.version.3-10) is enabled. operator-id=2qr99
2024-07-22T07:12:01Z INF Starting arangodb-operator (Community), version 1.2.41 build bc898c71 operator-id=2qr99 pod-name=arango-arangodb-operator-6c6586d89-2qr99 pod-namespace=arango
2024-07-22T07:12:01Z INF CRD Update not required crd=arangoclustersynchronizations.database.arangodb.com operator-id=2qr99 version=5de0cceda99c91fe708370e9996f8c98
2024-07-22T07:12:01Z INF Update Operations is not allowed. Continue crd=arangomlstorages.ml.arangodb.com operator-id=2qr99 schema=7434ce36a4137aec0f1791ea12ef58f8 version=0784b0c3ed19a07f82be89cf732e688a
2024-07-22T07:12:01Z INF Update Operations is not allowed. Continue crd=graphanalyticsengines.analytics.arangodb.com operator-id=2qr99 schema=bb8e406e376ca72490036183f301abab version=a7cd6bb879625020634bea59274db2e4
2024-07-22T07:12:01Z INF CRD Update not required crd=arangodeployments.database.arangodb.com operator-id=2qr99 version=2b3c736ba875c72a57da26c7b3b754f5
2024-07-22T07:12:01Z INF Create Operations is not allowed but CRD is missing. Continue crd=arangolocalstorages.storage.arangodb.com operator-id=2qr99 version=ba7d9af888270bab37493079eff61855
2024-07-22T07:12:01Z INF CRD Update not required crd=arangodeploymentreplications.replication.database.arangodb.com operator-id=2qr99 version=55f7b3b00702d80708fe29615da6c446
2024-07-22T07:12:01Z INF Update Operations is not allowed. Continue crd=arangomlextensions.ml.arangodb.com operator-id=2qr99 schema=eae959fb737b49dac46f7238c3bb534b version=53cff97ca37fa8e1878e879386a41cc3
2024-07-22T07:12:01Z INF Update Operations is not allowed. Continue crd=arangomlcronjobs.ml.arangodb.com operator-id=2qr99 schema=c53a044e9decb09c4e5ff12b4ad20dbc version=071152afd23e0c6212e6ff16de3ec2fb
2024-07-22T07:12:01Z INF CRD Update not required crd=arangobackups.backup.arangodb.com operator-id=2qr99 version=c0e6bf20aecad187ac0ac881bb7207ff
2024-07-22T07:12:01Z INF CRD Update not required crd=arangobackuppolicies.backup.arangodb.com operator-id=2qr99 version=f1e80ff799576a674d6f2ebd4a1fca3f
2024-07-22T07:12:01Z INF CRD Update not required crd=arangojobs.apps.arangodb.com operator-id=2qr99 version=adc765de8dab4e31cd715a172b694a68
2024-07-22T07:12:01Z INF CRD Update not required crd=arangomembers.database.arangodb.com operator-id=2qr99 version=dfa84092e1a390a9f6180f194c42ce22
2024-07-22T07:12:01Z INF Update Operations is not allowed. Continue crd=arangomlbatchjobs.ml.arangodb.com operator-id=2qr99 schema=d67b904151fc753f2b9a1dcd174cc908 version=3da979e8cc522c84fa8d7bf3d04599a4
2024-07-22T07:12:01Z INF Update Operations
is not allowed. Continue crd=arangoprofiles.scheduler.arangodb.com operator-id=2qr99 version=3a71d123e7d0bf3902f91ebcdc4cac77
2024-07-22T07:12:02Z INF CRD Update not required crd=arangotasks.database.arangodb.com operator-id=2qr99 version=de60b4008e910ada265f8066551a28e4
2024-07-22T07:12:02Z INF Serving GRPC API on 0.0.0.0:8728 operator-id=2qr99
2024-07-22T07:12:02Z INF Serving HTTP API on 0.0.0.0:8628 operator-id=2qr99
2024-07-22T07:12:02Z INF Serving on 0.0.0.0:8528 operator-id=2qr99
2024-07-22T07:12:02Z INF Event(v1.ObjectReference{Kind:"Deployment", Namespace:"arango", Name:"arango-arangodb-operator", UID:"16227787-caba-49ae-b51f-319c8db08645", APIVersion:"apps/v1", ResourceVersion:"777346611", FieldPath:""}): type: 'Normal' reason: 'Leader Election Skipped' Pod arango-arangodb-operator-6c6586d89-2qr99 is running as leader operator-id=2qr99
2024-07-22T07:12:03Z INF Event(v1.ObjectReference{Kind:"Deployment", Namespace:"arango", Name:"arango-arangodb-operator", UID:"16227787-caba-49ae-b51f-319c8db08645", APIVersion:"apps/v1", ResourceVersion:"777346611", FieldPath:""}): type: 'Normal' reason: 'Leader Election Skipped' Pod arango-arangodb-operator-6c6586d89-2qr99 is running as leader operator-id=2qr99
2024-07-22T07:12:11Z WRN ACS cache not yet initialised name=arangodb namespace=arango operator-id=2qr99
2024-07-22T07:12:12Z ERR Unable to get secret error="secrets \"arangodb-jwt-folder\" not found" name=arangodb namespace=arango operator-id=2qr99
Hello!
Can you please enable the debug log level in the Operator?
--log.level=debug
argument to the Operator Pod.
BR, Adam.
Of course, here are the logs :
2024-07-24T12:22:10Z INF nice to meet you operator-id=q8clz
2024-07-24T12:22:10Z INF Operator Feature agency-poll (deployment.feature.agency-poll) is disabled. operator-id=q8clz
2024-07-24T12:22:10Z INF Operator Feature async-backup-creation (deployment.feature.async-backup-creation) is enabled. operator-id=q8clz
2024-07-24T12:22:10Z INF Operator Feature backup-cleanup (deployment.feature.backup-cleanup) is disabled. operator-id=q8clz
2024-07-24T12:22:10Z INF Operator Feature deployment-spec-defaults-restore (deployment.feature.deployment-spec-defaults-restore) is enabled. operator-id=q8clz
2024-07-24T12:22:10Z INF Operator Feature encryption-rotation (deployment.feature.encryption-rotation) is disabled. operator-id=q8clz
2024-07-24T12:22:10Z INF Operator Feature enforced-resign-leadership (deployment.feature.enforced-resign-leadership) is enabled. operator-id=q8clz
2024-07-24T12:22:10Z INF Operator Feature ephemeral-volumes (deployment.feature.ephemeral-volumes) is disabled. operator-id=q8clz
2024-07-24T12:22:10Z INF Operator Feature failover-leadership (deployment.feature.failover-leadership) is disabled. operator-id=q8clz
2024-07-24T12:22:10Z INF Operator Feature force-rebuild-out-synced-shards (deployment.feature.force-rebuild-out-synced-shards) is disabled. operator-id=q8clz
2024-07-24T12:22:10Z INF Operator Feature graceful-shutdown (deployment.feature.graceful-shutdown) is enabled. operator-id=q8clz
2024-07-24T12:22:10Z INF Operator Feature init-containers-copy-resources (deployment.feature.init-containers-copy-resources) is enabled. operator-id=q8clz
2024-07-24T12:22:10Z INF Operator Feature init-containers-upscale-resources (deployment.feature.init-containers-upscale-resources) is enabled. operator-id=q8clz
2024-07-24T12:22:10Z INF Operator Feature jwt-rotation (deployment.feature.jwt-rotation) is enabled. operator-id=q8clz
2024-07-24T12:22:10Z INF Operator Feature local-storage.pass-reclaim-policy (deployment.feature.local-storage.pass-reclaim-policy) is disabled. operator-id=q8clz
2024-07-24T12:22:10Z INF Operator Feature local-volume-replacement-check (deployment.feature.local-volume-replacement-check) is disabled. operator-id=q8clz
2024-07-24T12:22:10Z INF Operator Feature maintenance (deployment.feature.maintenance) is enabled. operator-id=q8clz
2024-07-24T12:22:10Z INF Operator Feature metrics-exporter (deployment.feature.metrics-exporter) is enabled. operator-id=q8clz
2024-07-24T12:22:10Z INF Operator Feature optional-graceful-shutdown (deployment.feature.optional-graceful-shutdown) is disabled. operator-id=q8clz
2024-07-24T12:22:10Z INF Operator Feature random-pod-names (deployment.feature.random-pod-names) is disabled. operator-id=q8clz
2024-07-24T12:22:10Z INF Operator Feature rebalancer-v2 (deployment.feature.rebalancer-v2) is disabled. operator-id=q8clz
2024-07-24T12:22:10Z INF Operator Feature restart-policy-always (deployment.feature.restart-policy-always) is disabled. operator-id=q8clz
2024-07-24T12:22:10Z INF Operator Feature secured-containers (deployment.feature.secured-containers) is disabled. operator-id=q8clz
2024-07-24T12:22:10Z INF Operator Feature sensitive-information-protection (deployment.feature.sensitive-information-protection) is disabled. operator-id=q8clz
2024-07-24T12:22:10Z INF Operator Feature short-pod-names (deployment.feature.short-pod-names) is disabled. operator-id=q8clz
2024-07-24T12:22:10Z INF Operator Feature timezone-management (deployment.feature.timezone-management) is disabled. operator-id=q8clz
2024-07-24T12:22:10Z INF Operator Feature tls-rotation (deployment.feature.tls-rotation) is enabled. operator-id=q8clz
2024-07-24T12:22:10Z INF Operator Feature tls-sni (deployment.feature.tls-sni) is enabled. operator-id=q8clz
2024-07-24T12:22:10Z INF Operator Feature upgrade-version-check (deployment.feature.upgrade-version-check) is enabled. operator-id=q8clz
2024-07-24T12:22:10Z INF Operator Feature upgrade-version-check-v2 (deployment.feature.upgrade-version-check-v2) is disabled. operator-id=q8clz
2024-07-24T12:22:10Z INF Operator Feature version.3-10 (deployment.feature.version.3-10) is enabled. operator-id=q8clz
2024-07-24T12:22:10Z INF Starting arangodb-operator (Community), version 1.2.41 build bc898c71 operator-id=q8clz pod-name=arango-arangodb-operator-6cf85fc55f-q8clz pod-namespace=arango
2024-07-24T12:22:10Z INF CRD Update not required crd=arangoclustersynchronizations.database.arangodb.com operator-id=q8clz version=5de0cceda99c91fe708370e9996f8c98
2024-07-24T12:22:10Z INF CRD Update not required crd=arangojobs.apps.arangodb.com operator-id=q8clz version=adc765de8dab4e31cd715a172b694a68
2024-07-24T12:22:10Z INF Update Operations is not allowed. Continue crd=arangomlbatchjobs.ml.arangodb.com operator-id=q8clz schema=d67b904151fc753f2b9a1dcd174cc908 version=3da979e8cc522c84fa8d7bf3d04599a4
2024-07-24T12:22:10Z INF CRD Update not required crd=arangobackuppolicies.backup.arangodb.com operator-id=q8clz version=f1e80ff799576a674d6f2ebd4a1fca3f
2024-07-24T12:22:10Z INF CRD Update not required crd=arangomembers.database.arangodb.com operator-id=q8clz version=dfa84092e1a390a9f6180f194c42ce22
2024-07-24T12:22:10Z INF Update Operations is not allowed. Continue crd=arangomlextensions.ml.arangodb.com operator-id=q8clz schema=eae959fb737b49dac46f7238c3bb534b version=53cff97ca37fa8e1878e879386a41cc3
2024-07-24T12:22:10Z INF Update Operations is not allowed. Continue crd=arangomlcronjobs.ml.arangodb.com operator-id=q8clz schema=c53a044e9decb09c4e5ff12b4ad20dbc version=071152afd23e0c6212e6ff16de3ec2fb
2024-07-24T12:22:10Z INF CRD Update not required crd=arangotasks.database.arangodb.com operator-id=q8clz version=de60b4008e910ada265f8066551a28e4
2024-07-24T12:22:10Z INF Update Operations is not allowed. Continue crd=graphanalyticsengines.analytics.arangodb.com operator-id=q8clz schema=bb8e406e376ca72490036183f301abab version=a7cd6bb879625020634bea59274db2e4
2024-07-24T12:22:10Z INF CRD Update not required crd=arangodeploymentreplications.replication.database.arangodb.com operator-id=q8clz version=55f7b3b00702d
80708fe29615da6c446
2024-07-24T12:22:10Z INF CRD Update not required crd=arangodeployments.database.arangodb.com operator-id=q8clz version=2b3c736ba875c72a57da26c7b3b754f5
2024-07-24T12:22:10Z INF Create Operations is not allowed but CRD is missing. Continue crd=arangolocalstorages.storage.arangodb.com operator-id=q8clz version=ba7d9af888270bab37493079eff61855
2024-07-24T12:22:10Z INF Update Operations is not allowed. Continue crd=arangomlstorages.ml.arangodb.com operator-id=q8clz schema=7434ce36a4137aec0f1791ea12ef58f8 version=0784b0c3ed19a07f82be89cf732e688a
2024-07-24T12:22:11Z INF Update Operations is not allowed. Continue crd=arangoprofiles.scheduler.arangodb.com operator-id=q8clz version=3a71d123e7d0bf3902f91ebcdc4cac77
2024-07-24T12:22:11Z INF CRD Update not required crd=arangobackups.backup.arangodb.com operator-id=q8clz version=c0e6bf20aecad187ac0ac881bb7207ff
2024-07-24T12:22:11Z INF Serving HTTP API on 0.0.0.0:8628 operator-id=q8clz
2024-07-24T12:22:11Z INF Serving GRPC API on 0.0.0.0:8728 operator-id=q8clz
2024-07-24T12:22:11Z INF Serving on 0.0.0.0:8528 operator-id=q8clz
2024-07-24T12:22:11Z INF Event(v1.ObjectReference{Kind:"Deployment", Namespace:"arango", Name:"arango-arangodb-operator", UID:"7d2ca09d-9fb2-4be3-867e-249fc5cf1410", APIVersion:"apps/v1", ResourceVersion:"779273459", FieldPath:""}): type: 'Normal' reason: 'Leader Election Skipped' Pod arango-arangodb-operator-6cf85fc55f-q8clz is running as leader operator-id=q8clz
2024-07-24T12:22:11Z INF Event(v1.ObjectReference{Kind:"Deployment", Namespace:"arango", Name:"arango-arangodb-operator", UID:"7d2ca09d-9fb2-4be3-867e-249fc5cf1410", APIVersion:"apps/v1", ResourceVersion:"779273459", FieldPath:""}): type: 'Normal' reason: 'Leader Electi
on Skipped' Pod arango-arangodb-operator-6cf85fc55f-q8clz is running as leader operator-id=q8clz
2024-07-24T12:22:12Z DBG Waiting for arangodeployments.database.arangodb.com CRD to be ready - namespace=arango operator-id=q8clz
2024-07-24T12:22:12Z DBG CRDs ready namespace=arango operator-id=q8clz
2024-07-24T12:22:12Z DBG ArangoDeployment added name=arangodb namespace=arango operator-id=q8clz
2024-07-24T12:22:12Z WRN ACS cache not yet initialised name=arangodb namespace=arango operator-id=q8clz
2024-07-24T12:22:14Z ERR Unable to get secret error="secrets \"arangodb-jwt-folder\" not found" name=arangodb namespace=arango operator-id=q8clz
@vfouqueron It is pretty strange - as the Operator gets stuck after a new ArangoDeployment is added - it can be there is some timeout, can you wait for 30 min and check if the Operator prints more logs?
Best Regards, Adam.
Sorry, I've been a bit busy on another subject. I've uninstalled everything, put in debug and ask a new deployment, maybe these logs will be more useful :
2024-07-26T12:41:38Z INF Operator Feature agency-poll (deployment.feature.agency-poll) is disabled. operator-id=gkp7s
2024-07-26T12:41:38Z INF Operator Feature async-backup-creation (deployment.feature.async-backup-creation) is enabled. operator-id=gkp7s
2024-07-26T12:41:38Z INF Operator Feature backup-cleanup (deployment.feature.backup-cleanup) is disabled. operator-id=gkp7s
2024-07-26T12:41:38Z INF Operator Feature deployment-spec-defaults-restore (deployment.feature.deployment-spec-defaults-restore) is enabled. operator-id=gkp7s
2024-07-26T12:41:38Z INF Operator Feature encryption-rotation (deployment.feature.encryption-rotation) is disabled. operator-id=gkp7s
2024-07-26T12:41:38Z INF Operator Feature enforced-resign-leadership (deployment.feature.enforced-resign-leadership) is enabled. operator-id=gkp7s
2024-07-26T12:41:38Z INF Operator Feature ephemeral-volumes (deployment.feature.ephemeral-volumes) is disabled. operator-id=gkp7s
2024-07-26T12:41:38Z INF Operator Feature failover-leadership (deployment.feature.failover-leadership) is disabled. operator-id=gkp7s
2024-07-26T12:41:38Z INF Operator Feature force-rebuild-out-synced-shards (deployment.feature.force-rebuild-out-synced-shards) is disabled. operator-id=gkp7s
2024-07-26T12:41:38Z INF Operator Feature graceful-shutdown (deployment.feature.graceful-shutdown) is enabled. operator-id=gkp7s
2024-07-26T12:41:38Z INF Operator Feature init-containers-copy-resources (deployment.feature.init-containers-copy-resources) is enabled. operator-id=gkp7s
2024-07-26T12:41:38Z INF Operator Feature init-containers-upscale-resources (deployment.feature.init-containers-upscale-resources) is enabled. operator-id=gkp7s
2024-07-26T12:41:38Z INF Operator Feature jwt-rotation (deployment.feature.jwt-rotation) is enabled. operator-id=gkp7s
2024-07-26T12:41:38Z INF Operator Feature local-storage.pass-reclaim-policy (deployment.feature.local-storage.pass-reclaim-policy) is disabled. operator-id=gkp7s
2024-07-26T12:41:38Z INF Operator Feature local-volume-replacement-check (deployment.feature.local-volume-replacement-check) is disabled. operator-id=gkp7s
2024-07-26T12:41:38Z INF Operator Feature maintenance (deployment.feature.maintenance) is enabled. operator-id=gkp7s
2024-07-26T12:41:38Z INF Operator Feature metrics-exporter (deployment.feature.metrics-exporter) is enabled. operator-id=gkp7s
2024-07-26T12:41:38Z INF Operator Feature optional-graceful-shutdown (deployment.feature.optional-graceful-shutdown) is disabled. operator-id=gkp7s
2024-07-26T12:41:38Z INF Operator Feature random-pod-names (deployment.feature.random-pod-names) is disabled.
operator-id=gkp7s
2024-07-26T12:41:38Z INF Operator Feature rebalancer-v2 (deployment.feature.rebalancer-v2) is disabled. operator-id=gkp7s
2024-07-26T12:41:38Z INF Operator Feature restart-policy-always (deployment.feature.restart-policy-always) is disabled. operator-id=gkp7s
2024-07-26T12:41:38Z INF Operator Feature secured-containers (deployment.feature.secured-containers) is disabled. operator-id=gkp7s
2024-07-26T12:41:38Z INF Operator Feature sensitive-information-protection (deployment.feature.sensitive-information-protection) is disabled. operator-id=gkp7s
2024-07-26T12:41:38Z INF Operator Feature short-pod-names (deployment.feature.short-pod-names) is disabled. operator-id=gkp7s
2024-07-26T12:41:38Z INF Operator Feature timezone-management (deployment.feature.timezone-management) is disabled. operator-id=gkp7s
2024-07-26T12:41:38Z INF Operator Feature tls-rotation (deployment.feature.tls-rotation) is enabled. operator-id=gkp7s
2024-07-26T12:41:38Z INF Operator Feature tls-sni (deployment.feature.tls-sni) is enabled. operator-id=gkp7s
2024-07-26T12:41:38Z INF Operator Feature upgrade-version-check (deployment.feature.upgrade-version-check) is enabled. operator-id=gkp7s
2024-07-26T12:41:38Z INF Operator Feature upgrade-version-check-v2 (deployment.feature.upgrade-version-check-v2) is disabled. operator-id=gkp7s
2024-07-26T12:41:38Z INF Operator Feature version.3-10 (deployment.feature.version.3-10) is enabled. operator-id=gkp7s
2024-07-26T12:41:38Z INF Starting arangodb-operator (Community), version 1.2.41 build bc898c71 operator-id=gkp7s pod-name=arango-arangodb-operator-6cf85fc55f-gkp7s pod-namespace=arango
2024-07-26T12:41:38Z INF CRD Update not required crd=arangobackuppolicies.backup.arangodb.com operator-id=gkp7s version=f1e80ff799576a674d6f2ebd4a1fca3f
2024-07-26T12:41:38Z INF Update Operations is not allowed. Continue crd=arangoprofiles.scheduler.arangodb.com operator-id=gkp7s version=3a71d123e7d0bf3902f91ebcdc4cac77
2024-07-26T12:41:38Z INF CRD Update not required crd=arangojobs.apps.arangodb.com operator-id=gkp7s version=adc765de8dab4e31cd715a172b694a68
2024-07-26T12:41:38Z INF Create Operations is not allowed but CRD is missing. Continue crd=arangolocalstorages.storage.arangodb.com operator-id=gkp7s version=ba7d9af888270bab37493079eff61855
2024-07-26T12:41:38Z INF Update Operations is not allowed. Continue crd=arangomlextensions.ml.arangodb.com operator-id=gkp7s schema=eae959fb737b49dac46f7238c3bb534b version=53cff97ca37fa8e1878e879386a41cc3
2024-07-26T12:41:38Z INF Update Operations is not allowed. Continue crd=arangomlcronjobs.ml.arangodb.com operator-id=gkp7s schema=c53a044e9decb09c4e5ff12b4ad20dbc version=071152afd23e0c6212e6ff16de3ec2fb
2024-07-26T12:41:39Z INF CRD Update not required crd=arangotasks.database.arangodb.com operator-id=gkp7s version=de60b4008e910ada265f8066551a28e4
2024-07-26T12:41:39Z INF Update Operations is not
allowed. Continue crd=graphanalyticsengines.analytics.arangodb.com operator-id=gkp7s schema=bb8e406e376ca72490036183f301abab version=a7cd6bb879625020634bea59274db2e4
2024-07-26T12:41:39Z INF CRD Update not required crd=arangodeploymentreplications.replication.database.arangodb.com operator-id=gkp7s version=55f7b3b00702d80708fe29615da6c446
2024-07-26T12:41:39Z INF CRD Update not required crd=arangodeployments.database.arangodb.com operator-id=gkp7s version=2b3c736ba875c72a57da26c7b3b754f5
2024-07-26T12:41:39.152773097Z 2024-07-26T12:41:39Z INF CRD Update not required crd=arangomembers.database.arangodb.com operator-id=gkp7s version=dfa84092e1a390a9f6180f194c42ce22
2024-07-26T12:41:39Z INF Update Operations is not allowed. Continue crd=arangomlstorages.ml.arangodb.com operator-id=gkp7s schema=7434ce36a4137aec0f1791ea12ef58f8 version=0784b0c3ed19a07f82be89cf732e688a
2024-07-26T12:41:39Z INF Update Operations is not allowed. Continue crd=arangomlbatchjobs.ml.arangodb.com operator-id=gkp7s schema=d67b904151fc753f2b9a1dcd174cc908 version=3da979e8cc522c84fa8d7bf3d04599a4
2024-07-26T12:41:39Z INF CRD Update not required crd=arangobackups.backup.arangodb.com operator-id=gkp7s version=c0e6bf20aecad187ac0ac881bb7207ff
2024-07-26T12:41:39Z INF CRD Update not required crd=arangoclustersynchronizations.database.arangodb.com operator-id=gkp7s version=5de0cceda99c91fe708370e9996f8c98
2024-07-26T12:41:39Z INF Serving GRPC API on 0.0.0.0:8728 operator-id=gkp7s
2024-07-26T12:41:39Z INF Serving on 0.0.0.0:8528 operator-id=gkp7s
2024-07-26T12:41:39Z INF Serving HTTP API on 0.0.0.0:8628 operator-id=gkp7s
2024-07-26T12:41:40Z INF Event(v1.ObjectReference{Kind:"Deployment", Namespace:"arango", Name:"arango-arangodb-operator", UID:"dfcc4b4f-ee19-450f-8731-aaa1b908ea6e", APIVersion:"apps/v1", ResourceVersion:"781082719", FieldPath:""}): type: 'Normal' reason: 'Leader Election Skipped' Pod arango-arangodb-operator-6cf85fc55f-gkp7s is running as leader operator-id=gkp7s
2024-07-26T12:41:40Z INF Event(v1.ObjectReference{Kind:"Deployment", Namespace:"arango", Name:"arango-arangodb-operator", UID:"dfcc4b4f-ee19-450f-8731-aaa1b908ea6e", APIVersion:"apps/v1", ResourceVersion:"781082719", FieldPath:""}): type: 'Normal' reason: 'Leader Election Skipped' Pod arango-arangodb-operator-6cf85fc55f-gkp7s is running as leader operator-id=gkp7s
2024-07-26T12:41:40Z DBG Waiting for arangodeployments.database.arangodb.com CRD to be ready - namespace=arango operator-id=gkp7s
2024-07-26T12:41:40Z DBG CRDs ready namespace=arango operator-id=gkp7s
2024-07-26T12:41:40Z DBG Waiting for arangodeploymentreplications.replication.database.arangodb.com CRD to be ready - namespace=arango operator-id=gkp7s
2024-07-26T12:41:40.861181869Z 2024-07-26T12:41:40Z DBG CRDs ready namespace=arango operator-id=gkp7s
2024-07-26T12:42:54Z DBG ArangoDeployment added name=arangodb namespace=arango operator-id=gkp7s
2024-07-26T12:42:54Z WRN ACS cache not yet initialised name=arangodb namespace=arango operator-id=gkp7s
2024-07-26T12:42:54Z INF start running... name=arangodb namespace=arango operator-id=gkp7s
2024-07-26T12:42:54Z ERR Unable to get secret error="secrets \"arangodb-jwt-folder\" not found" name=arangodb namespace=arango operator-id=gkp7s
2024-07-26T12:42:54Z DBG Clean failed error="JWT Secret is invalid" name=arangodb namespace=arango operator-id=gkp7s
2024-07-26T12:42:54Z DBG Looking for ServiceMonitor CRD... name=arangodb namespace=arango operator-id=gkp7s
2024-07-26T12:42:54Z INF ...have discovered ServiceMonitor CRD name=arangodb namespace=arango operator-id=gkp7s
2024-07-26T12:42:54Z DBG Initially inspect deployment... name=arangodb namespace=arango operator-id=gkp7s
2024-07-26T12:42:54Z INF Accepted new spec name=arangodb namespace=arango operator-id=gkp7s
2024-07-26T12:42:54Z DBG ...deployment inspect started interval=250ms name=arangodb namespace=arango operator-id=gkp7s
2024-07-26T12:42:55Z INF Accepted new spec name=arangodb namespace=arango operator-id=gkp7s
2024-07-26T12:42:55Z INF Updated condition condition=SpecAccepted hash=141104aae08dfe1ff9d073461e7f0f715196b4f3672fbc6d70820ac090c7d96c name=arangodb namespace=arango operator-id=gkp7s reason="Spec Accepted" status=true
2024-07-26T12:42:55Z INF Kubernetes request has been send kind=Service name=arangodb-int operator-id=gkp7s verb=create
2024-07-26T12:42:55Z DBG Created headless service name=arangodb namespace=arango operator-id=gkp7s section=service service=arangodb-int
2024-07-26T12:42:55Z INF Kubernetes request has been send kind=Service name=arangodb operator-id=gkp7s verb=create
2024-07-26T12:42:55Z DBG Created database client service name=arangodb namespace=arango operator-id=gkp7s section=service service=arangodb
2024-07-26T12:42:55Z INF Kubernetes request has been send kind=Service name=arangodb-ea operator-id=gkp7s verb=create
2024-07-26T12:42:55Z DBG Created %!s(MISSING) external access service name=arangodb namespace=arango operator-id=gkp7s role=coordinator section=service-ea service=arangodb-ea
2024-07-26T12:42:55Z INF Kubernetes request has been send kind=Secret name=arangodb-jwt operator-id=gkp7s verb=create
2024-07-26T12:42:55Z INF Kubernetes request has been send kind=Secret name=arangodb-ca operator-id=gkp7s verb=create
2024-07-26T12:42:55Z DBG Created CA Secret name=arangodb namespace=arango operator-id=gkp7s secret=arangodb-ca section=tls
2024-07-26T12:42:56Z INF Kubernetes request has been send kind=Secret name=arangodb-truststore operator-id=gkp7s verb=create
2024-07-26T12:42:56Z ERR Unable to get secret error="secrets \"arangodb-jwt-folder\" not found" name=arangodb namespace=arango operator-id=gkp7s
Here is the ArangoDeployment CRD :
apiVersion: database.arangodb.com/v1
kind: ArangoDeployment
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: >
{"apiVersion":"database.arangodb.com/v1","kind":"ArangoDeployment","metadata":{"annotations":{},"name":"arangodb","namespace":"arango"},"spec":{"agents":{"volumeClaimTemplate":{"spec":{"storageClassName":"longhorn"}}},"dbservers":{"volumeClaimTemplate":{"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"50Gi"}},"storageClassName":"longhorn"}}},"mode":"Cluster","timezone":"Europe/Paris"}}
creationTimestamp: '2024-07-26T12:42:54Z'
finalizers:
- database.arangodb.com/remove-child-finalizers
generation: 9
managedFields:
- apiVersion: database.arangodb.com/v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:kubectl.kubernetes.io/last-applied-configuration: {}
f:spec:
.: {}
f:agents:
.: {}
f:volumeClaimTemplate:
.: {}
f:spec:
.: {}
f:storageClassName: {}
f:dbservers:
.: {}
f:volumeClaimTemplate:
.: {}
f:spec:
.: {}
f:accessModes: {}
f:resources:
.: {}
f:requests:
.: {}
f:storage: {}
f:storageClassName: {}
f:mode: {}
f:timezone: {}
manager: HashiCorp
operation: Update
time: '2024-07-26T12:42:54Z'
- apiVersion: database.arangodb.com/v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:finalizers:
.: {}
v:"database.arangodb.com/remove-child-finalizers": {}
f:spec:
f:agents:
f:count: {}
f:resources: {}
f:volumeClaimTemplate:
f:metadata:
.: {}
f:creationTimestamp: {}
f:spec:
f:resources: {}
f:status: {}
f:auth:
.: {}
f:jwtSecretName: {}
f:bootstrap:
.: {}
f:passwordSecretNames:
.: {}
f:root: {}
f:chaos:
.: {}
f:interval: {}
f:kill-pod-probability: {}
f:coordinators:
.: {}
f:count: {}
f:resources: {}
f:dbservers:
f:count: {}
f:resources: {}
f:volumeClaimTemplate:
f:metadata:
.: {}
f:creationTimestamp: {}
f:status: {}
f:environment: {}
f:externalAccess: {}
f:image: {}
f:imagePullPolicy: {}
f:license: {}
f:lifecycle:
.: {}
f:resources: {}
f:metrics:
.: {}
f:authentication:
.: {}
f:jwtTokenSecretName: {}
f:enabled: {}
f:resources: {}
f:rocksdb:
.: {}
f:encryption: {}
f:single:
.: {}
f:resources: {}
f:volumeClaimTemplate:
.: {}
f:metadata:
.: {}
f:creationTimestamp: {}
f:spec:
.: {}
f:accessModes: {}
f:resources:
.: {}
f:requests:
.: {}
f:storage: {}
f:volumeMode: {}
f:status: {}
f:storageEngine: {}
f:sync:
.: {}
f:auth:
.: {}
f:clientCASecretName: {}
f:jwtSecretName: {}
f:externalAccess: {}
f:image: {}
f:monitoring:
.: {}
f:tokenSecretName: {}
f:tls:
.: {}
f:caSecretName: {}
f:ttl: {}
f:syncmasters:
.: {}
f:resources: {}
f:syncworkers:
.: {}
f:resources: {}
f:tls:
.: {}
f:caSecretName: {}
f:ttl: {}
f:status:
.: {}
f:accepted-spec:
.: {}
f:agents:
.: {}
f:count: {}
f:resources: {}
f:volumeClaimTemplate:
.: {}
f:metadata:
.: {}
f:creationTimestamp: {}
f:spec:
.: {}
f:resources: {}
f:storageClassName: {}
f:status: {}
f:auth:
.: {}
f:jwtSecretName: {}
f:bootstrap:
.: {}
f:passwordSecretNames:
.: {}
f:root: {}
f:chaos:
.: {}
f:interval: {}
f:kill-pod-probability: {}
f:coordinators:
.: {}
f:count: {}
f:resources: {}
f:dbservers:
.: {}
f:count: {}
f:resources: {}
f:volumeClaimTemplate:
.: {}
f:metadata:
.: {}
f:creationTimestamp: {}
f:spec:
.: {}
f:accessModes: {}
f:resources:
.: {}
f:requests:
.: {}
f:storage: {}
f:storageClassName: {}
f:status: {}
f:environment: {}
f:externalAccess: {}
f:image: {}
f:imagePullPolicy: {}
f:license: {}
f:lifecycle:
.: {}
f:resources: {}
f:metrics:
.: {}
f:authentication:
.: {}
f:jwtTokenSecretName: {}
f:enabled: {}
f:resources: {}
f:mode: {}
f:rocksdb:
.: {}
f:encryption: {}
f:single:
.: {}
f:resources: {}
f:volumeClaimTemplate:
.: {}
f:metadata:
.: {}
f:creationTimestamp: {}
f:spec:
.: {}
f:accessModes: {}
f:resources:
.: {}
f:requests:
.: {}
f:storage: {}
f:volumeMode: {}
f:status: {}
f:storageEngine: {}
f:sync:
.: {}
f:auth:
.: {}
f:clientCASecretName: {}
f:jwtSecretName: {}
f:externalAccess: {}
f:image: {}
f:monitoring:
.: {}
f:tokenSecretName: {}
f:tls:
.: {}
f:caSecretName: {}
f:ttl: {}
f:syncmasters:
.: {}
f:resources: {}
f:syncworkers:
.: {}
f:resources: {}
f:timezone: {}
f:tls:
.: {}
f:caSecretName: {}
f:ttl: {}
f:acceptedSpecVersion: {}
f:agency:
.: {}
f:ids: {}
f:size: {}
f:agents: {}
f:appliedVersion: {}
f:conditions: {}
f:hashes:
.: {}
f:jwt: {}
f:rocksDBEncryption: {}
f:tls: {}
f:members: {}
f:phase: {}
f:secret-hashes:
.: {}
f:auth-jwt: {}
f:serviceName: {}
f:version: {}
manager: arangodb_operator
operation: Update
time: '2024-07-26T12:42:56Z'
name: arangodb
namespace: arango
resourceVersion: '781083570'
uid: 11649eb6-4cf5-49bc-9713-3ab139112731
spec:
agents:
count: 3
resources: {}
volumeClaimTemplate:
metadata:
creationTimestamp: null
spec:
resources: {}
storageClassName: longhorn
status: {}
auth:
jwtSecretName: arangodb-jwt
bootstrap:
passwordSecretNames:
root: None
chaos:
interval: 60000000000
kill-pod-probability: 50
coordinators:
count: 3
resources: {}
dbservers:
count: 3
resources: {}
volumeClaimTemplate:
metadata:
creationTimestamp: null
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi
storageClassName: longhorn
status: {}
environment: Development
externalAccess: {}
image: arangodb/arangodb:latest
imagePullPolicy: IfNotPresent
license: {}
lifecycle:
resources: {}
metrics:
authentication:
jwtTokenSecretName: arangodb-exporter-jwt-token
enabled: false
resources: {}
mode: Cluster
rocksdb:
encryption: {}
single:
resources: {}
volumeClaimTemplate:
metadata:
creationTimestamp: null
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 8Gi
volumeMode: Filesystem
status: {}
storageEngine: RocksDB
sync:
auth:
clientCASecretName: arangodb-sync-client-auth-ca
jwtSecretName: arangodb-sync-jwt
externalAccess: {}
image: null
monitoring:
tokenSecretName: arangodb-sync-mt
tls:
caSecretName: arangodb-sync-ca
ttl: 2610h
syncmasters:
resources: {}
syncworkers:
resources: {}
timezone: Europe/Paris
tls:
caSecretName: arangodb-ca
ttl: 2610h
status:
accepted-spec:
agents:
count: 3
resources: {}
volumeClaimTemplate:
metadata:
creationTimestamp: null
spec:
resources: {}
storageClassName: longhorn
status: {}
auth:
jwtSecretName: arangodb-jwt
bootstrap:
passwordSecretNames:
root: None
chaos:
interval: 60000000000
kill-pod-probability: 50
coordinators:
count: 3
resources: {}
dbservers:
count: 3
resources: {}
volumeClaimTemplate:
metadata:
creationTimestamp: null
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi
storageClassName: longhorn
status: {}
environment: Development
externalAccess: {}
image: arangodb/arangodb:latest
imagePullPolicy: IfNotPresent
license: {}
lifecycle:
resources: {}
metrics:
authentication:
jwtTokenSecretName: arangodb-exporter-jwt-token
enabled: false
resources: {}
mode: Cluster
rocksdb:
encryption: {}
single:
resources: {}
volumeClaimTemplate:
metadata:
creationTimestamp: null
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 8Gi
volumeMode: Filesystem
status: {}
storageEngine: RocksDB
sync:
auth:
clientCASecretName: arangodb-sync-client-auth-ca
jwtSecretName: arangodb-sync-jwt
externalAccess: {}
image: null
monitoring:
tokenSecretName: arangodb-sync-mt
tls:
caSecretName: arangodb-sync-ca
ttl: 2610h
syncmasters:
resources: {}
syncworkers:
resources: {}
timezone: Europe/Paris
tls:
caSecretName: arangodb-ca
ttl: 2610h
acceptedSpecVersion: 141104aae08dfe1ff9d073461e7f0f715196b4f3672fbc6d70820ac090c7d96c
agency:
ids:
- AGNT-cwmhiffm
- AGNT-0k0zgwyh
- AGNT-iliydshz
size: 3
agents: {}
appliedVersion: ''
conditions:
- hash: 141104aae08dfe1ff9d073461e7f0f715196b4f3672fbc6d70820ac090c7d96c
lastTransitionTime: '2024-07-26T12:42:55Z'
lastUpdateTime: '2024-07-26T12:42:55Z'
message: Spec Object accepted
reason: Spec Accepted
status: 'True'
type: SpecAccepted
hashes:
jwt: {}
rocksDBEncryption: {}
tls: {}
members: {}
phase: Running
secret-hashes:
auth-jwt: 128ae53dbdc4665f04b293e35c01e5c23a0b7543e4c7f75888b8b09e04d9c00e
serviceName: arangodb
version: 1.2.8.1
I will post other logs in 30 minutes !
I could not noticing anything else, except : 2024-07-26T12:43:56Z INF LoadBalancerIP of external access service is not set, switching to NodePort name=arangodb namespace=arango operator-id=gkp7s role=coordinator section=service-ea service=arangodb-ea
Does it help ? It is an internal instance, we do not have a LoadBalancer, we will probably add one but our ingress is exposed on a NodePort currently
You can find full logs here : https://gist.github.com/vfouqueron/3c64a07642b40658b59fcb11514073ce
Hello!
It now helps a lot!
The issue is with ImageDiscovery - can you check if a pod with name -id was created? If yes, could you check why it did not start/fail?
Best Regards, Adam.
Yes, it loops on creating these pods. They fail but they don't print any log :/
Can you share describe?
Here it is :
Name: arangodb-id-b026ee
Namespace: arango
Priority: 0
Service Account: default
Node: vmallkub02/192.168.9.32
Start Time: Tue, 30 Jul 2024 07:53:47 +0000
Labels: app=arangodb
arango_deployment=arangodb
deployment.arangodb.com/member=b026ee
role=id
Annotations: cni.projectcalico.org/containerID: d4013a18422e8e1e741ed08cfd4c4c2e095f0c072d20c88023ba47a04db2c2da
cni.projectcalico.org/podIP:
cni.projectcalico.org/podIPs:
Status: Failed
IP: 10.42.0.99
IPs:
IP: 10.42.0.99
Controlled By: ArangoDeployment/arangodb
Containers:
server:
Container ID: containerd://d1d12ea7ec6fe81d6311c68aaaf9311017a0aaf3b63325382d6ca1b3969c5cea
Image: arangodb/arangodb:latest
Image ID: docker.io/arangodb/arangodb@sha256:47c73683ac5e250adf71506a41c03ac9775eefa8f06183f9674981804cebc12d
Port: 8529/TCP
Host Port: 0/TCP
Command:
/usr/sbin/arangod
--database.directory=/data
--log.output=+
--server.authentication=false
--server.endpoint=tcp://[::]:8529
State: Terminated
Reason: Error
Exit Code: 132
Started: Tue, 30 Jul 2024 07:53:48 +0000
Finished: Tue, 30 Jul 2024 07:53:48 +0000
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/data from arangod-data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-k7bqk (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
arangod-data:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
kube-api-access-k7bqk:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.alpha.kubernetes.io/unreachable:NoExecute op=Exists for 5s
node.kubernetes.io/not-ready:NoExecute op=Exists for 5s
node.kubernetes.io/unreachable:NoExecute op=Exists for 5s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 13s default-scheduler Successfully assigned arango/arangodb-id-b026ee to vmallkub02
Normal Pulled 12s kubelet Container image "arangodb/arangodb:latest" already present on machine
Normal Created 12s kubelet Created container server
Normal Started 12s kubelet Started container server
Thank you a lot @ajanikow, I found the issue thanks to your precious debugging instructions. It is not related to ArangoDB itself, my kubernetes cluster is running in a virtualized environment running with Proxmox and the emulated processor had no access to amd64-avx instructions. I had to change the type of processor in proxmox to match the host and I just got ArangoDB up and running :)
I just tried to deploy arango in my Kubernetes cluster in version 1.2.41. The pod fails instantly with no error.
In the arango-arangodb-operator, I have the following error :
Unable to get secret error="secrets \"arangodb-jwt-folder\" not found" name=arangodb namespace=arango operator-id=phrlx
I found similar issue https://github.com/arangodb/kube-arangodb/issues/993 but it describes an issue solved two years ago. ANy idea on what is happening ?
Here is my ArangoDeployment :