search

canonical / bundle-kubeflow

Charmed Kubeflow
Apache License 2.0
97 stars 47 forks source link

Charm goes into blocked with "Not all resources found in cluster." #930

Open orfeas-k opened 3 weeks ago

orfeas-k commented 3 weeks ago

Bug Description

argo-controller/0*            blocked   idle   10.244.0.39                  [kubernetes:crds-cm-and-secrets] Not all resources found in cluster.  This may be transient if we haven't tried to de...

This probably relates to how kubernetesComponent works.

That being said, it resolves itself after a while (sth like 20 minutes later?)

To Reproduce

Deploy CKF from latest/edge bundle.yaml file.

Environment

AKS 1.28 juju 3.5.1 with agent 3.5.0

Relevant Log Output

$ jst (juju status)
Model     Controller     Cloud/Region    Version  SLA          Timestamp
kubeflow  kf-controller  aks/westeurope  3.5.0    unsupported  11:54:28+03:00

App                        Version                  Status   Scale  Charm                    Channel              Rev  Address       Exposed  Message
admission-webhook                                   active       1  admission-webhook        latest/edge          312  10.0.49.139   no       
argo-controller                                     waiting      1  argo-controller          latest/edge          489  10.0.96.114   no       installing agent
dex-auth                                            active       1  dex-auth                 latest/edge          474  10.0.120.239  no       
envoy                                               active       1  envoy                    latest/edge          207  10.0.83.231   no       
istio-ingressgateway                                active       1  istio-gateway            latest/edge          913  10.0.63.210   no       
istio-pilot                                         active       1  istio-pilot              latest/edge          872  10.0.57.131   no       
jupyter-controller                                  active       1  jupyter-controller       latest/edge          962  10.0.253.162  no       
jupyter-ui                                          active       1  jupyter-ui               latest/edge          884  10.0.79.228   no       
katib-controller                                    active       1  katib-controller         latest/edge/pr-185   553  10.0.209.225  no       
katib-db                   8.0.36-0ubuntu0.22.04.1  active       1  mysql-k8s                8.0/edge             146  10.0.94.241   no       
katib-db-manager                                    active       1  katib-db-manager         latest/edge/pr-185   517  10.0.80.0     no       
katib-ui                                            active       1  katib-ui                 latest/edge          518  10.0.46.198   no       
kfp-api                                             active       1  kfp-api                  latest/edge         1348  10.0.69.72    no       
kfp-db                     8.0.36-0ubuntu0.22.04.1  active       1  mysql-k8s                8.0/edge             146  10.0.69.59    no       
kfp-metadata-writer                                 active       1  kfp-metadata-writer      latest/edge          392  10.0.117.239  no       
kfp-persistence                                     active       1  kfp-persistence          latest/edge         1355  10.0.10.135   no       
kfp-profile-controller                              active       1  kfp-profile-controller   latest/edge         1313  10.0.15.8     no       
kfp-schedwf                                         active       1  kfp-schedwf              latest/edge         1367  10.0.112.16   no       
kfp-ui                                              active       1  kfp-ui                   latest/edge         1349  10.0.183.58   no       
kfp-viewer                                          active       1  kfp-viewer               latest/edge         1381  10.0.125.51   no       
kfp-viz                                             active       1  kfp-viz                  latest/edge         1300  10.0.83.53    no       
knative-eventing                                    active       1  knative-eventing         latest/edge          401  10.0.238.181  no       
knative-operator                                    active       1  knative-operator         latest/edge          376  10.0.251.51   no       
knative-serving                                     active       1  knative-serving          latest/edge          402  10.0.205.88   no       
kserve-controller                                   active       1  kserve-controller        latest/edge          543  10.0.45.208   no       
kubeflow-dashboard                                  active       1  kubeflow-dashboard       latest/edge          543  10.0.199.224  no       
kubeflow-profiles                                   active       1  kubeflow-profiles        latest/edge          385  10.0.91.22    no       
kubeflow-roles                                      active       1  kubeflow-roles           latest/edge          218  10.0.255.1    no       
kubeflow-volumes                                    active       1  kubeflow-volumes         latest/edge          307  10.0.70.51    no       
metacontroller-operator                             active       1  metacontroller-operator  latest/edge          280  10.0.32.176   no       
minio                      res:oci-image@1755999    active       1  minio                    latest/edge          316  10.0.150.254  no       
mlmd                                                active       1  mlmd                     latest/edge          184  10.0.63.139   no       
oidc-gatekeeper                                     active       1  oidc-gatekeeper          latest/edge          382  10.0.93.201   no       
pvcviewer-operator                                  active       1  pvcviewer-operator       latest/edge           95  10.0.251.221  no       
seldon-controller-manager                           active       1  seldon-core              latest/edge          691  10.0.129.45   no       
tensorboard-controller                              active       1  tensorboard-controller   latest/edge          288  10.0.60.4     no       
tensorboards-web-app                                active       1  tensorboards-web-app     latest/edge          276  10.0.242.228  no       
training-operator                                   active       1  training-operator        latest/edge          378  10.0.70.1     no       

Unit                          Workload  Agent  Address       Ports          Message
admission-webhook/0*          active    idle   10.244.0.38                  
argo-controller/0*            blocked   idle   10.244.0.39                  [kubernetes:crds-cm-and-secrets] Not all resources found in cluster.  This may be transient if we havent tried to de...
dex-auth/0*                   active    idle   10.244.0.40                  
envoy/0*                      active    idle   10.244.0.43                  
istio-ingressgateway/0*       active    idle   10.244.0.41                  
istio-pilot/0*                active    idle   10.244.0.42                  
jupyter-controller/0*         active    idle   10.244.0.44                  
jupyter-ui/0*                 active    idle   10.244.1.118                 
katib-controller/0*           active    idle   10.244.0.45                  
katib-db-manager/0*           active    idle   10.244.1.120                 
katib-db/0*                   active    idle   10.244.0.47                  Primary
katib-ui/0*                   active    idle   10.244.0.46                  
kfp-api/0*                    active    idle   10.244.1.121                 
kfp-db/0*                     active    idle   10.244.1.122                 Primary
kfp-metadata-writer/0*        active    idle   10.244.0.49                  
kfp-persistence/0*            active    idle   10.244.0.52                  
kfp-profile-controller/0*     active    idle   10.244.0.53                  
kfp-schedwf/0*                active    idle   10.244.0.54                  
kfp-ui/0*                     active    idle   10.244.1.123                 
kfp-viewer/0*                 active    idle   10.244.0.57                  
kfp-viz/0*                    active    idle   10.244.1.124                 
knative-eventing/0*           active    idle   10.244.0.48                  
knative-operator/0*           active    idle   10.244.0.60                  
knative-serving/0*            active    idle   10.244.0.50                  
kserve-controller/0*          active    idle   10.244.1.127                 
kubeflow-dashboard/0*         active    idle   10.244.0.58                  
kubeflow-profiles/0*          active    idle   10.244.1.129                 
kubeflow-roles/0*             active    idle   10.244.0.55                  
kubeflow-volumes/0*           active    idle   10.244.1.125                 
metacontroller-operator/0*    active    idle   10.244.0.56                  
minio/0*                      active    idle   10.244.0.59   9000-9001/TCP  
mlmd/0*                       active    idle   10.244.0.62                  
oidc-gatekeeper/0*            active    idle   10.244.1.126                 
pvcviewer-operator/0*         active    idle   10.244.1.130                 
seldon-controller-manager/0*  active    idle   10.244.0.61                  
tensorboard-controller/0*     active    idle   10.244.1.128                 
tensorboards-web-app/0*       active    idle   10.244.0.63                  
training-operator/0*          active    idle   10.244.0.64         

╰─$ kfl argo-controller-0  -c argo-controller
2024-06-11T08:04:24.989Z [pebble] HTTP API server listening on ":38813".
2024-06-11T08:04:24.989Z [pebble] Started daemon.
2024-06-11T08:04:55.240Z [pebble] GET /v1/notices?timeout=30s 30.000296088s 200
2024-06-11T08:05:25.241Z [pebble] GET /v1/notices?timeout=30s 30.000677203s 200
2024-06-11T08:05:55.241Z [pebble] GET /v1/notices?timeout=30s 30.000277459s 200
2024-06-11T08:06:25.242Z [pebble] GET /v1/notices?timeout=30s 30.000342723s 200
2024-06-11T08:06:55.243Z [pebble] GET /v1/notices?timeout=30s 30.000644026s 200
2024-06-11T08:07:25.244Z [pebble] GET /v1/notices?timeout=30s 30.000251745s 200
2024-06-11T08:07:55.245Z [pebble] GET /v1/notices?timeout=30s 30.000304423s 200
2024-06-11T08:08:16.378Z [pebble] GET /v1/plan?format=yaml 86.602µs 200
2024-06-11T08:08:16.380Z [pebble] POST /v1/layers 360.207µs 200
2024-06-11T08:08:16.472Z [pebble] POST /v1/services 90.928064ms 202
2024-06-11T08:08:16.571Z [pebble] GET /v1/notices?timeout=30s 21.325890557s 200
2024-06-11T08:08:16.571Z [pebble] Service "argo-controller" starting: workflow-controller --configmap argo-workflow-controller-configmap --executor-image gcr.io/ml-pipeline/argoexec:v3.4.16-license-compliance
2024-06-11T08:08:16.589Z [argo-controller] time="2024-06-11T08:08:16Z" level=info msg="index config" indexWorkflowSemaphoreKeys=true
2024-06-11T08:08:16.591Z [argo-controller] time="2024-06-11T08:08:16Z" level=info msg="cron config" cronSyncPeriod=10s
2024-06-11T08:08:16.591Z [argo-controller] time="2024-06-11T08:08:16Z" level=info msg="Memoization caches will be garbage-collected if they have not been hit after" gcAfterNotHitDuration=30s
2024-06-11T08:08:16.591Z [argo-controller] time="2024-06-11T08:08:16.591Z" level=info msg="not enabling pprof debug endpoints"
2024-06-11T08:08:16.623Z [argo-controller] time="2024-06-11T08:08:16.623Z" level=info msg="Configuration:\nartifactRepository:\n  archiveLogs: true\n  s3:\n    accessKeySecret:\n      key: accesskey\n      name: mlpipeline-minio-artifact\n    bucket: mlpipeline\n    endpoint: minio.kubeflow:9000\n    insecure: true\n    secretKeySecret:\n      key: secretkey\n      name: mlpipeline-minio-artifact\nexecutor:\n  imagePullPolicy: IfNotPresent\n  name: \"\"\n  resources: {}\ninitialDelay: 0s\nmetricsConfig: {}\nnodeEvents: {}\npodSpecLogStrategy: {}\nsso:\n  clientId:\n    key: \"\"\n  clientSecret:\n    key: \"\"\n  issuer: \"\"\n  redirectUrl: \"\"\n  sessionExpiry: 0s\ntelemetryConfig: {}\n"
2024-06-11T08:08:16.623Z [argo-controller] time="2024-06-11T08:08:16.623Z" level=info msg="Persistence configuration disabled"
2024-06-11T08:08:16.623Z [argo-controller] time="2024-06-11T08:08:16.623Z" level=info executorImage="gcr.io/ml-pipeline/argoexec:v3.4.16-license-compliance" executorImagePullPolicy=IfNotPresent managedNamespace=
2024-06-11T08:08:16.624Z [argo-controller] I0611 08:08:16.624311      16 leaderelection.go:248] attempting to acquire leader lease kubeflow/workflow-controller...
2024-06-11T08:08:16.624Z [argo-controller] time="2024-06-11T08:08:16.624Z" level=info msg="Starting dummy metrics server at localhost:9090/metrics"
2024-06-11T08:08:16.791Z [argo-controller] I0611 08:08:16.791838      16 leaderelection.go:258] successfully acquired lease kubeflow/workflow-controller
2024-06-11T08:08:16.792Z [argo-controller] time="2024-06-11T08:08:16.791Z" level=info msg="new leader" leader=argo-controller
2024-06-11T08:08:16.792Z [argo-controller] time="2024-06-11T08:08:16.792Z" level=info msg="Starting Workflow Controller" defaultRequeueTime=10s version=v3.4.16
2024-06-11T08:08:16.792Z [argo-controller] time="2024-06-11T08:08:16.792Z" level=info msg="Current Worker Numbers" podCleanup=4 workflowTtlWorkers=4 workflowWorkers=32
2024-06-11T08:08:16.792Z [argo-controller] time="2024-06-11T08:08:16.792Z" level=info msg="Successfully shutdown dummy metrics server at localhost:9090/metrics"
2024-06-11T08:08:16.792Z [argo-controller] time="2024-06-11T08:08:16.792Z" level=info msg="Starting prometheus metrics server at localhost:9090/metrics"
2024-06-11T08:08:16.792Z [argo-controller] time="2024-06-11T08:08:16.792Z" level=info msg="Watching task results" labelSelector="!workflows.argoproj.io/controller-instanceid,workflows.argoproj.io/workflow"
2024-06-11T08:08:16.792Z [argo-controller] time="2024-06-11T08:08:16.792Z" level=info msg=Plugins executorPlugins=false
2024-06-11T08:08:16.807Z [argo-controller] time="2024-06-11T08:08:16.807Z" level=info msg="Manager initialized successfully"
2024-06-11T08:08:16.841Z [argo-controller] time="2024-06-11T08:08:16.841Z" level=info msg="Received Workflow Controller config map kubeflow/argo-workflow-controller-configmap update"
2024-06-11T08:08:16.887Z [argo-controller] time="2024-06-11T08:08:16.887Z" level=info msg="Configuration:\nartifactRepository:\n  archiveLogs: true\n  s3:\n    accessKeySecret:\n      key: accesskey\n      name: mlpipeline-minio-artifact\n    bucket: mlpipeline\n    endpoint: minio.kubeflow:9000\n    insecure: true\n    secretKeySecret:\n      key: secretkey\n      name: mlpipeline-minio-artifact\nexecutor:\n  imagePullPolicy: IfNotPresent\n  name: \"\"\n  resources: {}\ninitialDelay: 0s\nmetricsConfig: {}\nnodeEvents: {}\npodSpecLogStrategy: {}\nsso:\n  clientId:\n    key: \"\"\n  clientSecret:\n    key: \"\"\n  issuer: \"\"\n  redirectUrl: \"\"\n  sessionExpiry: 0s\ntelemetryConfig: {}\n"
2024-06-11T08:08:16.887Z [argo-controller] time="2024-06-11T08:08:16.887Z" level=info msg="Persistence configuration disabled"
2024-06-11T08:08:16.887Z [argo-controller] time="2024-06-11T08:08:16.887Z" level=info executorImage="gcr.io/ml-pipeline/argoexec:v3.4.16-license-compliance" executorImagePullPolicy=IfNotPresent managedNamespace=
2024-06-11T08:08:17.005Z [argo-controller] time="2024-06-11T08:08:17.005Z" level=info msg="Performing periodic GC" periodicity=5m0s
2024-06-11T08:08:17.005Z [argo-controller] time="2024-06-11T08:08:17.005Z" level=info msg="Persistence disabled - so archived workflow GC disabled - you must restart the controller if you enable this"
2024-06-11T08:08:17.005Z [argo-controller] time="2024-06-11T08:08:17.005Z" level=info msg="Starting CronWorkflow controller"
2024-06-11T08:08:17.006Z [argo-controller] time="2024-06-11T08:08:17.005Z" level=info msg="Starting workflow garbage collector controller (retentionWorkers 4)"
2024-06-11T08:08:17.006Z [argo-controller] time="2024-06-11T08:08:17.006Z" level=info msg="Started workflow garbage collection"
2024-06-11T08:08:17.006Z [argo-controller] W0611 08:08:17.006122      16 shared_informer.go:401] The sharedIndexInformer has started, run more than once is not allowed
2024-06-11T08:08:17.636Z [pebble] GET /v1/changes/1/wait?timeout=4.000s 1.162624211s 200
2024-06-11T08:08:17.637Z [pebble] GET /v1/notices?after=2024-06-11T08%3A08%3A16.473056513Z&timeout=30s 1.065463412s 200
2024-06-11T08:08:17.660Z [pebble] GET /v1/services 156.603µs 200
2024-06-11T08:08:47.641Z [pebble] GET /v1/notices?after=2024-06-11T08%3A08%3A17.572753164Z&timeout=30s 30.000299529s 200
2024-06-11T08:09:17.642Z [pebble] GET /v1/notices?after=2024-06-11T08%3A08%3A17.572753164Z&timeout=30s 30.000222777s 200
2024-06-11T08:09:47.643Z [pebble] GET /v1/notices?after=2024-06-11T08%3A08%3A17.572753164Z&timeout=30s 30.001175966s 200
2024-06-11T08:10:17.651Z [pebble] GET /v1/notices?after=2024-06-11T08%3A08%3A17.572753164Z&timeout=30s 30.000888236s 200
2024-06-11T08:10:47.652Z [pebble] GET /v1/notices?after=2024-06-11T08%3A08%3A17.572753164Z&timeout=30s 30.000410318s 200
2024-06-11T08:11:17.654Z [pebble] GET /v1/notices?after=2024-06-11T08%3A08%3A17.572753164Z&timeout=30s 30.001256712s 200
2024-06-11T08:11:47.655Z [pebble] GET /v1/notices?after=2024-06-11T08%3A08%3A17.572753164Z&timeout=30s 30.000528584s 200
2024-06-11T08:12:17.656Z [pebble] GET /v1/notices?after=2024-06-11T08%3A08%3A17.572753164Z&timeout=30s 30.00049291s 200
2024-06-11T08:12:47.657Z [pebble] GET /v1/notices?after=2024-06-11T08%3A08%3A17.572753164Z&timeout=30s 30.000524557s 200

─$ juju debug-log -i argo-controller -l info
unit-argo-controller-0: 11:04:24 INFO juju.cmd running containerAgent [3.5.0 01a596920ff5a632babf5aacf485cc78fe1ed5df gc go1.21.9]
unit-argo-controller-0: 11:04:24 INFO juju.cmd.containeragent.unit start "unit"
unit-argo-controller-0: 11:04:24 INFO juju.worker.upgradesteps upgrade steps for 3.5.0 have already been run.
unit-argo-controller-0: 11:04:24 INFO juju.worker.probehttpserver starting http server on 127.0.0.1:65301
unit-argo-controller-0: 11:04:24 INFO juju.api connection established to "wss://10.0.230.216:17070/model/b11aee1c-fb16-4a93-85ec-d6a3785776ca/api"
unit-argo-controller-0: 11:04:24 INFO juju.worker.apicaller [b11aee] "unit-argo-controller-0" successfully connected to "10.0.230.216:17070"
unit-argo-controller-0: 11:04:24 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-argo-controller-0: 11:04:24 INFO juju.worker.logger logger worker started
unit-argo-controller-0: 11:04:24 WARNING juju.worker.proxyupdater unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: ""
unit-argo-controller-0: 11:04:43 INFO unit.argo-controller/0.juju-log Running legacy hooks/install.
unit-argo-controller-0: 11:04:44 INFO unit.argo-controller/0.juju-log HTTP Request: GET https://10.0.0.1/api/v1/namespaces/kubeflow/services/argo-controller "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:04:44 INFO unit.argo-controller/0.juju-log HTTP Request: PATCH https://10.0.0.1/api/v1/namespaces/kubeflow/services/argo-controller "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:04:44 INFO unit.argo-controller/0.juju-log Kubernetes service 'argo-controller' patched successfully
unit-argo-controller-0: 11:04:44 INFO unit.argo-controller/0.juju-log Starting `execute_components` for event 'ArgoControllerOperator/on/install[1]'
unit-argo-controller-0: 11:04:44 INFO unit.argo-controller/0.juju-log Executing component: 'leadership-gate'
unit-argo-controller-0: 11:04:44 INFO unit.argo-controller/0.juju-log Execution for component 'leadership-gate' complete.  Component now has status 'ActiveStatus('')'
unit-argo-controller-0: 11:04:44 INFO unit.argo-controller/0.juju-log Executing component: 'relation:object_storage'
unit-argo-controller-0: 11:04:45 INFO unit.argo-controller/0.juju-log Execution for component 'relation:object_storage' complete.  Component now has status 'BlockedStatus('Expected data from exactly 1 related applications - got 0.')'
unit-argo-controller-0: 11:04:45 INFO unit.argo-controller/0.juju-log execute_components execution loop complete.
unit-argo-controller-0: 11:04:45 INFO unit.argo-controller/0.juju-log Status of all CharmReconciler Components:
unit-argo-controller-0: 11:04:45 INFO unit.argo-controller/0.juju-log Status: BlockedStatus('[relation:object_storage] Expected data from exactly 1 related applications - got 0.')
unit-argo-controller-0: 11:04:45 INFO unit.argo-controller/0.juju-log Status: MaintenanceStatus("[kubernetes:crds-cm-and-secrets] Execution pending - waiting on relation:object_storage (BlockedStatus('Expected data from exactly 1 related applications - got 0.')).")
unit-argo-controller-0: 11:04:45 INFO unit.argo-controller/0.juju-log Status: MaintenanceStatus('[container:argo-controller] Execution pending - waiting on kubernetes:crds-cm-and-secrets (MaintenanceStatus("Execution pending - waiting on relation:object_storage (BlockedStatus(\'Expected data from exactly 1 related applications - got 0.\')).")), relation:object_storage (BlockedStatus(\'Expected data from exactly 1 related applications - got 0.\')).')
unit-argo-controller-0: 11:04:45 INFO unit.argo-controller/0.juju-log Status: ActiveStatus('[leadership-gate] ')
unit-argo-controller-0: 11:04:45 INFO unit.argo-controller/0.juju-log Status of unit set to: BlockedStatus('[relation:object_storage] Expected data from exactly 1 related applications - got 0.')
unit-argo-controller-0: 11:04:47 INFO unit.argo-controller/0.juju-log Starting `execute_components` for event 'ArgoControllerOperator/on/leader_elected[11]'
unit-argo-controller-0: 11:04:47 INFO unit.argo-controller/0.juju-log Executing component: 'leadership-gate'
unit-argo-controller-0: 11:04:47 INFO unit.argo-controller/0.juju-log Execution for component 'leadership-gate' complete.  Component now has status 'ActiveStatus('')'
unit-argo-controller-0: 11:04:47 INFO unit.argo-controller/0.juju-log Executing component: 'relation:object_storage'
unit-argo-controller-0: 11:04:47 INFO unit.argo-controller/0.juju-log Execution for component 'relation:object_storage' complete.  Component now has status 'WaitingStatus('List of <ops.model.Relation object-storage:8> versions not found for apps: minio')'
unit-argo-controller-0: 11:04:47 INFO unit.argo-controller/0.juju-log execute_components execution loop complete.
unit-argo-controller-0: 11:04:47 INFO unit.argo-controller/0.juju-log Status of all CharmReconciler Components:
unit-argo-controller-0: 11:04:47 INFO unit.argo-controller/0.juju-log Status: WaitingStatus('[relation:object_storage] List of <ops.model.Relation object-storage:8> versions not found for apps: minio')
unit-argo-controller-0: 11:04:47 INFO unit.argo-controller/0.juju-log Status: MaintenanceStatus("[kubernetes:crds-cm-and-secrets] Execution pending - waiting on relation:object_storage (WaitingStatus('List of <ops.model.Relation object-storage:8> versions not found for apps: minio')).")
unit-argo-controller-0: 11:04:47 INFO unit.argo-controller/0.juju-log Status: MaintenanceStatus('[container:argo-controller] Execution pending - waiting on kubernetes:crds-cm-and-secrets (MaintenanceStatus("Execution pending - waiting on relation:object_storage (WaitingStatus(\'List of <ops.model.Relation object-storage:8> versions not found for apps: minio\')).")), relation:object_storage (WaitingStatus(\'List of <ops.model.Relation object-storage:8> versions not found for apps: minio\')).')
unit-argo-controller-0: 11:04:47 INFO unit.argo-controller/0.juju-log Status: ActiveStatus('[leadership-gate] ')
unit-argo-controller-0: 11:04:47 INFO unit.argo-controller/0.juju-log Status of unit set to: WaitingStatus('[relation:object_storage] List of <ops.model.Relation object-storage:8> versions not found for apps: minio')
unit-argo-controller-0: 11:04:48 INFO unit.argo-controller/0.juju-log Starting `execute_components` for event 'ArgoControllerOperator/on/argo_controller_pebble_ready[16]'
unit-argo-controller-0: 11:04:48 INFO unit.argo-controller/0.juju-log Executing component: 'leadership-gate'
unit-argo-controller-0: 11:04:48 INFO unit.argo-controller/0.juju-log Execution for component 'leadership-gate' complete.  Component now has status 'ActiveStatus('')'
unit-argo-controller-0: 11:04:48 INFO unit.argo-controller/0.juju-log Executing component: 'relation:object_storage'
unit-argo-controller-0: 11:04:48 INFO unit.argo-controller/0.juju-log Execution for component 'relation:object_storage' complete.  Component now has status 'WaitingStatus('List of <ops.model.Relation object-storage:8> versions not found for apps: minio')'
unit-argo-controller-0: 11:04:48 INFO unit.argo-controller/0.juju-log execute_components execution loop complete.
unit-argo-controller-0: 11:04:48 INFO unit.argo-controller/0.juju-log Status of all CharmReconciler Components:
unit-argo-controller-0: 11:04:48 INFO unit.argo-controller/0.juju-log Status: WaitingStatus('[relation:object_storage] List of <ops.model.Relation object-storage:8> versions not found for apps: minio')
unit-argo-controller-0: 11:04:48 INFO unit.argo-controller/0.juju-log Status: MaintenanceStatus("[kubernetes:crds-cm-and-secrets] Execution pending - waiting on relation:object_storage (WaitingStatus('List of <ops.model.Relation object-storage:8> versions not found for apps: minio')).")
unit-argo-controller-0: 11:04:48 INFO unit.argo-controller/0.juju-log Status: MaintenanceStatus('[container:argo-controller] Execution pending - waiting on kubernetes:crds-cm-and-secrets (MaintenanceStatus("Execution pending - waiting on relation:object_storage (WaitingStatus(\'List of <ops.model.Relation object-storage:8> versions not found for apps: minio\')).")), relation:object_storage (WaitingStatus(\'List of <ops.model.Relation object-storage:8> versions not found for apps: minio\')).')
unit-argo-controller-0: 11:04:48 INFO unit.argo-controller/0.juju-log Status: ActiveStatus('[leadership-gate] ')
unit-argo-controller-0: 11:04:48 INFO unit.argo-controller/0.juju-log Status of unit set to: WaitingStatus('[relation:object_storage] List of <ops.model.Relation object-storage:8> versions not found for apps: minio')
unit-argo-controller-0: 11:04:49 INFO unit.argo-controller/0.juju-log Starting `execute_components` for event 'ArgoControllerOperator/on/config_changed[21]'
unit-argo-controller-0: 11:04:49 INFO unit.argo-controller/0.juju-log Executing component: 'leadership-gate'
unit-argo-controller-0: 11:04:49 INFO unit.argo-controller/0.juju-log Execution for component 'leadership-gate' complete.  Component now has status 'ActiveStatus('')'
unit-argo-controller-0: 11:04:49 INFO unit.argo-controller/0.juju-log Executing component: 'relation:object_storage'
unit-argo-controller-0: 11:04:49 INFO unit.argo-controller/0.juju-log Execution for component 'relation:object_storage' complete.  Component now has status 'WaitingStatus('List of <ops.model.Relation object-storage:8> versions not found for apps: minio')'
unit-argo-controller-0: 11:04:49 INFO unit.argo-controller/0.juju-log execute_components execution loop complete.
unit-argo-controller-0: 11:04:50 INFO unit.argo-controller/0.juju-log Status of all CharmReconciler Components:
unit-argo-controller-0: 11:04:50 INFO unit.argo-controller/0.juju-log Status: WaitingStatus('[relation:object_storage] List of <ops.model.Relation object-storage:8> versions not found for apps: minio')
unit-argo-controller-0: 11:04:50 INFO unit.argo-controller/0.juju-log Status: MaintenanceStatus("[kubernetes:crds-cm-and-secrets] Execution pending - waiting on relation:object_storage (WaitingStatus('List of <ops.model.Relation object-storage:8> versions not found for apps: minio')).")
unit-argo-controller-0: 11:04:50 INFO unit.argo-controller/0.juju-log Status: MaintenanceStatus('[container:argo-controller] Execution pending - waiting on kubernetes:crds-cm-and-secrets (MaintenanceStatus("Execution pending - waiting on relation:object_storage (WaitingStatus(\'List of <ops.model.Relation object-storage:8> versions not found for apps: minio\')).")), relation:object_storage (WaitingStatus(\'List of <ops.model.Relation object-storage:8> versions not found for apps: minio\')).')
unit-argo-controller-0: 11:04:50 INFO unit.argo-controller/0.juju-log Status: ActiveStatus('[leadership-gate] ')
unit-argo-controller-0: 11:04:50 INFO unit.argo-controller/0.juju-log Status of unit set to: WaitingStatus('[relation:object_storage] List of <ops.model.Relation object-storage:8> versions not found for apps: minio')
unit-argo-controller-0: 11:04:51 INFO unit.argo-controller/0.juju-log Running legacy hooks/start.
unit-argo-controller-0: 11:04:54 INFO unit.argo-controller/0.juju-log Starting `execute_components` for event 'ArgoControllerOperator/on/config_changed[31]'
unit-argo-controller-0: 11:04:54 INFO unit.argo-controller/0.juju-log Executing component: 'leadership-gate'
unit-argo-controller-0: 11:04:54 INFO unit.argo-controller/0.juju-log Execution for component 'leadership-gate' complete.  Component now has status 'ActiveStatus('')'
unit-argo-controller-0: 11:04:54 INFO unit.argo-controller/0.juju-log Executing component: 'relation:object_storage'
unit-argo-controller-0: 11:04:54 INFO unit.argo-controller/0.juju-log Execution for component 'relation:object_storage' complete.  Component now has status 'WaitingStatus('List of <ops.model.Relation object-storage:8> versions not found for apps: minio')'
unit-argo-controller-0: 11:04:54 INFO unit.argo-controller/0.juju-log execute_components execution loop complete.
unit-argo-controller-0: 11:04:54 INFO unit.argo-controller/0.juju-log Status of all CharmReconciler Components:
unit-argo-controller-0: 11:04:54 INFO unit.argo-controller/0.juju-log Status: WaitingStatus('[relation:object_storage] List of <ops.model.Relation object-storage:8> versions not found for apps: minio')
unit-argo-controller-0: 11:04:54 INFO unit.argo-controller/0.juju-log Status: MaintenanceStatus("[kubernetes:crds-cm-and-secrets] Execution pending - waiting on relation:object_storage (WaitingStatus('List of <ops.model.Relation object-storage:8> versions not found for apps: minio')).")
unit-argo-controller-0: 11:04:54 INFO unit.argo-controller/0.juju-log Status: MaintenanceStatus('[container:argo-controller] Execution pending - waiting on kubernetes:crds-cm-and-secrets (MaintenanceStatus("Execution pending - waiting on relation:object_storage (WaitingStatus(\'List of <ops.model.Relation object-storage:8> versions not found for apps: minio\')).")), relation:object_storage (WaitingStatus(\'List of <ops.model.Relation object-storage:8> versions not found for apps: minio\')).')
unit-argo-controller-0: 11:04:54 INFO unit.argo-controller/0.juju-log Status: ActiveStatus('[leadership-gate] ')
unit-argo-controller-0: 11:04:54 INFO unit.argo-controller/0.juju-log Status of unit set to: WaitingStatus('[relation:object_storage] List of <ops.model.Relation object-storage:8> versions not found for apps: minio')
unit-argo-controller-0: 11:06:20 INFO unit.argo-controller/0.juju-log object-storage:8: Starting `execute_components` for event 'ArgoControllerOperator/on/object_storage_relation_changed[41]'
unit-argo-controller-0: 11:06:20 INFO unit.argo-controller/0.juju-log object-storage:8: Executing component: 'leadership-gate'
unit-argo-controller-0: 11:06:20 INFO unit.argo-controller/0.juju-log object-storage:8: Execution for component 'leadership-gate' complete.  Component now has status 'ActiveStatus('')'
unit-argo-controller-0: 11:06:20 INFO unit.argo-controller/0.juju-log object-storage:8: Executing component: 'relation:object_storage'
unit-argo-controller-0: 11:06:20 INFO unit.argo-controller/0.juju-log object-storage:8: Execution for component 'relation:object_storage' complete.  Component now has status 'WaitingStatus('List of <ops.model.Relation object-storage:8> versions not found for apps: minio')'
unit-argo-controller-0: 11:06:20 INFO unit.argo-controller/0.juju-log object-storage:8: execute_components execution loop complete.
unit-argo-controller-0: 11:06:20 INFO unit.argo-controller/0.juju-log object-storage:8: Status of all CharmReconciler Components:
unit-argo-controller-0: 11:06:20 INFO unit.argo-controller/0.juju-log object-storage:8: Status: WaitingStatus('[relation:object_storage] List of <ops.model.Relation object-storage:8> versions not found for apps: minio')
unit-argo-controller-0: 11:06:20 INFO unit.argo-controller/0.juju-log object-storage:8: Status: MaintenanceStatus("[kubernetes:crds-cm-and-secrets] Execution pending - waiting on relation:object_storage (WaitingStatus('List of <ops.model.Relation object-storage:8> versions not found for apps: minio')).")
unit-argo-controller-0: 11:06:20 INFO unit.argo-controller/0.juju-log object-storage:8: Status: MaintenanceStatus('[container:argo-controller] Execution pending - waiting on kubernetes:crds-cm-and-secrets (MaintenanceStatus("Execution pending - waiting on relation:object_storage (WaitingStatus(\'List of <ops.model.Relation object-storage:8> versions not found for apps: minio\')).")), relation:object_storage (WaitingStatus(\'List of <ops.model.Relation object-storage:8> versions not found for apps: minio\')).')
unit-argo-controller-0: 11:06:20 INFO unit.argo-controller/0.juju-log object-storage:8: Status: ActiveStatus('[leadership-gate] ')
unit-argo-controller-0: 11:06:20 INFO unit.argo-controller/0.juju-log object-storage:8: Status of unit set to: WaitingStatus('[relation:object_storage] List of <ops.model.Relation object-storage:8> versions not found for apps: minio')
unit-argo-controller-0: 11:06:31 INFO unit.argo-controller/0.juju-log object-storage:8: Starting `execute_components` for event 'ArgoControllerOperator/on/object_storage_relation_changed[46]'
unit-argo-controller-0: 11:06:31 INFO unit.argo-controller/0.juju-log object-storage:8: Executing component: 'leadership-gate'
unit-argo-controller-0: 11:06:31 INFO unit.argo-controller/0.juju-log object-storage:8: Execution for component 'leadership-gate' complete.  Component now has status 'ActiveStatus('')'
unit-argo-controller-0: 11:06:31 INFO unit.argo-controller/0.juju-log object-storage:8: Executing component: 'relation:object_storage'
unit-argo-controller-0: 11:06:31 INFO unit.argo-controller/0.juju-log object-storage:8: Execution for component 'relation:object_storage' complete.  Component now has status 'BlockedStatus('Expected data from exactly 1 related applications - got 0.')'
unit-argo-controller-0: 11:06:31 INFO unit.argo-controller/0.juju-log object-storage:8: execute_components execution loop complete.
unit-argo-controller-0: 11:06:31 INFO unit.argo-controller/0.juju-log object-storage:8: Status of all CharmReconciler Components:
unit-argo-controller-0: 11:06:31 INFO unit.argo-controller/0.juju-log object-storage:8: Status: BlockedStatus('[relation:object_storage] Expected data from exactly 1 related applications - got 0.')
unit-argo-controller-0: 11:06:31 INFO unit.argo-controller/0.juju-log object-storage:8: Status: MaintenanceStatus("[kubernetes:crds-cm-and-secrets] Execution pending - waiting on relation:object_storage (BlockedStatus('Expected data from exactly 1 related applications - got 0.')).")
unit-argo-controller-0: 11:06:31 INFO unit.argo-controller/0.juju-log object-storage:8: Status: MaintenanceStatus('[container:argo-controller] Execution pending - waiting on kubernetes:crds-cm-and-secrets (MaintenanceStatus("Execution pending - waiting on relation:object_storage (BlockedStatus(\'Expected data from exactly 1 related applications - got 0.\')).")), relation:object_storage (BlockedStatus(\'Expected data from exactly 1 related applications - got 0.\')).')
unit-argo-controller-0: 11:06:31 INFO unit.argo-controller/0.juju-log object-storage:8: Status: ActiveStatus('[leadership-gate] ')
unit-argo-controller-0: 11:06:31 INFO unit.argo-controller/0.juju-log object-storage:8: Status of unit set to: BlockedStatus('[relation:object_storage] Expected data from exactly 1 related applications - got 0.')
unit-argo-controller-0: 11:07:05 INFO unit.argo-controller/0.juju-log object-storage:8: Starting `execute_components` for event 'ArgoControllerOperator/on/object_storage_relation_changed[51]'
unit-argo-controller-0: 11:07:05 INFO unit.argo-controller/0.juju-log object-storage:8: Executing component: 'leadership-gate'
unit-argo-controller-0: 11:07:05 INFO unit.argo-controller/0.juju-log object-storage:8: Execution for component 'leadership-gate' complete.  Component now has status 'ActiveStatus('')'
unit-argo-controller-0: 11:07:05 INFO unit.argo-controller/0.juju-log object-storage:8: Executing component: 'relation:object_storage'
unit-argo-controller-0: 11:07:06 INFO unit.argo-controller/0.juju-log object-storage:8: Execution for component 'relation:object_storage' complete.  Component now has status 'BlockedStatus('Expected data from exactly 1 related applications - got 0.')'
unit-argo-controller-0: 11:07:06 INFO unit.argo-controller/0.juju-log object-storage:8: execute_components execution loop complete.
unit-argo-controller-0: 11:07:06 INFO unit.argo-controller/0.juju-log object-storage:8: Status of all CharmReconciler Components:
unit-argo-controller-0: 11:07:06 INFO unit.argo-controller/0.juju-log object-storage:8: Status: BlockedStatus('[relation:object_storage] Expected data from exactly 1 related applications - got 0.')
unit-argo-controller-0: 11:07:06 INFO unit.argo-controller/0.juju-log object-storage:8: Status: MaintenanceStatus("[kubernetes:crds-cm-and-secrets] Execution pending - waiting on relation:object_storage (BlockedStatus('Expected data from exactly 1 related applications - got 0.')).")
unit-argo-controller-0: 11:07:06 INFO unit.argo-controller/0.juju-log object-storage:8: Status: MaintenanceStatus('[container:argo-controller] Execution pending - waiting on kubernetes:crds-cm-and-secrets (MaintenanceStatus("Execution pending - waiting on relation:object_storage (BlockedStatus(\'Expected data from exactly 1 related applications - got 0.\')).")), relation:object_storage (BlockedStatus(\'Expected data from exactly 1 related applications - got 0.\')).')
unit-argo-controller-0: 11:07:06 INFO unit.argo-controller/0.juju-log object-storage:8: Status: ActiveStatus('[leadership-gate] ')
unit-argo-controller-0: 11:07:06 INFO unit.argo-controller/0.juju-log object-storage:8: Status of unit set to: BlockedStatus('[relation:object_storage] Expected data from exactly 1 related applications - got 0.')
unit-argo-controller-0: 11:08:06 INFO unit.argo-controller/0.juju-log object-storage:8: Starting `execute_components` for event 'ArgoControllerOperator/on/object_storage_relation_changed[56]'
unit-argo-controller-0: 11:08:06 INFO unit.argo-controller/0.juju-log object-storage:8: Executing component: 'leadership-gate'
unit-argo-controller-0: 11:08:06 INFO unit.argo-controller/0.juju-log object-storage:8: Execution for component 'leadership-gate' complete.  Component now has status 'ActiveStatus('')'
unit-argo-controller-0: 11:08:06 INFO unit.argo-controller/0.juju-log object-storage:8: Executing component: 'relation:object_storage'
unit-argo-controller-0: 11:08:06 INFO unit.argo-controller/0.juju-log object-storage:8: Execution for component 'relation:object_storage' complete.  Component now has status 'ActiveStatus('')'
unit-argo-controller-0: 11:08:06 INFO unit.argo-controller/0.juju-log object-storage:8: Executing component: 'kubernetes:crds-cm-and-secrets'
unit-argo-controller-0: 11:08:07 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:08 INFO unit.argo-controller/0.juju-log object-storage:8: Rendering manifests
unit-argo-controller-0: 11:08:08 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: PATCH https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions/clusterworkflowtemplates.argoproj.io?force=true&fieldManager=lightkube "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:08 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: PATCH https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions/compositecontrollers.metacontroller.k8s.io?force=true&fieldManager=lightkube "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:08 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: PATCH https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions/controllerrevisions.metacontroller.k8s.io?force=true&fieldManager=lightkube "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:09 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: PATCH https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions/cronworkflows.argoproj.io?force=true&fieldManager=lightkube "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:09 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: PATCH https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions/decoratorcontrollers.metacontroller.k8s.io?force=true&fieldManager=lightkube "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:09 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: PATCH https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions/scheduledworkflows.kubeflow.org?force=true&fieldManager=lightkube "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:09 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: PATCH https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions/viewers.kubeflow.org?force=true&fieldManager=lightkube "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:09 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: PATCH https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions/workflowartifactgctasks.argoproj.io?force=true&fieldManager=lightkube "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:09 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: PATCH https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions/workfloweventbindings.argoproj.io?force=true&fieldManager=lightkube "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:09 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: PATCH https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions/workflows.argoproj.io?force=true&fieldManager=lightkube "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:10 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: PATCH https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions/workflowtaskresults.argoproj.io?force=true&fieldManager=lightkube "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:10 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: PATCH https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions/workflowtasksets.argoproj.io?force=true&fieldManager=lightkube "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:10 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: PATCH https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions/workflowtemplates.argoproj.io?force=true&fieldManager=lightkube "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:10 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: PATCH https://10.0.0.1/api/v1/namespaces/kubeflow/secrets/mlpipeline-minio-artifact?force=true&fieldManager=lightkube "HTTP/1.1 201 Created"
unit-argo-controller-0: 11:08:10 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: PATCH https://10.0.0.1/api/v1/namespaces/kubeflow/configmaps/argo-workflow-controller-configmap?force=true&fieldManager=lightkube "HTTP/1.1 201 Created"
unit-argo-controller-0: 11:08:10 INFO unit.argo-controller/0.juju-log object-storage:8: Reconcile completed successfully
unit-argo-controller-0: 11:08:12 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:12 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/api/v1/configmaps?labelSelector=app.kubernetes.io%2Finstance%3Dargo-controller-kubeflow%2Ckubernetes-resource-handler-scope%3Dcrds-cm-and-secrets "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:12 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions?labelSelector=app.kubernetes.io%2Finstance%3Dargo-controller-kubeflow%2Ckubernetes-resource-handler-scope%3Dcrds-cm-and-secrets "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:13 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/api/v1/secrets?labelSelector=app.kubernetes.io%2Finstance%3Dargo-controller-kubeflow%2Ckubernetes-resource-handler-scope%3Dcrds-cm-and-secrets "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:13 INFO unit.argo-controller/0.juju-log object-storage:8: Rendering manifests
unit-argo-controller-0: 11:08:13 INFO unit.argo-controller/0.juju-log object-storage:8: Execution for component 'kubernetes:crds-cm-and-secrets' complete.  Component now has status 'ActiveStatus('')'
unit-argo-controller-0: 11:08:15 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:15 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/api/v1/configmaps?labelSelector=app.kubernetes.io%2Finstance%3Dargo-controller-kubeflow%2Ckubernetes-resource-handler-scope%3Dcrds-cm-and-secrets "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:15 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions?labelSelector=app.kubernetes.io%2Finstance%3Dargo-controller-kubeflow%2Ckubernetes-resource-handler-scope%3Dcrds-cm-and-secrets "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:15 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/api/v1/secrets?labelSelector=app.kubernetes.io%2Finstance%3Dargo-controller-kubeflow%2Ckubernetes-resource-handler-scope%3Dcrds-cm-and-secrets "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:15 INFO unit.argo-controller/0.juju-log object-storage:8: Rendering manifests
unit-argo-controller-0: 11:08:16 INFO unit.argo-controller/0.juju-log object-storage:8: Executing component: 'container:argo-controller'
unit-argo-controller-0: 11:08:16 INFO unit.argo-controller/0.juju-log object-storage:8: PebbleServiceComponent.get_layer executing
unit-argo-controller-0: 11:08:17 INFO unit.argo-controller/0.juju-log object-storage:8: PebbleServiceComponent.get_layer executing
unit-argo-controller-0: 11:08:17 INFO unit.argo-controller/0.juju-log object-storage:8: Execution for component 'container:argo-controller' complete.  Component now has status 'ActiveStatus('')'
unit-argo-controller-0: 11:08:18 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:19 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/api/v1/configmaps?labelSelector=app.kubernetes.io%2Finstance%3Dargo-controller-kubeflow%2Ckubernetes-resource-handler-scope%3Dcrds-cm-and-secrets "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:19 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions?labelSelector=app.kubernetes.io%2Finstance%3Dargo-controller-kubeflow%2Ckubernetes-resource-handler-scope%3Dcrds-cm-and-secrets "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:19 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/api/v1/secrets?labelSelector=app.kubernetes.io%2Finstance%3Dargo-controller-kubeflow%2Ckubernetes-resource-handler-scope%3Dcrds-cm-and-secrets "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:19 INFO unit.argo-controller/0.juju-log object-storage:8: Rendering manifests
unit-argo-controller-0: 11:08:20 INFO unit.argo-controller/0.juju-log object-storage:8: execute_components execution loop complete.
unit-argo-controller-0: 11:08:21 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:21 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/api/v1/configmaps?labelSelector=app.kubernetes.io%2Finstance%3Dargo-controller-kubeflow%2Ckubernetes-resource-handler-scope%3Dcrds-cm-and-secrets "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:22 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions?labelSelector=app.kubernetes.io%2Finstance%3Dargo-controller-kubeflow%2Ckubernetes-resource-handler-scope%3Dcrds-cm-and-secrets "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:22 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/api/v1/secrets?labelSelector=app.kubernetes.io%2Finstance%3Dargo-controller-kubeflow%2Ckubernetes-resource-handler-scope%3Dcrds-cm-and-secrets "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:22 INFO unit.argo-controller/0.juju-log object-storage:8: Rendering manifests
unit-argo-controller-0: 11:08:23 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:23 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/api/v1/configmaps?labelSelector=app.kubernetes.io%2Finstance%3Dargo-controller-kubeflow%2Ckubernetes-resource-handler-scope%3Dcrds-cm-and-secrets "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:23 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions?labelSelector=app.kubernetes.io%2Finstance%3Dargo-controller-kubeflow%2Ckubernetes-resource-handler-scope%3Dcrds-cm-and-secrets "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:24 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/api/v1/secrets?labelSelector=app.kubernetes.io%2Finstance%3Dargo-controller-kubeflow%2Ckubernetes-resource-handler-scope%3Dcrds-cm-and-secrets "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:24 INFO unit.argo-controller/0.juju-log object-storage:8: Rendering manifests
unit-argo-controller-0: 11:08:25 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:25 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/api/v1/configmaps?labelSelector=app.kubernetes.io%2Finstance%3Dargo-controller-kubeflow%2Ckubernetes-resource-handler-scope%3Dcrds-cm-and-secrets "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:25 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions?labelSelector=app.kubernetes.io%2Finstance%3Dargo-controller-kubeflow%2Ckubernetes-resource-handler-scope%3Dcrds-cm-and-secrets "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:26 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/api/v1/secrets?labelSelector=app.kubernetes.io%2Finstance%3Dargo-controller-kubeflow%2Ckubernetes-resource-handler-scope%3Dcrds-cm-and-secrets "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:26 INFO unit.argo-controller/0.juju-log object-storage:8: Rendering manifests
unit-argo-controller-0: 11:08:26 INFO unit.argo-controller/0.juju-log object-storage:8: Status of all CharmReconciler Components:
unit-argo-controller-0: 11:08:26 INFO unit.argo-controller/0.juju-log object-storage:8: Status: BlockedStatus("[kubernetes:crds-cm-and-secrets] Not all resources found in cluster.  This may be transient if we haven't tried to deploy them yet.")
unit-argo-controller-0: 11:08:26 INFO unit.argo-controller/0.juju-log object-storage:8: Status: MaintenanceStatus('[container:argo-controller] Execution pending - waiting on kubernetes:crds-cm-and-secrets (BlockedStatus("Not all resources found in cluster.  This may be transient if we haven\'t tried to deploy them yet.")).')
unit-argo-controller-0: 11:08:26 INFO unit.argo-controller/0.juju-log object-storage:8: Status: ActiveStatus('[leadership-gate] ')
unit-argo-controller-0: 11:08:26 INFO unit.argo-controller/0.juju-log object-storage:8: Status: ActiveStatus('[relation:object_storage] ')
unit-argo-controller-0: 11:08:26 INFO unit.argo-controller/0.juju-log object-storage:8: Status of unit set to: BlockedStatus("[kubernetes:crds-cm-and-secrets] Not all resources found in cluster.  This may be transient if we haven't tried to deploy them yet.")

Additional Context

No response

orfeas-k commented 3 weeks ago

I deployed to a different cluster and observed the same behaviour with 2 other charms


kfp-profile-controller/0*     blocked      idle   10.244.0.21                 [kubernetes:secrets-and-compositecontroller] Not all resources found in cluster.  This may be transient if we haven't...
kfp-schedwf/0*                maintenance  idle   10.244.0.22                 [kfp-schedwf-pebble-service] Execution pending - waiting on kubernetes:auth-and-crds (BlockedStatus("Not all resource...
syncronize-issues-to-jira[bot] commented 2 weeks ago

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5883.

This message was autogenerated