search

nerc-project / operations

Issues related to the operation of the NERC OpenShift environment
2 stars 0 forks source link

Degraded argocd apps #687

Open larsks opened 3 months ago

larsks commented 3 months ago

@naved001 reports in slack:

FYI, the argocd app for cluster-scope-prod is degraded and out of sync. So, it’s not syncing automatically (I had to manually sync some of my changes after my PR was merged). app for nerc-ocp-infra is aslo degraded (among many other apps that are degraded)

Looking at the all the apps, we see:

$ k get apps | grep -i degraded
cluster-scope-infra                                              OutOfSync     Degraded
cluster-scope-prod                                               OutOfSync     Degraded
curator-prod                                                     OutOfSync     Degraded
dex-infra                                                        Synced        Degraded
grafana-infra                                                    Synced        Degraded
logging-infra                                                    Synced        Degraded
loki-infra                                                       Synced        Degraded
vault-backup-job-infra                                           Synced        Degraded

I'm looking into this to see what's failing.

larsks commented 2 months ago

The cluster-scope-prod overlay is failing to sync because:

error validating data: ValidationError(OdhDashboardConfig.spec.dashboardConfig): unknown field "modelMetricsNamespace" in io.opendatahub.v1alpha.OdhDashboardConfig.spec.dashboardConfig