search

gardener-attic / issues-foo

0 stars 1 forks source link

aaa #183

Closed neo-liang-sap closed 1 year ago

neo-liang-sap commented 1 year ago

Which cluster is affected?

Cluster Details Dashboard Link: URL

What happened?

What you expected to happen?

When did it happen or started to happen?

Absolute: Relative:

How would we reproduce it (concisely and precisely)?

Anything else we need to know?

Help us categorise this issue for faster resolution:

/area audit-logging auto-scaling backup certification control-plane cost delivery disaster-recovery documentation high-availability logging metering monitoring networking os performance quality security storage usability user-management /component gardener dashboard documentation /kind bug regression post-mortem /os garden-linux suse-chost /platform alicloud aws azure gcp converged-cloud

/priority normal

gardener-robot commented 1 year ago

@neo-liang-sap No more than 5 labels permitted, but 21 labels were given.

gardener-robot commented 1 year ago

@neo-liang-sap

Shoot: hubtest/ikdevtest v1.20.15             created at 2020-05-20 06:20 by Ingo Kober (BTP FP Gardener (SE))             on gcp in europe-west1 with purpose development             currently hibernated Seed: garden/gcp

🔴 Last Operation ``` description: 'Flow "Shoot cluster reconciliation" encountered task errors: [task "Waiting until shoot worker nodes have been reconciled" failed: Error while waiting for Worker shoot--hubtest--ikdevtest/ikdevtest to become ready: error during reconciliation: Error reconciling Worker: Failed while waiting for all machine deployments to be ready: retry failed with context deadline exceeded, last error: Waiting until all machines have been hibernated (1 still awake)...] Operation will be retried.' lastUpdateTime: '2023-01-03T01:36:05Z' progress: 81 state: Error type: Reconcile ```           **Last Error** ``` description: 'task "Waiting until shoot worker nodes have been reconciled" failed: Error while waiting for Worker shoot--hubtest--ikdevtest/ikdevtest to become ready: error during reconciliation: Error reconciling Worker: Failed while waiting for all machine deployments to be ready: retry failed with context deadline exceeded, last error: Waiting until all machines have been hibernated (1 still awake)...' lastUpdateTime: '2023-01-03T01:36:05Z' taskID: Waiting until shoot worker nodes have been reconciled ```
đŸŸĸ Shoot Conditions           đŸ’¤ APIServerAvailable (ConditionNotChecked)           đŸ’¤ ControlPlaneHealthy (ConditionNotChecked)           đŸ’¤ EveryNodeReady (ConditionNotChecked)           đŸ’¤ SystemComponentsHealthy (ConditionNotChecked)           đŸ’¤ HibernationPossible (ConstraintNotChecked)           đŸ’¤ MaintenancePreconditionsSatisfied (ConstraintNotChecked)           đŸ’¤ CACertificateValiditiesAcceptable (ConstraintNotChecked)
đŸŸĸ Seed Conditions           đŸŸĸ GardenletReady (GardenletReady)           đŸŸĸ Bootstrapped (BootstrappingSucceeded)           đŸŸĸ ExtensionsReady (AllExtensionsReady)           đŸŸĸ AuditlogServiceAvailability (AuditlogInstanceAttached)           đŸŸĸ BackupBucketsReady (BackupBucketsAvailable)           đŸŸĸ SeedSystemComponentsHealthy (SystemComponentsRunning)
đŸŸĸ Worker Pools | *Name* | *OS* | *Machine* | *Zones* | | :- | :- | :- | :- | | **`worker-fkdbj`**
`1:2` `-0` `+1` | `gardenlinux`
`v576.12.0` | `n1-standard-2`
`20Gi` `pd-standard` | `europe-west1-c`
  |
gardener-robot commented 1 year ago

@gardener-robot You have mentioned internal references in the public. Please check.

gardener-robot commented 1 year ago

This ticket has not been touched since 1 days. Please add a follow up comment and/or change the status/ label.

neo-liang-sap commented 1 year ago

/priority blocker

gardener-robot commented 1 year ago

@neo-liang-sap ℹī¸ Please take note of this issue.

gardener-robot commented 1 year ago

@neo-liang-sap has acknowledged the VictorOps alert and looking into this issue

gardener-robot commented 1 year ago

@neo-liang-sap This issue has not been touched since 1 days. Please add a follow up comment and/or change the status/ label.

gardener-robot commented 1 year ago

@neo-liang-sap [new] This issue has not been touched since 7 days. Please add a follow up comment and/or change the status/ label.

gardener-robot commented 1 year ago

@neo-liang-sap This issue has not been touched since 8 days. Please add a follow up comment and/or change the status/ label.

gardener-robot commented 1 year ago

@etiennnr This issue has not been touched since 10 work days. Please add a follow up comment and/or change the status/ label.

gardener-robot commented 1 year ago

@etiennnr This issue has not been touched since 11 work days. Please add a follow up comment and/or change the status/ label.

gardener-robot commented 1 year ago

@etiennnr This issue has not been touched since 12 work days. Please add a follow up comment and/or change the status/ label.

gardener-robot commented 1 year ago

@neo-liang-sap This issue has not been touched since 59 work days. Please add a follow up comment and/or change the status/ label.

gardener-robot commented 1 year ago

@neo-liang-sap This issue has not been touched since 70 work days. Please add a follow up comment and/or change the status/ label.

gardener-robot commented 1 year ago

@neo-liang-sap This issue has not been touched since 73 work days. Please add a follow up comment and/or change the status/ label.

gardener-robot commented 1 year ago

@neo-liang-sap This issue has not been touched since 73 work days. Please add a follow up comment and/or change the status/ label.

gardener-robot commented 1 year ago

@neo-liang-sap This issue has not been touched since 75 work days. Please add a follow up comment and/or change the status/ label.

gardener-robot commented 1 year ago

@bd3lage This issue has not been touched since 77 work days. Please add a follow up comment and/or change the status/ label.

neo-liang-sap commented 1 year ago

/diag

neo-liang-sap commented 1 year ago

/diag

gardener-robot commented 1 year ago

@neo-liang-sap

Shoot: hubtest/ikdevtest v1.22.17             created at 2020-05-20 06:20 by Ingo Kober (BTP FP Gardener (SE))             on gcp in europe-west1 with purpose development             at 1 nodes and 21 pods and 36 API server requests/second (max of last 24h each) Seed: garden/gcp

🔴 Last Operation ``` description: "Flow \"Shoot cluster reconciliation\" encountered task errors: [task\ \ \"Waiting until shoot infrastructure has been reconciled\" failed: Error while\ \ waiting for Infrastructure shoot--hubtest--ikdevtest/ikdevtest to become ready:\ \ error during reconciliation: Error reconciling infrastructure: failed to apply\ \ the terraform config: Terraform execution for command 'apply' could not be completed:\n\ \n* Error when reading or editing ComputeNetwork \"projects/sap-gcp-k8s-wm-dev/global/networks/shoot--hubtest--ikdevtest\"\ : Get \"https://compute.googleapis.com/compute/v1/projects/sap-gcp-k8s-wm-dev/global/networks/shoot--hubtest--ikdevtest?alt=json\"\ : oauth2: cannot fetch token: 400 Bad Request\nResponse: {\"error\":\"invalid_grant\"\ ,\"error_description\":\"Invalid JWT Signature.\"}\n with google_compute_network.network,\n\ \ on main.tf line 20, in resource \"google_compute_network\" \"network\":\n 20:\ \ resource \"google_compute_network\" \"network\" {\n* Error when reading or editing\ \ Service Account \"projects/sap-gcp-k8s-wm-dev/serviceAccounts/shoot--hubtest--ikdevtest@sap-gcp-k8s-wm-dev.iam.gserviceaccount.com\"\ : Get \"https://iam.googleapis.com/v1/projects/sap-gcp-k8s-wm-dev/serviceAccounts/shoot--hubtest--ikdevtest@sap-gcp-k8s-wm-dev.iam.gserviceaccount.com?alt=json&prettyPrint=false\"\ : oauth2: cannot fetch token: 400 Bad Request\nResponse: {\"error\":\"invalid_grant\"\ ,\"error_description\":\"Invalid JWT Signature.\"}\n with google_service_account.serviceaccount,\n\ \ on main.tf line 11, in resource \"google_service_account\" \"serviceaccount\"\ :\n 11: resource \"google_service_account\" \"serviceaccount\" {]" lastUpdateTime: '2023-05-23T20:44:39Z' progress: 23 state: Failed type: Reconcile ```           **Last Error** ``` codes: - ERR_INFRA_UNAUTHENTICATED - ERR_INFRA_UNAUTHORIZED description: "task \"Waiting until shoot infrastructure has been reconciled\" failed:\ \ Error while waiting for Infrastructure shoot--hubtest--ikdevtest/ikdevtest to\ \ become ready: error during reconciliation: Error reconciling infrastructure: failed\ \ to apply the terraform config: Terraform execution for command 'apply' could not\ \ be completed:\n\n* Error when reading or editing ComputeNetwork \"projects/sap-gcp-k8s-wm-dev/global/networks/shoot--hubtest--ikdevtest\"\ : Get \"https://compute.googleapis.com/compute/v1/projects/sap-gcp-k8s-wm-dev/global/networks/shoot--hubtest--ikdevtest?alt=json\"\ : oauth2: cannot fetch token: 400 Bad Request\nResponse: {\"error\":\"invalid_grant\"\ ,\"error_description\":\"Invalid JWT Signature.\"}\n with google_compute_network.network,\n\ \ on main.tf line 20, in resource \"google_compute_network\" \"network\":\n 20:\ \ resource \"google_compute_network\" \"network\" {\n* Error when reading or editing\ \ Service Account \"projects/sap-gcp-k8s-wm-dev/serviceAccounts/shoot--hubtest--ikdevtest@sap-gcp-k8s-wm-dev.iam.gserviceaccount.com\"\ : Get \"https://iam.googleapis.com/v1/projects/sap-gcp-k8s-wm-dev/serviceAccounts/shoot--hubtest--ikdevtest@sap-gcp-k8s-wm-dev.iam.gserviceaccount.com?alt=json&prettyPrint=false\"\ : oauth2: cannot fetch token: 400 Bad Request\nResponse: {\"error\":\"invalid_grant\"\ ,\"error_description\":\"Invalid JWT Signature.\"}\n with google_service_account.serviceaccount,\n\ \ on main.tf line 11, in resource \"google_service_account\" \"serviceaccount\"\ :\n 11: resource \"google_service_account\" \"serviceaccount\" {" lastUpdateTime: '2023-05-23T20:44:39Z' taskID: Waiting until shoot infrastructure has been reconciled ```
đŸŸĸ Shoot Conditions           đŸ’¤ APIServerAvailable (ConditionNotChecked)           đŸ’¤ ControlPlaneHealthy (ConditionNotChecked)           đŸ’¤ ObservabilityComponentsHealthy (ConditionNotChecked)           đŸ’¤ SystemComponentsHealthy (ConditionNotChecked)           đŸ’¤ EveryNodeReady (ConditionNotChecked)           đŸ’¤ HibernationPossible (ConstraintNotChecked)           đŸ’¤ MaintenancePreconditionsSatisfied (ConstraintNotChecked)           đŸ’¤ CACertificateValiditiesAcceptable (ConstraintNotChecked)
đŸŸĸ Seed Conditions           đŸŸĸ GardenletReady (GardenletReady)           đŸŸĸ Bootstrapped (BootstrappingSucceeded)           đŸŸĸ ExtensionsReady (AllExtensionsReady)           đŸŸĸ AuditlogServiceAvailability (AuditlogInstanceAttached)           đŸŸĸ BackupBucketsReady (BackupBucketsAvailable)           đŸŸĸ SeedSystemComponentsHealthy (SystemComponentsRunning)
🔴 Control Plane Deployments Not Healthy | *Name* | *Desired* | *Current* | *Ready* | *Up-To-Date* | *Available* | | :- | -: | -: | -: | -: | -: | | **`cloud-controller`**`-manager` | đŸŸĸ `1` | đŸŸĸ `1` | 🔴 `0` | đŸŸĸ `1` | 🔴 `0` | | **`csi-driver`**`-controller` | đŸŸĸ `1` | đŸŸĸ `1` | 🔴 `0` | đŸŸĸ `1` | 🔴 `0` |
🔴 Control Plane Pods Not Healthy | *Name* | *Status* | *Age* | | :- | :- | :- | | **`cloud-controller-manager-65954b6b9b-fhqc8`**
`10.243.128.190` | Running
🔴 `0`/`1` Ready | 9 hours, 44 mins
🔴 `187` Restarts (`Error`) | | **`csi-driver-controller-57d48568c9-xlhtt`**
`10.243.140.28` | Running
🔴 `5`/`6` Ready | 9 hours, 44 mins
🔴 `194` Restarts (`Error`) |
🟠 Control Plane Events Not Healthy | *Timestamp* | *Severity* | *Reason* | *Object* | *Details* | | :- | :- | :- | :- | :- | | `14:16:15` on
`2023-05-23` | 🟠 `Warning` | `BackOff` | `pod`/
`csi-driver-controller-57d48568c9-xlhtt` | `Back-off restarting failed container` | | `14:16:23` on
`2023-05-23` | 🟠 `Warning` | `BackOff` | `pod`/
`cloud-controller-manager-65954b6b9b-fhqc8` | `Back-off restarting failed container` | | `20:01:21` on
`2023-05-23` | 🟠 `Warning` | `Unhealthy` | `pod`/
`cloud-controller-manager-65954b6b9b-fhqc8` | `Liveness probe failed: Get "https://10.243.128.190:10258/healthz": net/http: TLS handshake timeout` | | `20:56:19` on
`2023-05-23` | 🟠 `Warning` | `Unhealthy` | `pod`/
`csi-driver-controller-57d48568c9-xlhtt` | `Liveness probe failed: Get "http://10.243.140.28:9808/healthz": dial tcp 10.243.140.28:9808: connect: connection refused` |
đŸŸĸ Worker Pools | *Name* | *OS* | *Machine* | *Zones* | | :- | :- | :- | :- | | **`worker-fkdbj`**
`1:2` `-0` `+1` | `gardenlinux`
`v934.7.0` | `n1-standard-2`
`20Gi` `pd-standard` | `europe-west1-c`
  |
🟠 Worker Events Not Healthy | *Timestamp* | *Severity* | *Reason* | *Object* | *Details* | | :- | :- | :- | :- | :- | | `16:01:10` on
`2023-05-23` | 🟠 `Warning` | `FailedGetResourceMetric` | `horizontalpodautoscaler`/
`coredns` | `failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)` |
gardener-robot commented 1 year ago

@neo-liang-sap Command "/diag" failed with "KeyError('support.gardener.cloud/eu-access-for-cluster-nodes')".

Additional Information ``` Redacted in public. Check backend logs. ```
gardener-robot commented 1 year ago

@neo-liang-sap

Shoot: hubtest/ikdevtest v1.22.17             created at 2020-05-20 06:20 by Ingo Kober (BTP FP Gardener (SE))             on gcp in europe-west1 with purpose development             at 1 nodes and 21 pods and 36 API server requests/second (max of last 24h each) Seed: garden/gcp

🔴 Last Operation ``` description: "Flow \"Shoot cluster reconciliation\" encountered task errors: [task\ \ \"Waiting until shoot infrastructure has been reconciled\" failed: Error while\ \ waiting for Infrastructure shoot--hubtest--ikdevtest/ikdevtest to become ready:\ \ error during reconciliation: Error reconciling infrastructure: failed to apply\ \ the terraform config: Terraform execution for command 'apply' could not be completed:\n\ \n* Error when reading or editing ComputeNetwork \"projects/sap-gcp-k8s-wm-dev/global/networks/shoot--hubtest--ikdevtest\"\ : Get \"https://compute.googleapis.com/compute/v1/projects/sap-gcp-k8s-wm-dev/global/networks/shoot--hubtest--ikdevtest?alt=json\"\ : oauth2: cannot fetch token: 400 Bad Request\nResponse: {\"error\":\"invalid_grant\"\ ,\"error_description\":\"Invalid JWT Signature.\"}\n with google_compute_network.network,\n\ \ on main.tf line 20, in resource \"google_compute_network\" \"network\":\n 20:\ \ resource \"google_compute_network\" \"network\" {\n* Error when reading or editing\ \ Service Account \"projects/sap-gcp-k8s-wm-dev/serviceAccounts/shoot--hubtest--ikdevtest@sap-gcp-k8s-wm-dev.iam.gserviceaccount.com\"\ : Get \"https://iam.googleapis.com/v1/projects/sap-gcp-k8s-wm-dev/serviceAccounts/shoot--hubtest--ikdevtest@sap-gcp-k8s-wm-dev.iam.gserviceaccount.com?alt=json&prettyPrint=false\"\ : oauth2: cannot fetch token: 400 Bad Request\nResponse: {\"error\":\"invalid_grant\"\ ,\"error_description\":\"Invalid JWT Signature.\"}\n with google_service_account.serviceaccount,\n\ \ on main.tf line 11, in resource \"google_service_account\" \"serviceaccount\"\ :\n 11: resource \"google_service_account\" \"serviceaccount\" {]" lastUpdateTime: '2023-05-23T20:44:39Z' progress: 23 state: Failed type: Reconcile ```           **Last Error** ``` codes: - ERR_INFRA_UNAUTHENTICATED - ERR_INFRA_UNAUTHORIZED description: "task \"Waiting until shoot infrastructure has been reconciled\" failed:\ \ Error while waiting for Infrastructure shoot--hubtest--ikdevtest/ikdevtest to\ \ become ready: error during reconciliation: Error reconciling infrastructure: failed\ \ to apply the terraform config: Terraform execution for command 'apply' could not\ \ be completed:\n\n* Error when reading or editing ComputeNetwork \"projects/sap-gcp-k8s-wm-dev/global/networks/shoot--hubtest--ikdevtest\"\ : Get \"https://compute.googleapis.com/compute/v1/projects/sap-gcp-k8s-wm-dev/global/networks/shoot--hubtest--ikdevtest?alt=json\"\ : oauth2: cannot fetch token: 400 Bad Request\nResponse: {\"error\":\"invalid_grant\"\ ,\"error_description\":\"Invalid JWT Signature.\"}\n with google_compute_network.network,\n\ \ on main.tf line 20, in resource \"google_compute_network\" \"network\":\n 20:\ \ resource \"google_compute_network\" \"network\" {\n* Error when reading or editing\ \ Service Account \"projects/sap-gcp-k8s-wm-dev/serviceAccounts/shoot--hubtest--ikdevtest@sap-gcp-k8s-wm-dev.iam.gserviceaccount.com\"\ : Get \"https://iam.googleapis.com/v1/projects/sap-gcp-k8s-wm-dev/serviceAccounts/shoot--hubtest--ikdevtest@sap-gcp-k8s-wm-dev.iam.gserviceaccount.com?alt=json&prettyPrint=false\"\ : oauth2: cannot fetch token: 400 Bad Request\nResponse: {\"error\":\"invalid_grant\"\ ,\"error_description\":\"Invalid JWT Signature.\"}\n with google_service_account.serviceaccount,\n\ \ on main.tf line 11, in resource \"google_service_account\" \"serviceaccount\"\ :\n 11: resource \"google_service_account\" \"serviceaccount\" {" lastUpdateTime: '2023-05-23T20:44:39Z' taskID: Waiting until shoot infrastructure has been reconciled ```
đŸŸĸ Shoot Conditions           đŸ’¤ APIServerAvailable (ConditionNotChecked)           đŸ’¤ ControlPlaneHealthy (ConditionNotChecked)           đŸ’¤ ObservabilityComponentsHealthy (ConditionNotChecked)           đŸ’¤ SystemComponentsHealthy (ConditionNotChecked)           đŸ’¤ EveryNodeReady (ConditionNotChecked)           đŸ’¤ HibernationPossible (ConstraintNotChecked)           đŸ’¤ MaintenancePreconditionsSatisfied (ConstraintNotChecked)           đŸ’¤ CACertificateValiditiesAcceptable (ConstraintNotChecked)
đŸŸĸ Seed Conditions           đŸŸĸ GardenletReady (GardenletReady)           đŸŸĸ Bootstrapped (BootstrappingSucceeded)           đŸŸĸ ExtensionsReady (AllExtensionsReady)           đŸŸĸ AuditlogServiceAvailability (AuditlogInstanceAttached)           đŸŸĸ BackupBucketsReady (BackupBucketsAvailable)           đŸŸĸ SeedSystemComponentsHealthy (SystemComponentsRunning)
🔴 Control Plane Deployments Not Healthy | *Name* | *Desired* | *Current* | *Ready* | *Up-To-Date* | *Available* | | :- | -: | -: | -: | -: | -: | | **`cloud-controller`**`-manager` | đŸŸĸ `1` | đŸŸĸ `1` | 🔴 `0` | đŸŸĸ `1` | 🔴 `0` | | **`csi-driver`**`-controller` | đŸŸĸ `1` | đŸŸĸ `1` | 🔴 `0` | đŸŸĸ `1` | 🔴 `0` |
🔴 Control Plane Pods Not Healthy | *Name* | *Status* | *Age* | | :- | :- | :- | | **`cloud-controller-manager-65954b6b9b-fhqc8`**
`10.243.128.190` | Running
🔴 `0`/`1` Ready | 9 hours, 44 mins
🔴 `187` Restarts (`Error`) | | **`csi-driver-controller-57d48568c9-xlhtt`**
`10.243.140.28` | Running
🔴 `5`/`6` Ready | 9 hours, 44 mins
🔴 `194` Restarts (`Error`) |
🟠 Control Plane Events Not Healthy | *Timestamp* | *Severity* | *Reason* | *Object* | *Details* | | :- | :- | :- | :- | :- | | `14:16:15` on
`2023-05-23` | 🟠 `Warning` | `BackOff` | `pod`/
`csi-driver-controller-57d48568c9-xlhtt` | `Back-off restarting failed container` | | `14:16:23` on
`2023-05-23` | 🟠 `Warning` | `BackOff` | `pod`/
`cloud-controller-manager-65954b6b9b-fhqc8` | `Back-off restarting failed container` | | `20:01:21` on
`2023-05-23` | 🟠 `Warning` | `Unhealthy` | `pod`/
`cloud-controller-manager-65954b6b9b-fhqc8` | `Liveness probe failed: Get "https://10.243.128.190:10258/healthz": net/http: TLS handshake timeout` | | `20:56:19` on
`2023-05-23` | 🟠 `Warning` | `Unhealthy` | `pod`/
`csi-driver-controller-57d48568c9-xlhtt` | `Liveness probe failed: Get "http://10.243.140.28:9808/healthz": dial tcp 10.243.140.28:9808: connect: connection refused` |
đŸŸĸ Worker Pools | *Name* | *OS* | *Machine* | *Zones* | | :- | :- | :- | :- | | **`worker-fkdbj`**
`1:2` `-0` `+1` | `gardenlinux`
`v934.7.0` | `n1-standard-2`
`20Gi` `pd-standard` | `europe-west1-c`
  |
🟠 Worker Events Not Healthy | *Timestamp* | *Severity* | *Reason* | *Object* | *Details* | | :- | :- | :- | :- | :- | | `16:01:10` on
`2023-05-23` | 🟠 `Warning` | `FailedGetResourceMetric` | `horizontalpodautoscaler`/
`coredns` | `failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)` |
gardener-robot commented 1 year ago

@neo-liang-sap Command "/diag" failed with "KeyError('support.gardener.cloud/eu-access-for-cluster-nodes')".

Additional Information ``` Redacted in public. Check backend logs. ```
gardener-robot commented 1 year ago

@gardener-robot You have mentioned internal references in the public. Please check.

gardener-robot commented 1 year ago

@gardener-robot You have mentioned internal references in the public. Please check.

neo-liang-sap commented 1 year ago

/diag

gardener-robot commented 1 year ago

@neo-liang-sap

Shoot: hubtest/ikdevtest v1.22.17             created at 2020-05-20 06:20 by Ingo Kober (BTP FP Gardener (SE))             on gcp in europe-west1 with purpose development             at 1 nodes and 21 pods and 36 API server requests/second (max of last 24h each) Seed: garden/gcp

🔴 Last Operation ``` description: "Flow \"Shoot cluster reconciliation\" encountered task errors: [task\ \ \"Waiting until shoot infrastructure has been reconciled\" failed: Error while\ \ waiting for Infrastructure shoot--hubtest--ikdevtest/ikdevtest to become ready:\ \ error during reconciliation: Error reconciling infrastructure: failed to apply\ \ the terraform config: Terraform execution for command 'apply' could not be completed:\n\ \n* Error when reading or editing ComputeNetwork \"projects/sap-gcp-k8s-wm-dev/global/networks/shoot--hubtest--ikdevtest\"\ : Get \"https://compute.googleapis.com/compute/v1/projects/sap-gcp-k8s-wm-dev/global/networks/shoot--hubtest--ikdevtest?alt=json\"\ : oauth2: cannot fetch token: 400 Bad Request\nResponse: {\"error\":\"invalid_grant\"\ ,\"error_description\":\"Invalid JWT Signature.\"}\n with google_compute_network.network,\n\ \ on main.tf line 20, in resource \"google_compute_network\" \"network\":\n 20:\ \ resource \"google_compute_network\" \"network\" {\n* Error when reading or editing\ \ Service Account \"projects/sap-gcp-k8s-wm-dev/serviceAccounts/shoot--hubtest--ikdevtest@sap-gcp-k8s-wm-dev.iam.gserviceaccount.com\"\ : Get \"https://iam.googleapis.com/v1/projects/sap-gcp-k8s-wm-dev/serviceAccounts/shoot--hubtest--ikdevtest@sap-gcp-k8s-wm-dev.iam.gserviceaccount.com?alt=json&prettyPrint=false\"\ : oauth2: cannot fetch token: 400 Bad Request\nResponse: {\"error\":\"invalid_grant\"\ ,\"error_description\":\"Invalid JWT Signature.\"}\n with google_service_account.serviceaccount,\n\ \ on main.tf line 11, in resource \"google_service_account\" \"serviceaccount\"\ :\n 11: resource \"google_service_account\" \"serviceaccount\" {]" lastUpdateTime: '2023-05-23T20:44:39Z' progress: 23 state: Failed type: Reconcile ```           **Last Error** ``` codes: - ERR_INFRA_UNAUTHENTICATED - ERR_INFRA_UNAUTHORIZED description: "task \"Waiting until shoot infrastructure has been reconciled\" failed:\ \ Error while waiting for Infrastructure shoot--hubtest--ikdevtest/ikdevtest to\ \ become ready: error during reconciliation: Error reconciling infrastructure: failed\ \ to apply the terraform config: Terraform execution for command 'apply' could not\ \ be completed:\n\n* Error when reading or editing ComputeNetwork \"projects/sap-gcp-k8s-wm-dev/global/networks/shoot--hubtest--ikdevtest\"\ : Get \"https://compute.googleapis.com/compute/v1/projects/sap-gcp-k8s-wm-dev/global/networks/shoot--hubtest--ikdevtest?alt=json\"\ : oauth2: cannot fetch token: 400 Bad Request\nResponse: {\"error\":\"invalid_grant\"\ ,\"error_description\":\"Invalid JWT Signature.\"}\n with google_compute_network.network,\n\ \ on main.tf line 20, in resource \"google_compute_network\" \"network\":\n 20:\ \ resource \"google_compute_network\" \"network\" {\n* Error when reading or editing\ \ Service Account \"projects/sap-gcp-k8s-wm-dev/serviceAccounts/shoot--hubtest--ikdevtest@sap-gcp-k8s-wm-dev.iam.gserviceaccount.com\"\ : Get \"https://iam.googleapis.com/v1/projects/sap-gcp-k8s-wm-dev/serviceAccounts/shoot--hubtest--ikdevtest@sap-gcp-k8s-wm-dev.iam.gserviceaccount.com?alt=json&prettyPrint=false\"\ : oauth2: cannot fetch token: 400 Bad Request\nResponse: {\"error\":\"invalid_grant\"\ ,\"error_description\":\"Invalid JWT Signature.\"}\n with google_service_account.serviceaccount,\n\ \ on main.tf line 11, in resource \"google_service_account\" \"serviceaccount\"\ :\n 11: resource \"google_service_account\" \"serviceaccount\" {" lastUpdateTime: '2023-05-23T20:44:39Z' taskID: Waiting until shoot infrastructure has been reconciled ```
đŸŸĸ Shoot Conditions           đŸ’¤ APIServerAvailable (ConditionNotChecked)           đŸ’¤ ControlPlaneHealthy (ConditionNotChecked)           đŸ’¤ ObservabilityComponentsHealthy (ConditionNotChecked)           đŸ’¤ SystemComponentsHealthy (ConditionNotChecked)           đŸ’¤ EveryNodeReady (ConditionNotChecked)           đŸ’¤ HibernationPossible (ConstraintNotChecked)           đŸ’¤ MaintenancePreconditionsSatisfied (ConstraintNotChecked)           đŸ’¤ CACertificateValiditiesAcceptable (ConstraintNotChecked)
đŸŸĸ Seed Conditions           đŸŸĸ GardenletReady (GardenletReady)           đŸŸĸ Bootstrapped (BootstrappingSucceeded)           đŸŸĸ ExtensionsReady (AllExtensionsReady)           đŸŸĸ AuditlogServiceAvailability (AuditlogInstanceAttached)           đŸŸĸ BackupBucketsReady (BackupBucketsAvailable)           đŸŸĸ SeedSystemComponentsHealthy (SystemComponentsRunning)
🔴 Control Plane Deployments Not Healthy | *Name* | *Desired* | *Current* | *Ready* | *Up-To-Date* | *Available* | | :- | -: | -: | -: | -: | -: | | **`cloud-controller`**`-manager` | đŸŸĸ `1` | đŸŸĸ `1` | 🔴 `0` | đŸŸĸ `1` | 🔴 `0` | | **`csi-driver`**`-controller` | đŸŸĸ `1` | đŸŸĸ `1` | 🔴 `0` | đŸŸĸ `1` | 🔴 `0` |
🔴 Control Plane Pods Not Healthy | *Name* | *Status* | *Age* | | :- | :- | :- | | **`cloud-controller-manager-65954b6b9b-fhqc8`**
`10.243.128.190` | Running
🔴 `0`/`1` Ready | 9 hours, 46 mins
🔴 `187` Restarts (`Error`) | | **`csi-driver-controller-57d48568c9-xlhtt`**
`10.243.140.28` | Running
🔴 `5`/`6` Ready | 9 hours, 46 mins
🔴 `196` Restarts (`Error`) |
🟠 Control Plane Events Not Healthy | *Timestamp* | *Severity* | *Reason* | *Object* | *Details* | | :- | :- | :- | :- | :- | | `14:16:15` on
`2023-05-23` | 🟠 `Warning` | `BackOff` | `pod`/
`csi-driver-controller-57d48568c9-xlhtt` | `Back-off restarting failed container` | | `14:16:23` on
`2023-05-23` | 🟠 `Warning` | `BackOff` | `pod`/
`cloud-controller-manager-65954b6b9b-fhqc8` | `Back-off restarting failed container` | | `20:01:21` on
`2023-05-23` | 🟠 `Warning` | `Unhealthy` | `pod`/
`cloud-controller-manager-65954b6b9b-fhqc8` | `Liveness probe failed: Get "https://10.243.128.190:10258/healthz": net/http: TLS handshake timeout` | | `20:56:19` on
`2023-05-23` | 🟠 `Warning` | `Unhealthy` | `pod`/
`csi-driver-controller-57d48568c9-xlhtt` | `Liveness probe failed: Get "http://10.243.140.28:9808/healthz": dial tcp 10.243.140.28:9808: connect: connection refused` |
đŸŸĸ Worker Pools | *Name* | *OS* | *Machine* | *Zones* | | :- | :- | :- | :- | | **`worker-fkdbj`**
`1:2` `-0` `+1` | `gardenlinux`
`v934.7.0` | `n1-standard-2`
`20Gi` `pd-standard` | `europe-west1-c`
  |
🟠 Worker Events Not Healthy | *Timestamp* | *Severity* | *Reason* | *Object* | *Details* | | :- | :- | :- | :- | :- | | `16:01:10` on
`2023-05-23` | 🟠 `Warning` | `FailedGetResourceMetric` | `horizontalpodautoscaler`/
`coredns` | `failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)` |
gardener-robot commented 1 year ago

@gardener-robot You have mentioned internal references in the public. Please check.