Open hariapollo opened 6 months ago
qq: is your HPA still working?
This error seems transient E0321 07:24:07.740818
I'm getting the same issue, in version v0.15.0-gke.0
, cluster version: 1.29.1-gke.1589017
.
The metric isn't registering in the metric-server. so hpa is not working.
@KDanisme could you give some detailed steps on reproducing this issue?
On GKE 1.30.3-gke.1969001 and also noticing. I can't say how to reproduce the x509 error state. On a fresh cluster you don't see these problems. Manifests from main https://raw.githubusercontent.com/GoogleCloudPlatform/k8s-stackdriver/04cb65cba36989d69d39635eb490b1790c7f165f/custom-metrics-stackdriver-adapter/deploy/production/adapter_new_resource_model.yaml
% k logs -n custom-metrics custom-metrics-stackdriver-adapter-89fdf8645-hz48x
I0925 20:21:59.649957 1 adapter.go:217] serverOptions: {true true true true false false false}
I0925 20:21:59.650046 1 adapter.go:227] ListFullCustomMetrics is disabled, which would only list 1 metric resource to reduce memory usage. Add --list-full-custom-metrics to list full metric resources for debugging.
I0925 20:22:00.759069 1 serving.go:374] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I0925 20:22:03.547600 1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0925 20:22:03.547631 1 shared_informer.go:311] Waiting for caches to sync for RequestHeaderAuthRequestController
I0925 20:22:03.547679 1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0925 20:22:03.547681 1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I0925 20:22:03.547727 1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0925 20:22:03.547734 1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0925 20:22:03.547943 1 secure_serving.go:213] Serving securely on [::]:443
I0925 20:22:03.547984 1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0925 20:22:03.547996 1 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::apiserver.local.config/certificates/apiserver.crt::apiserver.local.config/certificates/apiserver.key"
I0925 20:22:03.736329 1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0925 20:22:03.736742 1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
E0925 20:22:03.737045 1 authentication.go:73] "Unable to authenticate the request" err="verifying certificate SN=59641085036180362641382743570345175134, SKID=, AKID=4F:F7:2B:CD:0C:83:5B:0B:AB:E7:12:86:62:05:B3:38:BC:05:A8:64 failed: x509: certificate signed by unknown authority"
E0925 20:22:03.737103 1 authentication.go:73] "Unable to authenticate the request" err="verifying certificate SN=299510449008022657087393158120274504989, SKID=, AKID=4F:F7:2B:CD:0C:83:5B:0B:AB:E7:12:86:62:05:B3:38:BC:05:A8:64 failed: x509: certificate signed by unknown authority"
E0925 20:22:03.737290 1 authentication.go:73] "Unable to authenticate the request" err="verifying certificate SN=299510449008022657087393158120274504989, SKID=, AKID=4F:F7:2B:CD:0C:83:5B:0B:AB:E7:12:86:62:05:B3:38:BC:05:A8:64 failed: x509: certificate signed by unknown authority"
E0925 20:22:03.737430 1 authentication.go:73] "Unable to authenticate the request" err="verifying certificate SN=299510449008022657087393158120274504989, SKID=, AKID=4F:F7:2B:CD:0C:83:5B:0B:AB:E7:12:86:62:05:B3:38:BC:05:A8:64 failed: x509: certificate signed by unknown authority"
E0925 20:22:03.737567 1 authentication.go:73] "Unable to authenticate the request" err="verifying certificate SN=299510449008022657087393158120274504989, SKID=, AKID=4F:F7:2B:CD:0C:83:5B:0B:AB:E7:12:86:62:05:B3:38:BC:05:A8:64 failed: x509: certificate signed by unknown authority"
E0925 20:22:03.737680 1 authentication.go:73] "Unable to authenticate the request" err="verifying certificate SN=299510449008022657087393158120274504989, SKID=, AKID=4F:F7:2B:CD:0C:83:5B:0B:AB:E7:12:86:62:05:B3:38:BC:05:A8:64 failed: x509: certificate signed by unknown authority"
E0925 20:22:03.737768 1 authentication.go:73] "Unable to authenticate the request" err="verifying certificate SN=299510449008022657087393158120274504989, SKID=, AKID=4F:F7:2B:CD:0C:83:5B:0B:AB:E7:12:86:62:05:B3:38:BC:05:A8:64 failed: x509: certificate signed by unknown authority"
E0925 20:22:03.737881 1 authentication.go:73] "Unable to authenticate the request" err="verifying certificate SN=299510449008022657087393158120274504989, SKID=, AKID=4F:F7:2B:CD:0C:83:5B:0B:AB:E7:12:86:62:05:B3:38:BC:05:A8:64 failed: x509: certificate signed by unknown authority"
E0925 20:22:03.737970 1 authentication.go:73] "Unable to authenticate the request" err="verifying certificate SN=299510449008022657087393158120274504989, SKID=, AKID=4F:F7:2B:CD:0C:83:5B:0B:AB:E7:12:86:62:05:B3:38:BC:05:A8:64 failed: x509: certificate signed by unknown authority"
E0925 20:22:03.738083 1 authentication.go:73] "Unable to authenticate the request" err="verifying certificate SN=299510449008022657087393158120274504989, SKID=, AKID=4F:F7:2B:CD:0C:83:5B:0B:AB:E7:12:86:62:05:B3:38:BC:05:A8:64 failed: x509: certificate signed by unknown authority"
E0925 20:22:03.836324 1 authentication.go:73] "Unable to authenticate the request" err="verifying certificate SN=299510449008022657087393158120274504989, SKID=, AKID=4F:F7:2B:CD:0C:83:5B:0B:AB:E7:12:86:62:05:B3:38:BC:05:A8:64 failed: x509: certificate signed by unknown authority"
E0925 20:22:03.836415 1 authentication.go:73] "Unable to authenticate the request" err="verifying certificate SN=299510449008022657087393158120274504989, SKID=, AKID=4F:F7:2B:CD:0C:83:5B:0B:AB:E7:12:86:62:05:B3:38:BC:05:A8:64 failed: x509: certificate signed by unknown authority"
E0925 20:22:03.836570 1 authentication.go:73] "Unable to authenticate the request" err="verifying certificate SN=299510449008022657087393158120274504989, SKID=, AKID=4F:F7:2B:CD:0C:83:5B:0B:AB:E7:12:86:62:05:B3:38:BC:05:A8:64 failed: x509: certificate signed by unknown authority"
E0925 20:22:03.836595 1 authentication.go:73] "Unable to authenticate the request" err="verifying certificate SN=299510449008022657087393158120274504989, SKID=, AKID=4F:F7:2B:CD:0C:83:5B:0B:AB:E7:12:86:62:05:B3:38:BC:05:A8:64 failed: x509: certificate signed by unknown authority"
E0925 20:22:03.836811 1 authentication.go:73] "Unable to authenticate the request" err="verifying certificate SN=59641085036180362641382743570345175134, SKID=, AKID=4F:F7:2B:CD:0C:83:5B:0B:AB:E7:12:86:62:05:B3:38:BC:05:A8:64 failed: x509: certificate signed by unknown authority"
E0925 20:22:03.836869 1 authentication.go:73] "Unable to authenticate the request" err="verifying certificate SN=299510449008022657087393158120274504989, SKID=, AKID=4F:F7:2B:CD:0C:83:5B:0B:AB:E7:12:86:62:05:B3:38:BC:05:A8:64 failed: x509: certificate signed by unknown authority"
I0925 20:22:03.937710 1 shared_informer.go:318] Caches are synced for RequestHeaderAuthRequestController
E0925 20:22:03.937868 1 authentication.go:73] "Unable to authenticate the request" err="verifying certificate SN=89182377387980615165665092415368016443, SKID=, AKID=4F:F7:2B:CD:0C:83:5B:0B:AB:E7:12:86:62:05:B3:38:BC:05:A8:64 failed: x509: certificate signed by unknown authority"
E0925 20:22:04.036832 1 authentication.go:73] "Unable to authenticate the request" err="verifying certificate SN=89182377387980615165665092415368016443, SKID=, AKID=4F:F7:2B:CD:0C:83:5B:0B:AB:E7:12:86:62:05:B3:38:BC:05:A8:64 failed: x509: certificate signed by unknown authority"
E0925 20:22:04.136300 1 authentication.go:73] "Unable to authenticate the request" err="verifying certificate SN=89182377387980615165665092415368016443, SKID=, AKID=4F:F7:2B:CD:0C:83:5B:0B:AB:E7:12:86:62:05:B3:38:BC:05:A8:64 failed: x509: certificate signed by unknown authority"
On GKE 1.30.3-gke.1969001 and also noticing
Does a restart fix this issue?
As suggested here, we have tried to upgrade stack driver to newer version though we got the certificate authenticate issue. GKE Version:
1.26.11-gke.1055000
Custom stack driver version:v0.14.2-gke.0
After upgrading we got the below error.