search

OpenFunction / OpenFunction

Cloud Native Function-as-a-Service Platform (CNCF Sandbox Project)
https://openfunction.dev
Apache License 2.0
1.48k stars 160 forks source link

activator can't connect to autoscaler #478

Open mensylisir opened 10 months ago

mensylisir commented 10 months ago

Description A clear and concise description of what the bug is.

I install openfunction by helm package

helm install openfunction openfunction/openfunction -n openfunction --set revisionController.enable=true

Then the activator can't be ready

dapr-system        dapr-dashboard-797b56749f-c24c5                            1/1     Running   1 (5h14m ago)    23h
dapr-system        dapr-operator-7df6df9787-bxc6s                             1/1     Running   7 (5h14m ago)    23h
dapr-system        dapr-placement-server-0                                    1/1     Running   17 (5h14m ago)   23h
dapr-system        dapr-sentry-6fd7776f6-h7qx6                                1/1     Running   1 (5h14m ago)    23h
dapr-system        dapr-sidecar-injector-59d9fb8755-5kdgs                     1/1     Running   1 (5h14m ago)    23h
keda               keda-operator-6d55cf7dd8-zdj9j                             1/1     Running   1 (5h14m ago)    23h
keda               keda-operator-metrics-apiserver-685bc8f78f-t5p4d           1/1     Running   1 (5h14m ago)    23h
knative-serving    activator-5fbdcfb5fd-6lpzc                                 0/1     Running   6 (55s ago)      18m
knative-serving    autoscaler-86bc55dc7d-wk2qg                                1/1     Running   0                19m
knative-serving    controller-64cc8b4f75-b5pcv                                1/1     Running   1 (5h14m ago)    23h
knative-serving    domain-mapping-5d49fd9ff7-sf9jc                            1/1     Running   1 (5h14m ago)    23h
knative-serving    domainmapping-webhook-f684c795d-6fwc5                      1/1     Running   1 (5h14m ago)    23h
knative-serving    net-contour-controller-857cf8c796-wwr9h                    1/1     Running   1 (5h14m ago)    23h
knative-serving    webhook-5876cb94cc-qd6c7                                   1/1     Running   1 (5h14m ago)    23h
openfunction       openfunction-controller-manager-755d89c654-8hzz2           2/2     Running   2 (5h14m ago)    23h
openfunction       openfunction-revision-controller-854fb97cd5-8rzjb          1/1     Running   1 (5h14m ago)    23h
projectcontour     contour-contour-8454bd9f6-rkkwv                            1/1     Running   1 (5h14m ago)    23h
projectcontour     contour-envoy-zznlc                                        2/2     Running   2 (5h14m ago)    23h
shipwright-build   shipwright-build-controller-5954dff84f-dxd6h               1/1     Running   1 (5h14m ago)    23h
tekton-pipelines   tekton-pipelines-controller-5848858869-qrg66               1/1     Running   1 (5h14m ago)    23h
tekton-pipelines   tekton-pipelines-webhook-665545b554-qrlg9                  1/1     Running   1 (5h14m ago)    23h
{"severity":"ERROR","timestamp":"2023-08-10T06:06:17.144801644Z","logger":"activator","caller":"websocket/connection.go:192","message":"Failed to send ping message to ws://autoscaler.knative-serving.svc.cluster.local:8080","commit":"ac29233","knative.dev/controller":"activator","knative.dev/pod":"activator-5fbdcfb5fd-6lpzc","error":"connection has not yet been established","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func3\n\tknative.dev/pkg@v0.0.0-20220301181942-2fdd5f232e77/websocket/connection.go:192"}
{"severity":"ERROR","timestamp":"2023-08-10T06:06:18.488498248Z","logger":"activator","caller":"websocket/connection.go:145","message":"Websocket connection could not be established","commit":"ac29233","knative.dev/controller":"activator","knative.dev/pod":"activator-5fbdcfb5fd-6lpzc","error":"dial tcp: i/o timeout","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func1\n\tknative.dev/pkg@v0.0.0-20220301181942-2fdd5f232e77/websocket/connection.go:145\nknative.dev/pkg/websocket.(*ManagedConnection).connect.func1\n\tknative.dev/pkg@v0.0.0-20220301181942-2fdd5f232e77/websocket/connection.go:226\nk8s.io/apimachinery/pkg/util/wait.ConditionFunc.WithContext.func1\n\tk8s.io/apimachinery@v0.23.4/pkg/util/wait/wait.go:217\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext\n\tk8s.io/apimachinery@v0.23.4/pkg/util/wait/wait.go:230\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection\n\tk8s.io/apimachinery@v0.23.4/pkg/util/wait/wait.go:223\nk8s.io/apimachinery/pkg/util/wait.ExponentialBackoff\n\tk8s.io/apimachinery@v0.23.4/pkg/util/wait/wait.go:418\nknative.dev/pkg/websocket.(*ManagedConnection).connect\n\tknative.dev/pkg@v0.0.0-20220301181942-2fdd5f232e77/websocket/connection.go:223\nknative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20220301181942-2fdd5f232e77/websocket/connection.go:163"}
{"severity":"ERROR","timestamp":"2023-08-10T06:06:18.488645875Z","logger":"activator","caller":"websocket/connection.go:164","message":"Failed connecting to ws://autoscaler.knative-serving.svc.cluster.local:8080","commit":"ac29233","knative.dev/controller":"activator","knative.dev/pod":"activator-5fbdcfb5fd-6lpzc","error":"timed out waiting for the condition","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20220301181942-2fdd5f232e77/websocket/connection.go:164"}
{"severity":"INFO","timestamp":"2023-08-10T06:06:18.488675842Z","logger":"activator","caller":"websocket/connection.go:162","message":"Connecting to ws://autoscaler.knative-serving.svc.cluster.local:8080","commit":"ac29233","knative.dev/controller":"activator","knative.dev/pod":"activator-5fbdcfb5fd-6lpzc"}
{"severity":"WARNING","timestamp":"2023-08-10T06:06:19.357531373Z","logger":"activator","caller":"handler/healthz_handler.go:36","message":"Healthcheck failed: connection has not yet been established","commit":"ac29233","knative.dev/controller":"activator","knative.dev/pod":"activator-5fbdcfb5fd-6lpzc"}
{"severity":"ERROR","timestamp":"2023-08-10T06:06:20.477660307Z","logger":"activator","caller":"websocket/connection.go:192","message":"Failed to send ping message to ws://autoscaler.knative-serving.svc.cluster.local:8080","commit":"ac29233","knative.dev/controller":"activator","knative.dev/pod":"activator-5fbdcfb5fd-6lpzc","error":"connection has not yet been established","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func3\n\tknative.

Environmental Tell us what your cluster environment is like, e.g.

Expected behavior Tell us what should happen.

Actual behavior Tell us what happens instead. Provide a log message if relevant.

To Reproduce Steps to reproduce the behavior:

  1. '..'
  2. '...'
  3. '....'
  4. See error

Screenshots If applicable, add screenshots to help explain your problem.

Additional context Add any other context about the problem here.

mensylisir commented 10 months ago

autoscaler logs

2023/08/10 08:43:06 Registering 4 clients
2023/08/10 08:43:06 Registering 5 informer factories
2023/08/10 08:43:06 Registering 5 informers
2023/08/10 08:43:06 Registering 1 filtered informers
2023/08/10 08:43:06 Registering 2 controllers
{"severity":"INFO","timestamp":"2023-08-10T08:43:06.390358743Z","caller":"logging/config.go:116","message":"Successfully created the logger."}
{"severity":"INFO","timestamp":"2023-08-10T08:43:06.390387512Z","caller":"logging/config.go:117","message":"Logging level set to: info"}
{"severity":"INFO","timestamp":"2023-08-10T08:43:06.39047542Z","logger":"autoscaler","caller":"profiling/server.go:64","message":"Profiling enabled: false","commit":"ac29233"}
{"severity":"INFO","timestamp":"2023-08-10T08:43:06.391665896Z","logger":"autoscaler","caller":"kpa/controller.go:75","message":"Setting up ConfigMap receivers","commit":"ac29233"}
{"severity":"INFO","timestamp":"2023-08-10T08:43:06.391778447Z","logger":"autoscaler","caller":"kpa/controller.go:89","message":"Setting up KPA-Class event handlers","commit":"ac29233"}
{"severity":"INFO","timestamp":"2023-08-10T08:43:06.394667498Z","logger":"autoscaler","caller":"metrics/metrics_worker.go:76","message":"Flushing the existing exporter before setting up the new exporter.","commit":"ac29233"}
{"severity":"INFO","timestamp":"2023-08-10T08:43:06.394761745Z","logger":"autoscaler","caller":"metrics/prometheus_exporter.go:51","message":"Created Prometheus exporter with config: &{knative.dev/serving autoscaler prometheus 5000000000 false 9090 0.0.0.0}. Start the server for Prometheus exporter.","commit":"ac29233"}
{"severity":"INFO","timestamp":"2023-08-10T08:43:06.394770012Z","logger":"autoscaler","caller":"metrics/metrics_worker.go:91","message":"Successfully updated the metrics exporter; old config: ; new config &{knative.dev/serving autoscaler prometheus 5000000000 false 9090 0.0.0.0}","commit":"ac29233"}
{"severity":"INFO","timestamp":"2023-08-10T08:43:06.595885233Z","logger":"autoscaler","caller":"autoscaler/main.go:178","message":"Running with Standard leader election","commit":"ac29233"}
{"severity":"INFO","timestamp":"2023-08-10T08:43:06.596197349Z","logger":"autoscaler","caller":"leaderelection/context.go:143","message":"autoscaler-bucket-00-of-01 will run in leader-elected mode with id "autoscaler-86bc55dc7d-vqtmf_10.233.90.68"","commit":"ac29233"}
{"severity":"INFO","timestamp":"2023-08-10T08:43:06.596282852Z","logger":"autoscaler","caller":"controller/controller.go:479","message":"Starting controller and workers","commit":"ac29233","knative.dev/controller":"knative.dev.serving.pkg.reconciler.metric.reconciler","knative.dev/kind":"autoscaling.internal.knative.dev.Metric"}
{"severity":"INFO","timestamp":"2023-08-10T08:43:06.596296321Z","logger":"autoscaler","caller":"controller/controller.go:489","message":"Started workers","commit":"ac29233","knative.dev/controller":"knative.dev.serving.pkg.reconciler.metric.reconciler","knative.dev/kind":"autoscaling.internal.knative.dev.Metric"}
{"severity":"INFO","timestamp":"2023-08-10T08:43:06.59634485Z","logger":"autoscaler.stats-websocket-server","caller":"statserver/server.go:96","message":"Starting","commit":"ac29233","address":":8080"}
{"severity":"INFO","timestamp":"2023-08-10T08:43:06.596392521Z","logger":"autoscaler","caller":"leaderelection/context.go:143","message":"autoscaler-bucket-00-of-01 will run in leader-elected mode with id "autoscaler-86bc55dc7d-vqtmf_10.233.90.68"","commit":"ac29233"}
{"severity":"INFO","timestamp":"2023-08-10T08:43:06.596410327Z","logger":"autoscaler","caller":"controller/controller.go:479","message":"Starting controller and workers","commit":"ac29233","knative.dev/controller":"knative.dev.serving.pkg.reconciler.autoscaling.kpa.Reconciler","knative.dev/kind":"autoscaling.internal.knative.dev.PodAutoscaler"}
{"severity":"INFO","timestamp":"2023-08-10T08:43:06.596414191Z","logger":"autoscaler","caller":"controller/controller.go:489","message":"Started workers","commit":"ac29233","knative.dev/controller":"knative.dev.serving.pkg.reconciler.autoscaling.kpa.Reconciler","knative.dev/kind":"autoscaling.internal.knative.dev.PodAutoscaler"}
I0810 08:43:06.596458 1 leaderelection.go:248] attempting to acquire leader lease knative-serving/autoscaler-bucket-00-of-01...
I0810 08:43:06.596602 1 leaderelection.go:248] attempting to acquire leader lease knative-serving/autoscaler-bucket-00-of-01...
I0810 08:43:06.599735 1 leaderelection.go:258] successfully acquired lease knative-serving/autoscaler-bucket-00-of-01
E0810 08:43:06.599998 1 leaderelection.go:367] Failed to update lock: Operation cannot be fulfilled on leases.coordination.k8s.io "autoscaler-bucket-00-of-01": the object has been modified; please apply your changes to the latest version and try again
{"severity":"INFO","timestamp":"2023-08-10T08:43:06.60001642Z","logger":"autoscaler","caller":"leaderelection/context.go:152","message":""autoscaler-86bc55dc7d-vqtmf_10.233.90.68" has started leading "autoscaler-bucket-00-of-01"","commit":"ac29233"}
{"severity":"INFO","timestamp":"2023-08-10T08:43:06.600290765Z","logger":"autoscaler","caller":"kpa/kpa.go:148","message":"SKS should be in Proxy mode: want = -1, ebc = -100, #act's = 3 PA Inactive? = true","commit":"ac29233","knative.dev/controller":"knative.dev.serving.pkg.reconciler.autoscaling.kpa.Reconciler","knative.dev/kind":"autoscaling.internal.knative.dev.PodAutoscaler","knative.dev/traceid":"e63acec2-ff22-4474-807e-2cf6aebc16b4","knative.dev/key":"default/serving-cc7q7-ksvc-vcrjq-v200"}
{"severity":"INFO","timestamp":"2023-08-10T08:43:06.600314114Z","logger":"autoscaler","caller":"kpa/kpa.go:168","message":"PA scale got=1, want=-1, desiredPods=-1 ebc=-100","commit":"ac29233","knative.dev/controller":"knative.dev.serving.pkg.reconciler.autoscaling.kpa.Reconciler","knative.dev/kind":"autoscaling.internal.knative.dev.PodAutoscaler","knative.dev/traceid":"e63acec2-ff22-4474-807e-2cf6aebc16b4","knative.dev/key":"default/serving-cc7q7-ksvc-vcrjq-v200"}
{"severity":"INFO","timestamp":"2023-08-10T08:43:06.600321005Z","logger":"autoscaler","caller":"kpa/kpa.go:178","message":"Observed pod counts=kpa.podCounts{want:-1, ready:1, notReady:0, pending:0, terminating:0}","commit":"ac29233","knative.dev/controller":"knative.dev.serving.pkg.reconciler.autoscaling.kpa.Reconciler","knative.dev/kind":"autoscaling.internal.knative.dev.PodAutoscaler","knative.dev/traceid":"e63acec2-ff22-4474-807e-2cf6aebc16b4","knative.dev/key":"default/serving-cc7q7-ksvc-vcrjq-v200"}
{"severity":"INFO","timestamp":"2023-08-10T08:43:06.604439097Z","logger":"autoscaler","caller":"statforwarder/leases.go:194","message":"Created Service for Lease knative-serving/autoscaler-bucket-00-of-01","commit":"ac29233"}
{"severity":"INFO","timestamp":"2023-08-10T08:43:06.606467915Z","logger":"autoscaler","caller":"statforwarder/leases.go:274","message":"Bucket Endpoints autoscaler-bucket-00-of-01 updated with IP 10.233.90.68","commit":"ac29233"}
{"severity":"INFO","timestamp":"2023-08-10T08:43:06.606500084Z","logger":"autoscaler","caller":"statforwarder/leases.go:200","message":"Created Endpoints for Lease knative-serving/autoscaler-bucket-00-of-01","commit":"ac29233"}
{"severity":"INFO","timestamp":"2023-08-10T08:43:06.608884768Z","logger":"autoscaler","caller":"controller/controller.go:543","message":"Reconcile succeeded","commit":"ac29233","knative.dev/controller":"knative.dev.serving.pkg.reconciler.autoscaling.kpa.Reconciler","knative.dev/kind":"autoscaling.internal.knative.dev.PodAutoscaler","knative.dev/traceid":"e63acec2-ff22-4474-807e-2cf6aebc16b4","knative.dev/key":"default/serving-cc7q7-ksvc-vcrjq-v200","duration":"8.828637ms"}
{"severity":"INFO","timestamp":"2023-08-10T08:43:06.608994985Z","logger":"autoscaler","caller":"kpa/kpa.go:148","message":"SKS should be in Proxy mode: want = -1, ebc = -100, #act's = 3 PA Inactive? = true","commit":"ac29233","knative.dev/controller":"knative.dev.serving.pkg.reconciler.autoscaling.kpa.Reconciler","knative.dev/kind":"autoscaling.internal.knative.dev.PodAutoscaler","knative.dev/traceid":"b072cc9c-54e4-44ef-9a79-69959270f178","knative.dev/key":"default/serving-cc7q7-ksvc-vcrjq-v200"}
{"severity":"INFO","timestamp":"2023-08-10T08:43:06.609013385Z","logger":"autoscaler","caller":"kpa/kpa.go:168","message":"PA scale got=1, want=-1, desiredPods=-1 ebc=-100","commit":"ac29233","knative.dev/controller":"knative.dev.serving.pkg.reconciler.autoscaling.kpa.Reconciler","knative.dev/kind":"autoscaling.internal.knative.dev.PodAutoscaler","knative.dev/traceid":"b072cc9c-54e4-44ef-9a79-69959270f178","knative.dev/key":"default/serving-cc7q7-ksvc-vcrjq-v200"}
{"severity":"INFO","timestamp":"2023-08-10T08:43:06.609020153Z","logger":"autoscaler","caller":"kpa/kpa.go:178","message":"Observed pod counts=kpa.podCounts{want:-1, ready:1, notReady:0, pending:0, terminating:0}","commit":"ac29233","knative.dev/controller":"knative.dev.serving.pkg.reconciler.autoscaling.kpa.Reconciler","knative.dev/kind":"autoscaling.internal.knative.dev.PodAutoscaler","knative.dev/traceid":"b072cc9c-54e4-44ef-9a79-69959270f178","knative.dev/key":"default/serving-cc7q7-ksvc-vcrjq-v200"}
{"severity":"INFO","timestamp":"2023-08-10T08:43:06.609044838Z","logger":"autoscaler","caller":"controller/controller.go:543","message":"Reconcile succeeded","commit":"ac29233","knative.dev/controller":"knative.dev.serving.pkg.reconciler.autoscaling.kpa.Reconciler","knative.dev/kind":"autoscaling.internal.knative.dev.PodAutoscaler","knative.dev/traceid":"b072cc9c-54e4-44ef-9a79-69959270f178","knative.dev/key":"default/serving-cc7q7-ksvc-vcrjq-v200","duration":"113.263µs"}
{"severity":"ERROR","timestamp":"2023-08-10T08:43:08.601248947Z","logger":"autoscaler","caller":"scaling/autoscaler.go:170","message":"Failed to obtain metrics","commit":"ac29233","knative.dev/key":"default/serving-cc7q7-ksvc-vcrjq-v200","error":"no metrics are being collected for the requested resource","stacktrace":"knative.dev/serving/pkg/autoscaler/scaling.(*autoscaler).Scale\n\tknative.dev/serving/pkg/autoscaler/scaling/autoscaler.go:170\nknative.dev/serving/pkg/autoscaler/scaling.(*MultiScaler).tickScaler\n\tknative.dev/serving/pkg/autoscaler/scaling/multiscaler.go:329\nknative.dev/serving/pkg/autoscaler/scaling.(*MultiScaler).runScalerTicker.func1\n\tknative.dev/serving/pkg/autoscaler/scaling/multiscaler.go:292"}
{"severity":"ERROR","timestamp":"2023-08-10T08:43:10.600282178Z","logger":"autoscaler","caller":"scaling/autoscaler.go:170","message":"Failed to obtain metrics","commit":"ac29233","knative.dev/key":"default/serving-cc7q7-ksvc-vcrjq-v200","error":"no metrics are being collected for the requested resource","stacktrace":"knative.dev/serving/pkg/autoscaler/scaling.(*autoscaler).Scale\n\tknative.dev/serving/pkg/autoscaler/scaling/autoscaler.go:170\nknative.dev/serving/pkg/autoscaler/scaling.(*MultiScaler).tickScaler\n\tknative.dev/serving/pkg/autoscaler/scaling/multiscaler.go:329\nknative.dev/serving/pkg/autoscaler/scaling.(*MultiScaler).runScalerTicker.func1\n\tknative.dev/serving/pkg/autoscaler/scaling/multiscaler.go:292"}
{"severity":"ERROR","timestamp":"2023-08-10T08:43:12.600651252Z","logger":"autoscaler","caller":"scaling/autoscaler.go:170","message":"Failed to obtain metrics","commit":"ac29233","knative.dev/key":"default/serving-cc7q7-ksvc-vcrjq-v200","error":"no metrics are being collected for the requested resource","stacktrace":"knative.dev/serving/pkg/autoscaler/scaling.(*autoscaler).Scale\n\tknative.dev/serving/pkg/autoscaler/scaling/autoscaler.go:170\nknative.dev/serving/pkg/autoscaler/scaling.(*MultiScaler).tickScaler\n\tknative.dev/serving/pkg/autoscaler/scaling/multiscaler.go:329\nknative.dev/serving/pkg/autoscaler/scaling.(*MultiScaler).runScalerTicker.func1\n\tknative.dev/serving/pkg/autoscaler/scaling/multiscaler.go:292"}
{"severity":"ERROR","timestamp":"2023-08-10T08:43:14.601342371Z","logger":"autoscaler","caller":"scaling/autoscaler.go:170","message":"Failed to obtain metrics","commit":"ac29233","knative.dev/key":"default/serving-cc7q7-ksvc-vcrjq-v200","error":"no metrics are being collected for the requested resource","stacktrace":"knative.dev/serving/pkg/autoscaler/scaling.(*autoscaler).Scale\n\tknative.dev/serving/pkg/autoscaler/scaling/autoscaler.go:170\nknative.dev/serving/pkg/autoscaler/scaling.(*MultiScaler).tickScaler\n\tknative.dev/serving/pkg/autoscaler/scaling/multiscaler.go:329\nknative.dev/serving/pkg/autoscaler/scaling.(*MultiScaler).runScalerTicker.func1\n\tknative.dev/serving/pkg/autoscaler/scaling/multiscaler.go:292"}
{"severity":"ERROR","timestamp":"2023-08-10T08:43:16.600963556Z","logger":"autoscaler","caller":"scaling/autoscaler.go:170","message":"Failed to obtain metrics","commit":"ac29233","knative.dev/key":"default/serving-cc7q7-ksvc-vcrjq-v200","error":"no metrics are being collected for the requested resource","stacktrace":"knative.dev/serving/pkg/autoscaler/scaling.(*autoscaler).Scale\n\tknative.dev/serving/pkg/autoscaler/scaling/autoscaler.go:170\nknative.dev/serving/pkg/autoscaler/scaling.(*MultiScaler).tickScaler\n\tknative.dev/serving/pkg/autoscaler/scaling/multiscaler.go:329\nknative.dev/serving/pkg/autoscaler/scaling.(*MultiScaler).runScalerTicker.func1\n\tknative.dev/serving/pkg/autoscaler/scaling/multiscaler.go:292"}
I0810 08:43:17.768004 1 leaderelection.go:258] successfully acquired lease knative-serving/autoscaler-bucket-00-of-01
{"severity":"INFO","timestamp":"2023-08-10T08:43:17.768144024Z","logger":"autoscaler","caller":"leaderelection/context.go:152","message":""autoscaler-86bc55dc7d-vqtmf_10.233.90.68" has started leading "autoscaler-bucket-00-of-01"","commit":"ac29233"}
{"severity":"INFO","timestamp":"2023-08-10T08:43:17.768378415Z","logger":"autoscaler","caller":"controller/controller.go:543","message":"Reconcile succeeded","commit":"ac29233","knative.dev/controller":"knative.dev.serving.pkg.reconciler.metric.reconciler","knative.dev/kind":"autoscaling.internal.knative.dev.Metric","knative.dev/traceid":"fe973963-0327-48ee-91a0-87876265b0ce","knative.dev/key":"default/serving-cc7q7-ksvc-vcrjq-v200","duration":"132.704µs"}
{"severity":"INFO","timestamp":"2023-08-10T08:43:40.889563338Z","logger":"autoscaler","caller":"kpa/scaler.go:233","message":"Probing activator = false, err = error roundtripping [http://serving-cc7q7-ksvc-vcrjq-v200.default.svc.cluster.local:80//healthz](http://serving-cc7q7-ksvc-vcrjq-v200.default.svc.cluster.local//healthz): dial tcp 10.233.11.181:80: connect: connection refused","commit":"ac29233","knative.dev/controller":"knative.dev.serving.pkg.reconciler.autoscaling.kpa.Reconciler","knative.dev/kind":"autoscaling.internal.knative.dev.PodAutoscaler","knative.dev/traceid":"1a1da538-d017-4503-994f-6aae809cee1b","knative.dev/key":"default/serving-cc7q7-ksvc-vcrjq-v200"}
{"severity":"INFO","timestamp":"2023-08-10T08:43:40.88962591Z","logger":"autoscaler","caller":"kpa/scaler.go:280","message":"PA is not yet backed by activator, cannot scale to zero","commit":"ac29233","knative.dev/controller":"knative.dev.serving.pkg.reconciler.autoscaling.kpa.Reconciler","knative.dev/kind":"autoscaling.internal.knative.dev.PodAutoscaler","knative.dev/traceid":"1a1da538-d017-4503-994f-6aae809cee1b","knative.dev/key":"default/serving-cc7q7-ksvc-vcrjq-v200"}
{"severity":"INFO","timestamp":"2023-08-10T08:43:40.889674453Z","logger":"autoscaler","caller":"kpa/kpa.go:148","message":"SKS should be in Proxy mode: want = 0, ebc = -100, #act's = 3 PA Inactive? = true","commit":"ac29233","knative.dev/controller":"knative.dev.serving.pkg.reconciler.autoscaling.kpa.Reconciler","knative.dev/kind":"autoscaling.internal.knative.dev.PodAutoscaler","knative.dev/traceid":"1a1da538-d017-4503-994f-6aae809cee1b","knative.dev/key":"default/serving-cc7q7-ksvc-vcrjq-v200"}
{"severity":"INFO","timestamp":"2023-08-10T08:43:40.889732924Z","logger":"autoscaler","caller":"kpa/kpa.go:168","message":"PA scale got=1, want=0, desiredPods=0 ebc=-100","commit":"ac29233","knative.dev/controller":"knative.dev.serving.pkg.reconciler.autoscaling.kpa.Reconciler","knative.dev/kind":"autoscaling.internal.knative.dev.PodAutoscaler","knative.dev/traceid":"1a1da538-d017-4503-994f-6aae809cee1b","knative.dev/key":"default/serving-cc7q7-ksvc-vcrjq-v200"}
{"severity":"INFO","timestamp":"2023-08-10T08:43:40.889753348Z","logger":"autoscaler","caller":"kpa/kpa.go:178","message":"Observed pod counts=kpa.podCounts{want:0, ready:1, notReady:0, pending:0, terminating:0}","commit":"ac29233","knative.dev/controller":"knative.dev.serving.pkg.reconciler.autoscaling.kpa.Reconciler","knative.dev/kind":"autoscaling.internal.knative.dev.PodAutoscaler","knative.dev/traceid":"1a1da538-d017-4503-994f-6aae809cee1b","knative.dev/key":"default/serving-cc7q7-ksvc-vcrjq-v200"}
{"severity":"INFO","timestamp":"2023-08-10T08:43:40.89752313Z","logger":"autoscaler","caller":"controller/controller.go:543","message":"Reconcile succeeded","commit":"ac29233","knative.dev/controller":"knative.dev.serving.pkg.reconciler.autoscaling.kpa.Reconciler","knative.dev/kind":"autoscaling.internal.knative.dev.PodAutoscaler","knative.dev/traceid":"1a1da538-d017-4503-994f-6aae809cee1b","knative.dev/key":"default/serving-cc7q7-ksvc-vcrjq-v200","duration":"20.296107726s"}
{"severity":"INFO","timestamp":"2023-08-10T08:43:50.157691333Z","logger":"autoscaler","caller":"kpa/scaler.go:233","message":"Probing activator = false, err = error roundtripping [http://serving-cc7q7-ksvc-vcrjq-v200.default.svc.cluster.local:80//healthz](http://serving-cc7q7-ksvc-vcrjq-v200.default.svc.cluster.local//healthz): dial tcp 10.233.11.181:80: connect: connection refused","commit":"ac29233","knative.dev/controller":"knative.dev.serving.pkg.reconciler.autoscaling.kpa.Reconciler","knative.dev/kind":"autoscaling.internal.knative.dev.PodAutoscaler","knative.dev/traceid":"75095934-035b-4cb2-819d-a5e8ccf98693","knative.dev/key":"default/serving-cc7q7-ksvc-vcrjq-v200"}
{"severity":"INFO","timestamp":"2023-08-10T08:43:50.157753392Z","logger":"autoscaler","caller":"kpa/scaler.go:280","message":"PA is not yet backed by activator, cannot scale to zero","commit":"ac29233","knative.dev/controller":"knative.dev.serving.pkg.reconciler.autoscaling.kpa.Reconciler","knative.dev/kind":"autoscaling.internal.knative.dev.PodAutoscaler","knative.dev/traceid":"75095934-035b-4cb2-819d-a5e8ccf98693","knative.dev/key":"default/serving-cc7q7-ksvc-vcrjq-v200"}
{"severity":"INFO","timestamp":"2023-08-10T08:43:50.157771472Z","logger":"autoscaler","caller":"kpa/scaler.go:282","message":"Probe for revision is already in flight","commit":"ac29233","knative.dev/controller":"knative.dev.serving.pkg.reconciler.autoscaling.kpa.Reconciler","knative.dev/kind":"autoscaling.internal.knative.dev.PodAutoscaler","knative.dev/traceid":"75095934-035b-4cb2-819d-a5e8ccf98693","knative.dev/key":"default/serving-cc7q7-ksvc-vcrjq-v200"}
{"severity":"INFO","timestamp":"2023-08-10T08:43:50.157805439Z","logger":"autoscaler","caller":"kpa/kpa.go:148","message":"SKS should be in Proxy mode: want = 0, ebc = -100, #act's = 3 PA Inactive? = true","commit":"ac29233","knative.dev/controller":"knative.dev.serving.pkg.reconciler.autoscaling.kpa.Reconciler","knative.dev/kind":"autoscaling.internal.knative.dev.PodAutoscaler","knative.dev/traceid":"75095934-035b-4cb2-819d-a5e8ccf98693","knative.dev/key":"default/serving-cc7q7-ksvc-vcrjq-v200"}
{"severity":"INFO","timestamp":"2023-08-10T08:43:50.157870938Z","logger":"autoscaler","caller":"kpa/kpa.go:168","message":"PA scale got=1, want=0, desiredPods=0 ebc=-100","commit":"ac29233","knative.dev/controller":"knative.dev.serving.pkg.reconciler.autoscaling.kpa.Reconciler","knative.dev/kind":"autoscaling.internal.knative.dev.PodAutoscaler","knative.dev/traceid":"75095934-035b-4cb2-819d-a5e8ccf98693","knative.dev/key":"default/serving-cc7q7-ksvc-vcrjq-v200"}
{"severity":"INFO","timestamp":"2023-08-10T08:43:50.157902444Z","logger":"autoscaler","caller":"kpa/kpa.go:178","message":"Observed pod counts=kpa.podCounts{want:0, ready:1, notReady:0, pending:0, terminating:0}","commit":"ac29233","knative.dev/controller":"knative.dev.serving.pkg.reconciler.autoscaling.kpa.Reconciler","knative.dev/kind":"autoscaling.internal.knative.dev.PodAutoscaler","knative.dev/traceid":"75095934-035b-4cb2-819d-a5e8ccf98693","knative.dev/key":"default/serving-cc7q7-ksvc-vcrjq-v200"}
{"severity":"INFO","timestamp":"2023-08-10T08:43:50.158029408Z","logger":"autoscaler","caller":"controller/controller.go:543","message":"Reconcile succeeded","commit":"ac29233","knative.dev/controller":"knative.dev.serving.pkg.reconciler.autoscaling.kpa.Reconciler","knative.dev/kind":"autoscaling.internal.knative.dev.PodAutoscaler","knative.dev/traceid":"75095934-035b-4cb2-819d-a5e8ccf98693","knative.dev/key":"default/serving-cc7q7-ksvc-vcrjq-v200","duration":"9.260479374s"}