Open kallangerard opened 6 months ago
qq: does this issue happen starting from v0.14.2-gke.0?
I believe this has been going on for a while. We're on gcr.io/gke-release/custom-metrics-stackdriver-adapter:v0.12.2-gke.0
and I see this in our logs:
2024-03-14 08:23:25.409 EDT Trace[939984059]: [801.213144ms] [801.213144ms] END
2024-03-14 08:23:25.607 EDT Trace[911902081]: "SerializeObject" method:GET,url:/apis/custom.metrics.k8s.io/v1beta1,protocol:HTTP/2.0,mediaType:application/json,encoder:{"encoder":"{\"name\":\"json\",\"pretty\":\"false\",\"yaml\":\"false\"}","name":"versioning"} (14-Mar-2024 12:23:20.508) (total time: 5098ms):
2024-03-14 08:23:25.608 EDT Trace[911902081]: [5.098732769s] [5.098732769s] END
Thanks for providing this information!
/label custom-metrics-stackdriver-adapter
btw, @supersam654 did you find high memory issue in v0.12.2-gke.0? If so, try v0.14.2-gke.0 which fixed OOM/high memory issue.
Release note is here https://github.com/GoogleCloudPlatform/k8s-stackdriver/releases/tag/cm-sd-adapter-v0.14.0
These traces log are ignorable. Could you try increasing log level by
- /adapter
- --use-new-resource-model=false
- --v=1
@kallangerard
Trace prints out because whatever action the trace is about takes longer than what the author expects. This is expected for metrics adapter because it fetches data from Cloud Monitoring.
It appears that in
gcr.io/gke-release/custom-metrics-stackdriver-adapter:v0.14.2-gke.0
, traces are enabled and logging to stdout.We're using the manifests from this commit reference https://github.com/GoogleCloudPlatform/k8s-stackdriver/blob/303ec9ad1ca8d09e2ea93a7ff2ce3aea7a9f8a97/custom-metrics-stackdriver-adapter/deploy/production/adapter.yaml,
We end up with an enormous amount of logs like below: