Can we remove redundant logs about the server missing resource for datagatherer?
It makes it very difficult to actually see logs that do mean something.
What happened?
Take a look at this snippet of logs:
2024/09/23 13:41:32 retrying in 1m55.254386533s after error: post to server failed: received response with status code 404. Body: []
W0923 13:41:33.351182 1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:232: failed to list awspca.cert-manager.io/v1beta1, Resource=awspcaclusterissuers: the server could not find the requested resource
2024/09/23 13:41:33 server missing resource for datagatherer of "awspca.cert-manager.io/v1beta1, Resource=awspcaclusterissuers"
W0923 13:41:33.694672 1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:232: failed to list route.openshift.io/v1, Resource=routes: routes.route.openshift.io is forbidden: User "system:serviceaccount:venafi:venafi-kubernetes-agent" cannot list resource "routes" in API group "route.openshift.io" at the cluster scope
2024/09/23 13:41:33 datagatherer informer for "route.openshift.io/v1, Resource=routes" has failed and is backing off due to error: failed to list route.openshift.io/v1, Resource=routes: routes.route.openshift.io is forbidden: User "system:serviceaccount:venafi:venafi-kubernetes-agent" cannot list resource "routes" in API group "route.openshift.io" at the cluster scope
W0923 13:41:41.794445 1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:232: failed to list networking.istio.io/v1alpha3, Resource=virtualservices: the server could not find the requested resource
2024/09/23 13:41:41 server missing resource for datagatherer of "networking.istio.io/v1alpha3, Resource=virtualservices"
W0923 13:41:47.544162 1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:232: failed to list cas-issuer.jetstack.io/v1beta1, Resource=googlecasissuers: the server could not find the requested resource
2024/09/23 13:41:47 server missing resource for datagatherer of "cas-issuer.jetstack.io/v1beta1, Resource=googlecasissuers"
W0923 13:41:49.478712 1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:232: failed to list awspca.cert-manager.io/v1beta1, Resource=awspcaissuers: the server could not find the requested resource
2024/09/23 13:41:49 server missing resource for datagatherer of "awspca.cert-manager.io/v1beta1, Resource=awspcaissuers"
W0923 13:41:58.449284 1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:232: failed to list networking.istio.io/v1alpha3, Resource=gateways: the server could not find the requested resource
2024/09/23 13:41:58 server missing resource for datagatherer of "networking.istio.io/v1alpha3, Resource=gateways"
W0923 13:42:03.759841 1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:232: failed to list firefly.venafi.com/v1, Resource=issuers: issuers.firefly.venafi.com is forbidden: User "system:serviceaccount:venafi:venafi-kubernetes-agent" cannot list resource "issuers" in API group "firefly.venafi.com" at the cluster scope
2024/09/23 13:42:03 datagatherer informer for "firefly.venafi.com/v1, Resource=issuers" has failed and is backing off due to error: failed to list firefly.venafi.com/v1, Resource=issuers: issuers.firefly.venafi.com is forbidden: User "system:serviceaccount:venafi:venafi-kubernetes-agent" cannot list resource "issuers" in API group "firefly.venafi.com" at the cluster scope
W0923 13:42:13.822678 1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:232: failed to list cas-issuer.jetstack.io/v1beta1, Resource=googlecasclusterissuers: the server could not find the requested resource
2024/09/23 13:42:13 server missing resource for datagatherer of "cas-issuer.jetstack.io/v1beta1, Resource=googlecasclusterissuers"
W0923 13:42:17.917563 1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:232: failed to list networking.istio.io/v1alpha3, Resource=virtualservices: the server could not find the requested resource
2024/09/23 13:42:17 server missing resource for datagatherer of "networking.istio.io/v1alpha3, Resource=virtualservices"
W0923 13:42:26.587788 1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:232: failed to list awspca.cert-manager.io/v1beta1, Resource=awspcaclusterissuers: the server could not find the requested resource
2024/09/23 13:42:26 server missing resource for datagatherer of "awspca.cert-manager.io/v1beta1, Resource=awspcaclusterissuers"
W0923 13:42:27.221710 1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:232: failed to list route.openshift.io/v1, Resource=routes: routes.route.openshift.io is forbidden: User "system:serviceaccount:venafi:venafi-kubernetes-agent" cannot list resource "routes" in API group "route.openshift.io" at the cluster scope
2024/09/23 13:42:27 datagatherer informer for "route.openshift.io/v1, Resource=routes" has failed and is backing off due to error: failed to list route.openshift.io/v1, Resource=routes: routes.route.openshift.io is forbidden: User "system:serviceaccount:venafi:venafi-kubernetes-agent" cannot list resource "routes" in API group "route.openshift.io" at the cluster scope
W0923 13:42:35.502389 1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:232: failed to list awspca.cert-manager.io/v1beta1, Resource=awspcaissuers: the server could not find the requested resource
2024/09/23 13:42:35 server missing resource for datagatherer of "awspca.cert-manager.io/v1beta1, Resource=awspcaissuers"
W0923 13:42:47.105401 1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:232: failed to list cas-issuer.jetstack.io/v1beta1, Resource=googlecasissuers: the server could not find the requested resource
2024/09/23 13:42:47 server missing resource for datagatherer of "cas-issuer.jetstack.io/v1beta1, Resource=googlecasissuers"
W0923 13:42:50.029914 1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:232: failed to list cas-issuer.jetstack.io/v1beta1, Resource=googlecasclusterissuers: the server could not find the requested resource
2024/09/23 13:42:50 server missing resource for datagatherer of "cas-issuer.jetstack.io/v1beta1, Resource=googlecasclusterissuers"
W0923 13:42:53.687645 1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:232: failed to list networking.istio.io/v1alpha3, Resource=gateways: the server could not find the requested resource
2024/09/23 13:42:53 server missing resource for datagatherer of "networking.istio.io/v1alpha3, Resource=gateways"
W0923 13:42:58.046565 1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:232: failed to list firefly.venafi.com/v1, Resource=issuers: issuers.firefly.venafi.com is forbidden: User "system:serviceaccount:venafi:venafi-kubernetes-agent" cannot list resource "issuers" in API group "firefly.venafi.com" at the cluster scope
2024/09/23 13:42:58 datagatherer informer for "firefly.venafi.com/v1, Resource=issuers" has failed and is backing off due to error: failed to list firefly.venafi.com/v1, Resource=issuers: issuers.firefly.venafi.com is forbidden: User "system:serviceaccount:venafi:venafi-kubernetes-agent" cannot list resource "issuers" in API group "firefly.venafi.com" at the cluster scope
W0923 13:43:03.315884 1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:232: failed to list networking.istio.io/v1alpha3, Resource=virtualservices: the server could not find the requested resource
2024/09/23 13:43:03 server missing resource for datagatherer of "networking.istio.io/v1alpha3, Resource=virtualservices"
W0923 13:43:23.872934 1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:232: failed to list route.openshift.io/v1, Resource=routes: routes.route.openshift.io is forbidden: User "system:serviceaccount:venafi:venafi-kubernetes-agent" cannot list resource "routes" in API group "route.openshift.io" at the cluster scope
2024/09/23 13:43:23 datagatherer informer for "route.openshift.io/v1, Resource=routes" has failed and is backing off due to error: failed to list route.openshift.io/v1, Resource=routes: routes.route.openshift.io is forbidden: User "system:serviceaccount:venafi:venafi-kubernetes-agent" cannot list resource "routes" in API group "route.openshift.io" at the cluster scope
W0923 13:43:25.693858 1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:232: failed to list awspca.cert-manager.io/v1beta1, Resource=awspcaissuers: the server could not find the requested resource
2024/09/23 13:43:25 server missing resource for datagatherer of "awspca.cert-manager.io/v1beta1, Resource=awspcaissuers"
W0923 13:43:26.194047 1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:232: failed to list awspca.cert-manager.io/v1beta1, Resource=awspcaclusterissuers: the server could not find the requested resource
2024/09/23 13:43:26 server missing resource for datagatherer of "awspca.cert-manager.io/v1beta1, Resource=awspcaclusterissuers"
2024/09/23 13:43:27 Posting data to: https://api.venafi.cloud/
2024/09/23 13:43:27 retrying in 2m53.275055834s after error: post to server failed: received response with status code 404. Body: []
W0923 13:43:33.822935 1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:232: failed to list networking.istio.io/v1alpha3, Resource=gateways: the server could not find the requested resource
2024/09/23 13:43:33 server missing resource for datagatherer of "networking.istio.io/v1alpha3, Resource=gateways"
What should had happened?
If the cluster doesn't have a resource, stop looking for it.
Or at the very least limit the logs about this 1 line only.
Worst case the agent is being a very bad citizen in the cluster continually polling the k8s API for things that just aren't there.
Possible fixes
1) Log reduction
1) Possible logic change
I did previously try adding --strict but this seems to have no effect in my previous attempts.
Log Filtering
I manually tried taking a small agent log and filtering out a bunch of log lines to see what diference it would make. Here's my resutls:
k logs venafi-kubernetes-agent-5b99574597-8nh9p > agent-full.log
awk '!/server missing resource for datagatherer of/' agent-full.log > agent-processed.log
sed -i .bckp '/the server could not find the requested resource/d' agent-processed.log
sed -i .bckp '/cannot list resource/d' agent-processed.log
I've attached both log fils here to see which you prefer to look at:
Can we remove redundant logs about the
server missing resource for datagatherer
? It makes it very difficult to actually see logs that do mean something.What happened?
Take a look at this snippet of logs:
What should had happened?
If the cluster doesn't have a resource, stop looking for it. Or at the very least limit the logs about this 1 line only.
Worst case the agent is being a very bad citizen in the cluster continually polling the k8s API for things that just aren't there.
Possible fixes
1) Log reduction 1) Possible logic change
I did previously try adding
--strict
but this seems to have no effect in my previous attempts.Log Filtering
I manually tried taking a small agent log and filtering out a bunch of log lines to see what diference it would make. Here's my resutls:
Here's my very quick commands:
I've attached both log fils here to see which you prefer to look at: