search

openshift / must-gather

A client tool for gathering information about an operator managed component.
Apache License 2.0
104 stars 189 forks source link

must-gather.log spams logs about egressfirewalls and egressqoses #362

Closed liornoy closed 10 months ago

liornoy commented 1 year ago

Looking at the must-gather.log from my latest ci-test run, it seems like we throw INFO logs which make the log harder to read. See:

[must-gather-xxbbf] POD 2023-05-17T11:59:17.152066356Z Gathering data for ns/openshift-cluster-version...
[must-gather-xxbbf] POD 2023-05-17T11:59:19.555676602Z W0517 11:59:19.555610       8 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:19.558880211Z W0517 11:59:19.558851       8 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:19.663594946Z Gathering data for ns/default...
[must-gather-xxbbf] POD 2023-05-17T11:59:19.796052158Z W0517 11:59:19.796007       8 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:19.802501693Z W0517 11:59:19.802477       8 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:19.933839951Z Gathering data for ns/openshift...
[must-gather-xxbbf] POD 2023-05-17T11:59:20.179844728Z W0517 11:59:20.179806       8 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:20.182575941Z W0517 11:59:20.182553       8 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:20.279559456Z Gathering data for ns/kube-system...
[must-gather-xxbbf] POD 2023-05-17T11:59:20.408955535Z W0517 11:59:20.408913       8 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:20.411238487Z W0517 11:59:20.411218       8 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:20.678549529Z Gathering data for ns/openshift-etcd...
[must-gather-xxbbf] POD 2023-05-17T11:59:21.679718696Z W0517 11:59:21.679675       8 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:21.683752225Z W0517 11:59:21.683726       8 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:22.060183522Z Wrote inspect data to must-gather.
[must-gather-xxbbf] POD 2023-05-17T11:59:23.790812585Z Gathering data for ns/openshift-config...
[must-gather-xxbbf] POD 2023-05-17T11:59:23.928293099Z W0517 11:59:23.928253     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:23.932897522Z W0517 11:59:23.932869     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:24.044526544Z Gathering data for ns/openshift-config-managed...
[must-gather-xxbbf] POD 2023-05-17T11:59:24.313978398Z W0517 11:59:24.313931     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:24.319885490Z W0517 11:59:24.319850     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:24.411760399Z Gathering data for ns/openshift-authentication...
[must-gather-xxbbf] POD 2023-05-17T11:59:24.699325865Z W0517 11:59:24.699277     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:24.702712681Z W0517 11:59:24.702687     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:24.886439798Z Gathering data for ns/openshift-authentication-operator...
[must-gather-xxbbf] POD 2023-05-17T11:59:25.100911831Z W0517 11:59:25.100877     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T11:59:25.106497028Z W0517 11:59:25.106460     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
.
.
.
[must-gather-xxbbf] POD 2023-05-17T12:00:06.109510622Z Gathering data for ns/openshift-machine-config-operator...
[must-gather-xxbbf] POD 2023-05-17T12:00:06.616004749Z W0517 12:00:06.615962     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:06.619411887Z W0517 12:00:06.619376     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:07.024611216Z Gathering data for ns/openshift-kni-infra...
[must-gather-xxbbf] POD 2023-05-17T12:00:12.765883020Z W0517 12:00:12.765836     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:12.769646923Z W0517 12:00:12.769622     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:12.847601824Z Gathering data for ns/openshift-openstack-infra...
[must-gather-xxbbf] POD 2023-05-17T12:00:12.959084289Z W0517 12:00:12.959043     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:12.961865990Z W0517 12:00:12.961835     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:13.044327483Z Gathering data for ns/openshift-ovirt-infra...
[must-gather-xxbbf] POD 2023-05-17T12:00:13.165554607Z W0517 12:00:13.165507     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:13.168633601Z W0517 12:00:13.168601     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:13.255017243Z Gathering data for ns/openshift-vsphere-infra...
[must-gather-xxbbf] POD 2023-05-17T12:00:13.384294288Z W0517 12:00:13.384242     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:13.387208878Z W0517 12:00:13.387171     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:13.479537846Z Gathering data for ns/openshift-nutanix-infra...
[must-gather-xxbbf] POD 2023-05-17T12:00:13.619418134Z W0517 12:00:13.619377     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:13.635433684Z W0517 12:00:13.635400     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:13.736145710Z Gathering data for ns/openshift-marketplace...
[must-gather-xxbbf] POD 2023-05-17T12:00:14.031202793Z W0517 12:00:14.031171     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:14.034909709Z W0517 12:00:14.034879     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:14.237141996Z Gathering data for ns/openshift-user-workload-monitoring...
[must-gather-xxbbf] POD 2023-05-17T12:00:14.363284391Z W0517 12:00:14.363225     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:14.367420112Z W0517 12:00:14.367390     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:14.980714662Z Gathering data for ns/openshift-sdn...
[must-gather-xxbbf] POD 2023-05-17T12:00:15.717496223Z W0517 12:00:15.717454     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:15.720279047Z W0517 12:00:15.720254     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:16.021123774Z Gathering data for ns/openshift-host-network...
[must-gather-xxbbf] POD 2023-05-17T12:00:16.150152499Z W0517 12:00:16.150110     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:16.153603423Z W0517 12:00:16.153575     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:16.255697194Z Gathering data for ns/openshift-network-diagnostics...
[must-gather-xxbbf] POD 2023-05-17T12:00:16.513744169Z W0517 12:00:16.513703     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:16.516812516Z W0517 12:00:16.516630     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:16.742676795Z Gathering data for ns/openshift-network-operator...
[must-gather-xxbbf] POD 2023-05-17T12:00:17.719904200Z W0517 12:00:17.719867     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:17.722734390Z W0517 12:00:17.722701     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:17.856745666Z Gathering data for ns/openshift-cloud-network-config-controller...
[must-gather-xxbbf] POD 2023-05-17T12:00:17.971103700Z W0517 12:00:17.971046     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:17.977205771Z W0517 12:00:17.977095     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:18.120809852Z Gathering data for ns/openshift-apiserver-operator...
[must-gather-xxbbf] POD 2023-05-17T12:00:18.333571325Z W0517 12:00:18.333524     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:18.343028392Z W0517 12:00:18.342996     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:18.550916367Z Gathering data for ns/openshift-apiserver...
[must-gather-xxbbf] POD 2023-05-17T12:00:18.962479751Z W0517 12:00:18.962435     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:18.966718088Z W0517 12:00:18.966688     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:19.337594227Z Gathering data for ns/openshift-controller-manager-operator...
[must-gather-xxbbf] POD 2023-05-17T12:00:19.636247453Z W0517 12:00:19.636200     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:19.639258446Z W0517 12:00:19.639224     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:19.772732552Z Gathering data for ns/openshift-controller-manager...
[must-gather-xxbbf] POD 2023-05-17T12:00:20.012199454Z W0517 12:00:20.012149     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:20.016432685Z W0517 12:00:20.016389     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:20.168098552Z Gathering data for ns/openshift-route-controller-manager...
[must-gather-xxbbf] POD 2023-05-17T12:00:20.359217322Z W0517 12:00:20.359170     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:20.362151005Z W0517 12:00:20.362123     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:20.515942559Z Gathering data for ns/openshift-cluster-samples-operator...
[must-gather-xxbbf] POD 2023-05-17T12:00:20.740834006Z W0517 12:00:20.740216     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:20.753593652Z W0517 12:00:20.753547     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:21.223975051Z Gathering data for ns/openshift-operator-lifecycle-manager...
[must-gather-xxbbf] POD 2023-05-17T12:00:21.619559658Z W0517 12:00:21.619190     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:21.625162052Z W0517 12:00:21.625135     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:21.829182721Z Gathering data for ns/openshift-service-ca-operator...
[must-gather-xxbbf] POD 2023-05-17T12:00:21.975115463Z W0517 12:00:21.975067     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:21.981283096Z W0517 12:00:21.981246     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:22.086810692Z Gathering data for ns/openshift-service-ca...
[must-gather-xxbbf] POD 2023-05-17T12:00:22.241755336Z W0517 12:00:22.241703     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:22.246609074Z W0517 12:00:22.246575     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:22.352120347Z Gathering data for ns/openshift-cluster-csi-drivers...
[must-gather-xxbbf] POD 2023-05-17T12:00:22.454613572Z W0517 12:00:22.454565     176 util.go:119] the server doesn't have a resource type egressfirewalls, skipping the inspection
[must-gather-xxbbf] POD 2023-05-17T12:00:22.457398208Z W0517 12:00:22.457362     176 util.go:119] the server doesn't have a resource type egressqoses, skipping the inspection

I file this issue to suggest muting the the server doesn't have.. logs to provide a cleaner log output for the must-gather.

sferich888 commented 1 year ago

The the server doesn't have.. message is the same one any oc command gives when it can't access something.

This issue will need someone like @tssurya, @npinaeva or someone from the networking team to review.

The following are lines of code that generate this error; so https://github.com/openshift/must-gather/blob/master/collection-scripts/gather_network_logs#L78 https://github.com/openshift/must-gather/blob/master/collection-scripts/gather_network_logs#L91 (likely using related resources, to get to the qoses object)

npinaeva commented 1 year ago

I think the source is here https://github.com/openshift/oc/blob/master/pkg/cli/admin/inspect/util.go#L118-L119 I have seen this warning for other resource types too (not networking-specific), so maybe we need to change log level for it?

sferich888 commented 1 year ago

@soltysh / @ingvagabund can you investigate here if changing the log level in inspect is correct; or if this is a collection issue (IE: a request for a missing/bad resource)?

ingvagabund commented 1 year ago

The warning was introduced to avoid displaying an error when a non-existing resource is inspected. It's safe to increase the log level +1 over the default one. Just, it might be impossible to propagate the log level from the initial command down to the low level binary in the chain

ingvagabund commented 1 year ago

@liornoy I assume all the must-gather logs can be found in any CI job?

liornoy commented 1 year ago

@ingvagabund yes, each cnf-tests CI job of ours collects must-gather logs. But the logs about "egressfirewalls" and "egressqoses" don't reappear every time. I think I encountered that in a corner case where the cluster didn't setup correctly.

openshift-bot commented 10 months ago

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

liornoy commented 10 months ago

I will close this for now.