search

kubernetes / k8s.io

Code and configuration to manage Kubernetes project infrastructure, including various *.k8s.io sites
https://git.k8s.io/community/sig-k8s-infra
Apache License 2.0
735 stars 822 forks source link

audit: reduce noise in logging export #2111

Open spiffxp opened 3 years ago

spiffxp commented 3 years ago

https://github.com/kubernetes/k8s.io/pull/2102 introduced export of logging resources to the audit script. Based on review of the first audit job PR that used this (https://github.com/kubernetes/k8s.io/pull/2094) there is some noise we should filter out to ease review burden.

empty metrics.json

Currently there are lots of services/logging/metrics.json files with content []. If there are no metrics, we shouldn't export them.

e2e test logs

Logs appear to be showing up for all pods used in e2e tests. For example audit/projects/k8s-infra-e2e-boskos-010/services/logging/logs.json has a diff that looks like:

   "projects/k8s-infra-e2e-boskos-010/logs/events",
   "projects/k8s-infra-e2e-boskos-010/logs/externalname-service",
-  "projects/k8s-infra-e2e-boskos-010/logs/externalsvc",
-  "projects/k8s-infra-e2e-boskos-010/logs/filler-pod-0d464eed-160c-4f37-963b-36a011030391",
-  "projects/k8s-infra-e2e-boskos-010/logs/filler-pod-31c7c414-74fa-49ca-8a8f-48c571c143a3",
   "projects/k8s-infra-e2e-boskos-010/logs/filler-pod-3638847c-8c6c-47dc-9c89-32c571411622",
   "projects/k8s-infra-e2e-boskos-010/logs/filler-pod-b6d92649-31d0-4d9a-8634-f0f8fe06ebbe",
# ...
   "projects/k8s-infra-e2e-boskos-010/logs/test-container-subpath-projected-fwgf",
+  "projects/k8s-infra-e2e-boskos-010/logs/test-container-subpath-projected-2v7w",
+  "projects/k8s-infra-e2e-boskos-010/logs/test-container-subpath-projected-fwgf",

We should either choose to ignore/filter these out, or determine how to configure our e2e tests to not send any logs. I swear we had done this a while ago, but we only ever verified by way of costs going down.

/wg k8s-infra /area infra/auditing /priority important-longterm /milestone v1.22

spiffxp commented 3 years ago

https://github.com/kubernetes/k8s.io/pull/2133 removed export of logs for k8s-infra-e2e projects as a start

Should survey remaining log churn in audit PR's to track down what can be done

spiffxp commented 3 years ago

Log noise that seems like it shouldn't be present:

gcloud logging logs list --help says Only logs that contain log entries are listed.

Every project has at least these two buckets

$ gcloud logging buckets list
LOCATION  BUCKET_ID  RETENTION_DAYS  LIFECYCLE_STATE  LOCKED  CREATE_TIME  UPDATE_TIME
global    _Default   30              ACTIVE
global    _Required  400             ACTIVE           True

And at least these two sinks that route to them

$ gcloud logging sinks list --format=yaml
---
destination: logging.googleapis.com/projects/spiffxp-gke-dev/locations/global/buckets/_Required
filter: LOG_ID("cloudaudit.googleapis.com/activity") OR LOG_ID("externalaudit.googleapis.com/activity")
  OR LOG_ID("cloudaudit.googleapis.com/system_event") OR LOG_ID("externalaudit.googleapis.com/system_event")
  OR LOG_ID("cloudaudit.googleapis.com/access_transparency") OR LOG_ID("externalaudit.googleapis.com/access_transparency")
name: _Required
---
destination: logging.googleapis.com/projects/spiffxp-gke-dev/locations/global/buckets/_Default
filter: NOT LOG_ID("cloudaudit.googleapis.com/activity") AND NOT LOG_ID("externalaudit.googleapis.com/activity")
  AND NOT LOG_ID("cloudaudit.googleapis.com/system_event") AND NOT LOG_ID("externalaudit.googleapis.com/system_event")
  AND NOT LOG_ID("cloudaudit.googleapis.com/access_transparency") AND NOT LOG_ID("externalaudit.googleapis.com/access_transparency")
name: _Default

So are we losing system_event logs because nothing has happened to generate a log entry there in 400 days?

spiffxp commented 3 years ago

/remove-priority important-longterm /priority backlog It's annoying but it's not really creating a lot of additional review burden for me at this point

spiffxp commented 3 years ago

/milestone clear

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

ameukam commented 2 years ago

/remove-lifecycle stale

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

ameukam commented 2 years ago

/remove-lifecycle stale /lifecycle frozen