Updating Cache: QoL Improvments

[logyball]

What

• Adding "Id" to the provider struct, to hold project/profile/subscription
• Reducing logging from all of disk.Meta() to name, size, created
• Added polling interval as a command line flag

Why

Provider Id is grabbed via a switch statement in our exporter:

            var pid string
            switch name {
            case "gcp":
                pid = meta["project"]
            case "aws":
                pid = meta["profile"]
            case "azure":
                pid = meta["subscription"]
            default:
                pid = meta.String()
            }

Which could be just stored on the provider itself, as we're already storing it in some way via the metadata. This makes it a bit easier to grab.

As for logging, we are logging all metadata with the disk that is retrieved. I think this could be potentially useful, but I'm not sure we're using it. I'm not completely married to the idea, but I think it'll clean up the logs a bit. Happy to add it back in if it's useful, just not sure we're using it.

Log output Before and After:

Before:

time=2023-12-01T16:03:09.214-05:00 level=INFO msg="unused disk found" provider=GCP metadata="project=grafanalabs-dev" provider=GCP name=gke-dev-us-central1-2e-pvc-fd83188e-fd67-4eaf-85f4-b88a9d921f4e created=2021-03-10T11:39:47.064-08:00 kubernetes.io/created-for/pv/name=pvc-fd83188e-fd67-4eaf-85f4-b88a9d921f4e kubernetes.io/created-for/pvc/name=elasticsearch-pvc-elasticsearch-5-server-0 kubernetes.io/created-for/pvc/namespace=integration zone=us-central1-f
...
time=2023-12-01T16:00:57.469-05:00 level=INFO msg="unused disk found" provider=AWS metadata="profile=workloads-prod" provider=AWS name=pvc-feaf1cf1-0a00-4af2-a7ba-b2ae64f0385f created=2023-11-27T10:39:41.731Z ebs.csi.aws.com/cluster=true kubernetes.io/created-for/pv/name=pvc-feaf1cf1-0a00-4af2-a7ba-b2ae64f0385f kubernetes.io/created-for/pvc/name=grafana-agent-helm-grafana-agent-helm-7 kubernetes.io/created-for/pvc/namespace=grafana-agent zone=us-east-2a
...
time=2023-12-01T16:05:16.958-05:00 level=INFO msg="unused disk found" provider=Azure metadata="subscription=179c4f30-ebd8-489e-92bc-fb64588dadb3" provider=Azure name=grafana-enterprise-win2006-dev_OsDisk_1_f9e292dd008d489aa5b4f4f07a57d568 created=2021-03-11T09:18:05.558Z location=northeurope resource-group=GRAFANA-ENTERPRISE-DEV

After:

time=2023-12-01T16:06:24.526-05:00 level=INFO msg="unused disk found" provider=GCP provider_id=grafanalabs-dev name=gke-dev-eu-west-3-acbe-pvc-e48c7ed2-0f70-46c2-a963-a81837f87bc7 size_gb=300 created=2023-05-26T03:56:33.014-07:00
...
time=2023-12-01T16:06:02.400-05:00 level=INFO msg="unused disk found" provider=AWS provider_id=workloads-prod name=pvc-ad79f24f-28e3-4fc7-8179-58e013159679 size_gb=100 created=2023-10-06T18:24:03.262Z
...
time=2023-12-01T16:07:01.034-05:00 level=INFO msg="unused disk found" provider=Azure provider_id=179c4f30-ebd8-489e-92bc-fb64588dadb3 name=pvc-e56a18f0-3fad-40db-90ab-90f56cda0447 size_gb=130 created=2022-06-13T13:57:25.965Z

[inkel]

As for logging, we are logging all metadata with the disk that is retrieved. I think this could be potentially useful, but I'm not sure we're using it. I'm not completely married to the idea, but I think it'll clean up the logs a bit. Happy to add it back in if it's useful, just not sure we're using it.

Perhaps we could hide this behind a -v flag to enable verbose logging 🤔

One thing that I was thinking during the weekend was that if we include the metadata in the logs, we could use the groups feature of slog, I think that would make the logging easier to understand. WDYT?