Revise some queries to use Series API rather than Query API

[yeya24]

Overview

Currently, opencost has some queries that only require metadata like labels, annotations, etc. However, these queries are sent using the Prometheus Query API. The Query API will touch not only the labels, but also chunks. This affect the performance of downstream metrics stores like Cortex, Prometheus, Mimir, Thanos, etc.

All the queries are listed https://github.com/opencost/opencost/blob/develop/pkg/costmodel/allocation.go#L46-L56.

queryFmtNamespaceLabels          = `avg_over_time(kube_namespace_labels[%s])`
queryFmtNamespaceAnnotations     = `avg_over_time(kube_namespace_annotations[%s])`
queryFmtPodLabels                = `avg_over_time(kube_pod_labels[%s])`
queryFmtPodAnnotations           = `avg_over_time(kube_pod_annotations[%s])`
queryFmtServiceLabels            = `avg_over_time(service_selector_labels[%s])`
queryFmtDeploymentLabels         = `avg_over_time(deployment_match_labels[%s])`
queryFmtStatefulSetLabels        = `avg_over_time(statefulSet_match_labels[%s])`
queryFmtDaemonSetLabels          = `sum(avg_over_time(kube_pod_owner{owner_kind="DaemonSet"}[%s])) by (pod, owner_name, namespace, %s)`
queryFmtJobLabels                = `sum(avg_over_time(kube_pod_owner{owner_kind="Job"}[%s])) by (pod, owner_name, namespace ,%s)`
queryFmtPodsWithReplicaSetOwner  = `sum(avg_over_time(kube_pod_owner{owner_kind="ReplicaSet"}[%s])) by (pod, owner_name, namespace ,%s)`
queryFmtReplicaSetsWithoutOwners = `avg(avg_over_time(kube_replicaset_owner{owner_kind="<none>", owner_name="<none>"}[%s])) by (replicaset, namespace, %s)`

Those metrics only have value 1 or 0 and we are not using the calculated results anywhere, so things like avg_over_time doesn't make sense. Using https://prometheus.io/docs/prometheus/latest/querying/api/#finding-series-by-label-matchers API is probably a better option.

[AjayTripathy]

Thanks for digging this up! Anything that helps with DB performance is high priority for us. I'll make sure we investigate this in our upcoming release and get this scale tested.

[AjayTripathy]

Hi @yeya24 this remains high priority for us but I underestimated how much work this is; don't think we'll be able to get to this until v1.100 scheduled for January since there's a large chunk of APIs to swap. This still remains important to us, though.

[github-actions[bot]]

This issue has been marked as stale because it has been open for 180 days with no activity. Please remove the stale label or comment or this issue will be closed in 5 days.

[fkazem]

Not stale

[github-actions[bot]]

This issue has been marked as stale because it has been open for 360 days with no activity. Please remove the stale label or comment or this issue will be closed in 5 days.

[fkazem]

Still not stale