[Bug] Kubecost-based KSM deployment can no longer be deployed

[therealdwright]

Kubecost Helm Chart Version

v2.1.1

Kubernetes Version

1.25.16

Kubernetes Platform

EKS

Description

Since the introduction of #2912 you can no longer ship a kube-state-metrics with kubecost as described in the documentation.

This has been raised previously in #3114 which was closed as completed.

Steps to reproduce

1. Check out v2.1.1 and change to charts directory
2. Add the values as described in the documentation (prometheus.kubeStateMetrics.enabled: true and prometheus.kube-state-metrics.disabled: false
3. Render the template with helm template
4. Note that there is no kubecost-kube-state-metrics deployment rendered

Expected behavior

When following the docs "Enabling Kubecost-based KSM deployment", kubecost has its dedicated kube-state-metrics deployment rendered.

Impact

Prevents upgrades to v2.x for users who wish to use a dedicated KSM for kubecost.

Screenshots

No response

Logs

No response

Slack discussion

No response

Troubleshooting

• [X] I have read and followed the issue guidelines and this is a bug impacting only the Helm chart.
• [X] I have searched other issues in this repository and mine is not recorded.

[teevans]

@jessegoodier - I think you probably have the latest understanding here

[jessegoodier]

We will need to update our docs. KSM was removed from the chart because kubecost emits all KSM metrics it requires, and we are constantly patching utilities that are not required.

@therealdwright- Can you confirm you are using the kubecost bundled prometheus server?

And -would a solution of installing the kube-state-metrics helm chart and having kubecost-bundled prometheus scrape this work for you?

basic config would be:

helm install kube-state-metrics \
 --repo https://prometheus-community.github.io/helm-charts kube-state-metrics \
 --namespace kube-state-metrics --create-namespace

and add to kubecost helm values:

prometheus:
  extraScrapeConfigs: |
    - job_name: kubecost
      honor_labels: true
      scrape_interval: 1m
      scrape_timeout: 60s
      metrics_path: /metrics
      scheme: http
      dns_sd_configs:
      - names:
        - {{ template "cost-analyzer.serviceName" . }}
        type: 'A'
        port: 9003
    - job_name: kubecost-networking
      kubernetes_sd_configs:
        - role: pod
      relabel_configs:
      # Scrape only the the targets matching the following metadata
        - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_instance]
          action: keep
          regex:  kubecost
        - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
          action: keep
          regex:  network-costs
    - job_name: kube-state-metrics
      kubernetes_sd_configs:
        - role: pod
      relabel_configs:
      # Scrape only the the targets matching the following metadata
        - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_instance]
          action: keep
          regex:  kube-state-metrics
        - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
          action: keep
          regex:  kube-state-metrics

[therealdwright]

We will need to update our docs. KSM was removed from the chart because kubecost emits all KSM metrics it requires, and we are constantly patching utilities that are not required.

No problem, we removed the component yesterday (we were updating it with a kustomize patch which was what made us notice it was gone).

Forgive my ignorance, but was this removal mentioned in the release notes anywhere? I went over the release history from our old version (1.108.1) to latest and did not see any mention of this removal.

[jessegoodier]

We will need to update our docs. KSM was removed from the chart because kubecost emits all KSM metrics it requires, and we are constantly patching utilities that are not required.

No problem, we removed the component yesterday (we were updating it with a kustomize patch which was what made us notice it was gone).

Forgive my ignorance, but was this removal mentioned in the release notes anywhere? I went over the release history from our old version (1.108.1) to latest and did not see any mention of this removal.

You are not blind; I can't find it either. I sincerely apologize for that.

it was part of this: https://github.com/kubecost/cost-analyzer-helm-chart/pull/2912/files

2.0 was a massive overhaul. Which isn't a good excuse. But this KSM note omission was a causality of a huge PR with more change than we could easily track.

I have asked the team to do better with release notes and look for better ideas to make sure we don't miss this type of change in the future.

If you are good, please close the issue.