search

kubecost / cost-analyzer-helm-chart

Kubecost helm chart
http://kubecost.com/install
Apache License 2.0
466 stars 413 forks source link

Add Federated ETL Pipeline #1700

Closed kaelanspatel closed 10 months ago

kaelanspatel commented 1 year ago

Current federation of multiple clusters relies on Thanos and building ETL data from combined metrics. The goal of adding a federated ETL is to mitigate memory and efficiency issues with the Thanos implementation, including costly builds and shipment of unnecessary unaggregated metrics data.

This would be done by creating a pipeline that uses existing integrations with s3 to federate clusters' ETL data, designating an install of Kubecost to act as a "Federator" which consolidates the data.

┆Issue is synchronized with this Jira Task by Unito

dwbrown2 commented 1 year ago

This actually completed? Seems like a great tracking issue for Federated ETL work in 1.98!

Adam-Stack-PM commented 1 year ago

@dwbrown2 This part of the work is completed! We could have been more clear in what this issue covered.

This issue relates only to the initial beta of Federated ETL upload. This will designate an s3 bucket which will be used to upload ETL data from federated clusters to be later used for Federated ETL.

@kaelanspatel and I went back and forth about including this in the release notes.

We decided not to include it to avoid confusion as this part of the work does not add value without the other federated work planned for v1.98. At the same time, we would like to reach out to a few customers to get feedback on the upload beta work.

srpomeroy commented 1 year ago

The @kubecost/solutions-engineers team typically modifies ETL settings on secondary clusters to minimize the resource footprint. With this new functionality, will we have to re-enable ETL on secondaries?

AjayTripathy commented 1 year ago

I suspect we'll need to re-enable a very small subset of ETL. Like 3d worth? It will increase secondary usage a bit but not too much. That sound right @kaelanspatel ?

github-actions[bot] commented 10 months ago

This issue has been marked as stale because it has been open for 360 days with no activity. Please remove the stale label or comment or this issue will be closed in 5 days.

github-actions[bot] commented 10 months ago

This issue was closed because it has been inactive for 365 days with no activity.