search

cortexproject / cortex

A horizontally scalable, highly available, multi-tenant, long term Prometheus.
https://cortexmetrics.io/
Apache License 2.0
5.4k stars 782 forks source link

Obtain tenant ID from Prometheus timeseries label #3318

Closed blind-oracle closed 3 years ago

blind-oracle commented 3 years ago

We're using several HA pairs of Prometheus and want to be able to ingest to different tenants from a single HA pair based on labels in a trusted environment. We need this to separate teams/application groups.

Currently out of box this is not supported and I wrote a small proxy that sits between Prometheus & Cortex and does the job (https://github.com/blind-oracle/cortex-tenant). It looks for a predefined label and uses it as a tenant ID.

Do you think it's worth to implement something like this in Cortex directly somewhere in the Distributor?

roidelapluie commented 3 years ago

There is a auth gateway in the roadmap

https://cortexmetrics.io/docs/roadmap/

blind-oracle commented 3 years ago

@roidelapluie

Yeah, but from proposed functionality it doesn't look like that it would allow multiple tenants per single Prometheus... But anyway, if such entity is planned then probably this feature should go there.

pracucci commented 3 years ago

There are thoughts about adding relabeling support during ingestion (in distributors). This use case could be a good candidate for a custom relabelling rule (where you can change the tenant ID). @gouthamve thoughts?

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

blind-oracle commented 3 years ago

Any chance on this?

pracucci commented 3 years ago

Any chance on this?

No progress on this yet. I think doing it at a proxy level is still the only (and probably) best way to do it. In a previous message I suggested whether we could use relabelling support but, after more thoughts, I'm not sure it's easy to fit into the distributor.

roidelapluie commented 3 years ago

Any chance on this?

No progress on this yet. I think doing it at a proxy level is still the only (and probably) best way to do it. In a previous message I suggested whether we could use relabelling support but, after more thoughts, I'm not sure it's easy to fit into the distributor.

I tried that but many questions rise: which limits to apply, should the metrics be fed twice in the distributor ? (and how to prevent "loops" then), etc..

pracucci commented 3 years ago

I tried that but many questions rise: which limits to apply, should the metrics be fed twice in the distributor ? (and how to prevent "loops" then), etc..

I see the pain and I think it doesn't really fit. The whole Cortex has been designed to receive the tenant in the input request, so the distributor (like other services) have been designed this way. I think doing it at a proxy level is still the right (and best) way to do.

blind-oracle commented 3 years ago

Ok, thanks guys, then I guess let's leave it as is then for now and use the proxy 👍

blind-oracle commented 3 years ago

@pracucci btw maybe it's worth to add a link to the proxy in docs somewhere, probably it would be useful to somebody until the Auth gateway is released. Maybe even after that, depending on the use case.

pracucci commented 3 years ago

@pracucci btw maybe it's worth to add a link to the proxy in docs somewhere, probably it would be useful to somebody until the Auth gateway is released. Maybe even after that, depending on the use case.

@blind-oracle Not sure I follow you. Using a custom proxy (to be built based on your specific use case) was a suggestion, but we haven't built that proxy, so I'm not sure what we should link in the doc.

blind-oracle commented 3 years ago

@pracucci I meant the proxy that I've built to extract tenant ID from Prometheus labels, the link is in the initial message (https://github.com/blind-oracle/cortex-tenant). I guess that covers a lot of use cases when Cortex is running in a controlled environment and tenants are used to separate e.g. teams or applications.

At least that's how we use it at @swisscom

pracucci commented 3 years ago

@pracucci I meant the proxy that I've built to extract tenant ID from Prometheus labels, the link is in the initial message (https://github.com/blind-oracle/cortex-tenant). I guess that covers a lot of use cases when Cortex is running in a controlled environment and tenants are used to separate e.g. teams or applications.

Oh sorry. Yes sure. Please submit a PR to mention it in the doc.