Add a way to handle pyroscope.ebpf → pyroscope.write multitenancy

[towolf]

Request

I see no way to send captured ebpf samples from a multitenant cluster (say, one tenant per namespace) to a Pyroscope server using different X-Scope-OrgId headers.

This is possible with Loki but seemingly not with pyroscope.*

Use case

Mutitenancy with Pyroscope.

[towolf]

My config looks like this:

          pyroscope.ebpf "instance" {
            forward_to = [pyroscope.write.endpoint.receiver]
            targets = discovery.relabel.profiling_pods.output

            demangle = "full"
            python_enabled = false
          }

          pyroscope.write "endpoint" {
            endpoint {
              basic_auth {
                username = "${var.pyroscope_remote_username}"
                password = "${var.pyroscope_remote_password}"
              }
              url = "${var.pyroscope_remote_push_url}"
              headers = {
                "X-Scope-OrgID" = "main", # would like to place $namespace here
              }
            }

[hainenber]

hi @towolf, just curious if you've enabled multi-tenancy for the Pyroscope server, as depicted here 👀

[towolf]

Hi, I'm looking for a way to send samples in a multi-tenant way using grafana-agent. This is not about the server component.

Let's say I have a namespace A, and any samples from pods in namespace A should go to tenant A on the pyroscope server. And similarly for namespace B.

How do I configure grafana agent to do this?

Do I need multiple grafana agent demonsets?

Do I have to configure one ebpf component per namespace/tenant?

[hainenber]

I mean it appears that Pyroscope server can readily support multi-tenancy if clients sending X-Scope-OrgId header

Your config works fine as is but you might need to turn on multitenancy_enabled on Pyroscope's server side. That's what I'm trying to convey

[towolf]

We do have multitenancy enabled on the Pyroscope server!

But grafana-agent is always setting the same X-Scope-OrgId header when sending!

How can we make it send different headers depending on for example namespace labels?

[hainenber]

I think I've got your issue now. The current pyroscope.write component can only send static value of X-Scope-OrgId as of now, afaik 😭

Long term solution should be implementing Loki's multi-tenancy to Pyroscope's.

Short term solution, is

Do I have to configure one ebpf component per namespace/tenant?

Your idea here, couple with K8s discovery should be the unblocker to the issue at hand here.

[towolf]

Do I have to configure one ebpf component per namespace/tenant?

Your idea here, couple with K8s discovery should be the unblocker to the issue at hand here.

That's what I reckoned. But wouldn't this have performance implications if we duplicate the ebpf component multiple times, wouldn't the ebpf configs stack in their overhead?

[hainenber]

hi there 👋 , I missed your reply. Yes, it would have performance implications but until there's a proper implementation, I think of above approach as a way to unblock your works at hands.