Support for receiving profiles

[AndreZiviani]

Request

Would be really useful to be able to receive profiles on the agent, something similar to loki.source.api

Use case

The main reasons for me are:

• Pushing profiles to an agent on our infrastructure is faster (lower latency) than to Grafana Cloud via internet, less likely to impact our application
• Don't require devs to add infrastructure logic on the instrumentation, e.g. Grafana Cloud authentication / Tenant authentication, metadata labels (kubernetes labels, business labels, ....)
• Allows more flexibility on sampling

[ptodev]

Hi @AndreZiviani, thank you for raising an issue! Aer you referring to Pprof profiles? Is there a reason why pyroscope.scrape would not be sufficient for the purposes listed above?

[AndreZiviani]

Aer you referring to Pprof profiles?

I'm not sure, I want to be able to instrument my applications with the official Pyroscope libraries (not sure if it sends the profiles as pprof or something else) and send them to grafana agent. I forgot to mention Pyroscope on the original message :man_facepalming:

Is there a reason why pyroscope.scrape would not be sufficient for the purposes listed above?

Pyroscope libraries does not expose an endpoint for collection, it is only push-mode. I could look for other libraries but would like to stick with the official ones if possible

[ptodev]

I spoke with the Pyroscope team and they agree that this is a very valid request. The demand for this is likely to grow over time, so it is good to keep the issue open, but we cannot commit to when this will be implemented yet.

[AndreZiviani]

ok, thanks for the followup @ptodev

I guess its worth mentioning the future profiling support on opentelemetry, could be a long time before that's approved/implemented but I think its important to consider when implementing either/both options on the agent

[github-actions[bot]]

This issue has not had any activity in the past 30 days, so the needs-attention label has been added to it. If the opened issue is a bug, check to see if a newer release fixed your issue. If it is no longer relevant, please feel free to close this issue. The needs-attention label signals to maintainers that something has fallen through the cracks. No action is needed by you; your issue will be kept open and you do not have to respond to this comment. The label will be removed the next time this job runs if there is new activity. Thank you for your contributions!

[fede843]

I am looking forward this feature too! We are deploying an instance of Grafana Agent per server to gather, process/filter (as much as possible at that instance) and proxy all observability signals so far, and profiling is not fitting there yet. It is true that already existing scrape config might be enough for many cases, but not all. We are trying to link traces to profiles, using Java, and we need the jar agent, at least for now. It would be interesting to see how this develops in the future, along the adoption profiling into OTEL. That is the key I think.

[craigbehnke]

I was trying to deploy Pyroscope on my dotnet services in a k8s cluster, but had to abandon the idea because of this issue. For context, I use a service mesh where the only service authorized to communicate with *.grafana.net is the alloy deployment and not any first-party service.

It also looks like this capability (or similar) has been requested multiple times: Pyroscope forwarding through Alloy https://github.com/grafana/alloy/issues/984 Add push endpoint for jfr/pprof pyroscope/phlare protocol https://github.com/grafana/alloy/issues/395

Is there any roadmap or update on when this feature could be introduced?

[ptodev]

Hello! Both the Alloy and the Pyroscope teams would welcome the addition of such a receiver component. However, it is not on our roadmap and I can't guarantee if (and when) it would be prioritised. If anyone is interested in submitting a PR for it, please feel free to do so! We would be happy to review it.

I don't think OTel would impact the addition of this component. It'd be good to have a Pyroscope receiver component in Alloy even if there is a native OTel receiver for profiling. It could be useful for Pyroscope-instrumented applications. Not everyone may want to (or be able to) move to the OTel profiling instrumentation.

The future receiver component will most likely forward the profiles to pyroscope.write. It'll have to accept them as if it were Pyroscope's "ingest" HTTP endpoint.

It would be interesting look into scalability issues:

• Would it be ok to have multiple Alloy instances, each sending to Pyroscope? Could this cause any out of order errors in the same way as may do for Prometheus metrics?
• Can we configure the in-memory buffer in any way? E.g. should it drop data if it gets too large?
• Should there be any form of rate limiting? Like the OTel memory limiter processor?

[marcsanmi]

Hi everyone,

From the Pyroscope team, I wanted to provide a status update on this (thanks to the Alloy team for your help). Work has already begun, and we're aiming to have this completed by mid-November.

Here's a brief overview of what we're planning:

• We'll create a new pyroscope receiver component that will accept profiles via an /ingest endpoint, keeping the format SDKs are currently using from the Ingest API.
• This receiver will forward received profiles to pyroscope.write in their original format, without converting to pprof.
• We'll extend the existing pyroscope.write component to implement the Pyroscope Ingest API. This will allow it to support sending both push and ingest requests.
• The profiles will be sent to the Pyroscope /ingest endpoint in their original format.

This approach essentially acts as a proxy, maintaining simplicity in the user experience by using the same Alloy capsule as the current one exported by pyroscope.write.

We believe this solution will address the needs expressed in this issue, allowing for more flexible profile ingestion while maintaining compatibility with existing Pyroscope-instrumented applications.