Agent consistently fails to send data with error "sending on a closed channel"

[vorporeal]

Describe the bug you encountered:

I'm trying out using pyroscope-rs with the pprof-rs backend to profile my MacOS application. It works very intermittently - I rarely get any data showing up in my local Pyroscope server, and see Failed to send event to listener @ Sender { .. } - sending on a closed channel in the console every 10s.

What did you expect to happen instead?

I expected data to consistently get sent to Pyroscope.

How did you install pyroscope-rs?

Used brew to install the Pyroscope server, and set up pyroscope-rs as suggested in the documentation.

---

pyroscope-rs version and environment

MacOS 12.3, M1 Pro (aarch64), latest versions of the pyroscope and pyroscope_pprofrs crates, rustc 1.61.

[omarabid]

@vorporeal If you are able to provide code to reproduce this issue, that would be a good start. Otherwise, does it output any error message before failing the next round (the agent will send a request every 10s, my guess is something failed in between)?

[purkhusid]

I had the exact same issue when trying to add pyroscope to a service I wanted to profile. Unfortunately I did not find a way to minimally reporoduce it.

[omarabid]

@purkhusid Can you provide more details? OS, Rust Version, any logs that are output, application/service you are trying to profile, etc...

[omarabid]

@vorporeal @purkhusid Closed as can't reproduce. Please re-open if you can provide more information.

[Ronald-TR]

I had this issue too. I'm using MacOs M1. I installed using brew install pyroscope and run the server with pyroscope server.

I'm trying to use pyroscope to profile a grpc server that is built with tokio and tonic.

The code is very simple, is common a setup for a grpc server with tonic.

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let agent = PyroscopeAgent::builder("http://localhost:4040", "my_server")
        .tags([("Host", "Rust")].to_vec())
        .backend(pprof_backend(PprofConfig::new().sample_rate(100).report_thread_id().report_thread_name()))
        .build()?;
    let agent_running = agent.start().unwrap();

    // these signals are used to handle sigterm and sigint to do a graceful shutdown.
    let mut terminal_signals = Signals::new(&[SIGTERM, SIGINT]).unwrap();
    Server::builder()
        .add_service(my_server)
        .serve_with_shutdown(addr, async move {
            select! { _ = terminal_signals.next() => ()}
        })
        .await?;

        // stop pyroscope
    let agent_ready = agent_running.stop().unwrap();
    agent_ready.shutdown();

    Ok(())
}

Running this code I have the warning Failed to send event to listener @ Sender { .. } - sending on a closed channel and my application crashes instantly in sequence.

Any ideas how to use pyroscope to profile a grpc server?