[Request] Async profiler support

[grzesuav]

Describe the feature

Idea - support for https://github.com/async-profiler/async-profiler besides just jfr

Anything other information?

No response

[andrewazores]

Previous request on the old repo: https://github.com/cryostatio/cryostat-legacy/issues/1456

[grzesuav]

Got it, so options are (based on https://github.com/async-profiler/async-profiler/discussions/496#discussioncomment-1614455):

1. detect if async profiler is present in target JVM - requires async profiler being added on target application, similar to cryostat agent static attachement
2. from an external process using dynamic attach -something similar to dynamic attach of cryostat agent
3. within a Java application using Java API - not sure if this is feasible, maybe it can be integrated into cryostat agent ?
4. via JMX - not sure how it differs from 3. :thinking:

[grzesuav]

actually found similar discussion in https://github.com/cryostatio/cryostat-agent/pull/234#issue-1946139083

[grzesuav]

and a bit https://github.com/cryostatio/cryostat-agent/issues/381

[grzesuav]

Regarding loading an agent into target pod - one option would be to use ephemeral container with the agent image. https://kubernetes.io/docs/tasks/debug/debug-application/debug-running-pod/#ephemeral-container

[andrewazores]

I don't think those other discussions are particularly close to this one. The first one has to do with the ability to attach the Cryostat Agent to an already-running JVM, rather than requiring the user to configure the JVM with -javaagent and restart the deployment. The second one is mostly about automating the first part.

Supporting async profiler would mean need to detect if it's present, be able to collect that data, and be able to send that data to the Cryostat server. Then the Cryostat server needs to have API endpoints for accepting and storing this data and providing a way for users to download it (and even to request it to be collected over JMX), and there needs to be corresponding work in cryostat-web to create a UI for these actions.

If you use the AsyncProfiler MXBean as in the other discussion thread you linked, then that along with Hawt.io sounds like it could be a baseline to getting something working. I think you would still be dumping async profiler files to local container filesystem, and need some other way to extract that, so this is where some integration with the Cryostat Agent might come into play. It won't be plug-and-play however, it will need some development effort across various Cryostat components.

---

The ephemeral container part looks useful to go along with the Cryostat Agent container image. @ebaron has some work underway that is along those lines of making it simpler to get the Cryostat Agent attached to application workloads, but I don't think the solution is using ephemeral containers (at least not currently).

[grzesuav]

I don't think those other discussions are particularly close to this one. The first one has to do with the ability to attach the Cryostat Agent to an already-running JVM, rather than requiring the user to configure the JVM with -javaagent and restart the deployment. The second one is mostly about automating the first part.

got it, for me it was relevant as somehow async-profiler need to be attached to target VM.

Supporting async profiler would mean need to detect if it's present, be able to collect that data, and be able to send that data to the Cryostat server. Then the Cryostat server needs to have API endpoints for accepting and storing this data and providing a way for users to download it (and even to request it to be collected over JMX), and there needs to be corresponding work in cryostat-web to create a UI for these actions.

ok, thanks for clarifying that. I was also thinking that cryostat can handle loading/attaching the async-profiler into the target.

If you use the AsyncProfiler MXBean as in the other discussion thread you linked, then that along with Hawt.io sounds like it could be a baseline to getting something working. I think you would still be dumping async profiler files to local container filesystem, and need some other way to extract that, so this is where some integration with the Cryostat Agent might come into play. It won't be plug-and-play however, it will need some development effort across various Cryostat components.

got it, I am new to cryostat and I haven't been doing java for years that is why my understanding is bit chaotic 😅

[andrewazores]

got it, for me it was relevant as somehow async-profiler need to be attached to target VM.

Yes, it's a very similar problem, but mostly I think it just means that similar solutions need to be applied, not that the same solution would work for both -

ok, thanks for clarifying that. I was also thinking that cryostat can handle loading/attaching the async-profiler into the target.

- except with this idea. If somehow Cryostat automates the loading and dynamic attaching of async-profiler to the target JVM, then maybe it really would be the same solution. This is certainly an interesting idea for the future. Beyond the technical challenges we would also need to evaluate what this means for the downstream Cryostat product, because I don't think we would be able to offer meaningful support (in terms of development/QE support) for Async Profiler, so we need to be careful here about biting off more than we can chew.

I imagine this would look like:

1. Cryostat Agent becomes aware of Async Profiler somehow, ex. by looking for the Async Profiler MXBean being present in the JVM
2. Cryostat Agent learns how to grab Async Profiler files and push them to the Cryostat Server (similar or same functionality as its existing JFR Harvester), as well as expose some endpoints to allow remote management similar to it does for JFR
3. Cryostat (server) learns how to look for the Async Profiler over JMX, as well as query Agents over HTTP if Async Profiler is detected, so that profiling can be started/stopped/retrieved similar to what Cryostat does now with JFR
4. Cryostat Operator learns how to deploy a container containing the Async Profiler Agent JAR and get that to dynamically attach to the target JVM, very similar to the current work underway to do this for the Cryostat Agent JAR

got it, I am new to cryostat and I haven't been doing java for years that is why my understanding is bit chaotic 😅

No problem, you raise quite a few interesting points :-) I'm glad to have your thoughts and feedback, even if we can't always take immediate action on them.

[ebaron]

The ephemeral container part looks useful to go along with the Cryostat Agent container image. @ebaron has some work underway that is along those lines of making it simpler to get the Cryostat Agent attached to application workloads, but I don't think the solution is using ephemeral containers (at least not currently).

This could be an interesting way to do it. I'll have to verify if we can add volumes to an existing pod for things like certificates.