Profiling multiple event types concurrently

dmos62 commented 3 years ago

async-profiler has support for multiple concurrent event types, but only when using the Java Flight Recorder (JFR) output format. At the moment this Pyroscope integration uses the "collapsed call traces" format, which is also a format accepted by Pyroscope's HTTP API. What this means is that unless other output formats for multiple concurrent event types gain support upstream, this integration has to dump and parse JFR, and then has to split the parsed JFR dump into multiple Snapshots (one per event type, since event type options are specified per Snapshot), which includes coercing the JFR format into a format accepted by Pyroscope, such as the "collapsed call traces" format currently used.

To sum up, steps needed to implement this are:

dump JFR;
parse JFR: not clear how at the moment: if the AsyncProfiler.execute command dumps in string or byte format, there's scripts in async-profiler for converting JFR files to some other formats that we could use for reference;
introduce dedicated queue-snapshot-upload pipeline for each concurrently profiled event type;
format resulting sample data into a format understood by Pyroscope.

netamego commented 2 years ago

Hi, Profile cpu, wall-clock and lock at same time is a very important feature provide by async-profiler and not supported in pyroscope. Maybe the work done from @krzysztofslusarski collapse-jfr can be harnessed. Thanks.

krzysztofslusarski commented 2 years ago

For the record, you cannot profile both cpu and wall-clock at the same moment. You can profile wall-clock with JFR output. In the JFR you have a thread state, so my tool simply takes the stacktraces with thread state ACTIVE and dumps them to a separate file. That should be almost the same as CPU.

The difference is that you have no kernel frames in wall mode, so my tool cannot provide them either. You can profile wall,lock,alloc or cpu,lock,alloc, or any subset of those.

The parser of JFR is now opensourced:

        <dependency>
            <groupId>org.openjdk.jmc</groupId>
            <artifactId>flightrecorder</artifactId>
            <version>8.1.0</version>
        </dependency>

You can see in my project how to use it's API.

netamego commented 2 years ago

Hi,

As a workaround you can obtain events in jfr format, convert it to collapsed (with collapse-jfr) and then upload to Pyroscope. This a awful code (I know) but works for me. You have itimer,wall and lock at same time.

#!/bin/bash

# In java app you must start async-profiler in next way:
#  -agentpath:/dir/libasyncProfiler.so=start,event=wall,lock,loop=10s,file=/opt/async-profiler/jfr/myserv.myapp.interval.%t.jfr
#  for example:
#  -agentpath:/opt/async-profiler/build/libasyncProfiler.so=start,event=wall,lock,loop=10s,file=/opt/async-profiler/jfr/lnxhost1.jenkins.100.%t.jfr

pyroscopeserv="myserver:4040"
basedir="/opt/async-profiler"
jfrfiles="${basedir}/jfr"
jfrerr="${basedir}/jfrerr"
collapsedfiles="${basedir}/collapsedfiles"

if [ ! -d $collapsedfiles ] ; then
 mkdir $collapsedfiles
fi

if [ ! -d $jfrerr ] ; then
 mkdir $jfrerr
fi

cd $collapsedfiles

while true ; do

 if [ ! -n "$(find $jfrfiles -prune -empty)" ] ; then

   for jfr in `find $jfrfiles -type f -mmin +1 -exec basename {} \;` ; do

      control=0
      start=$(stat ${jfrfiles}/${jfr} | grep 'Access: [0-9].*' | awk '{print $2 " " $3}')
      end=$(stat ${jfrfiles}/${jfr} | grep 'Modify: [0-9].*' | awk '{print $2 " " $3}')
      startepoch=$(date -d "${start}" +"%s")
      endepoch=$(date -d "${end}" +"%s")
      convert=$(java -jar ${basedir}/collapse-jfr-full.jar -f ${jfrfiles}/${jfr} 2>&1)

      if [[ ! "${convert}" =~ "at" ]] ; then

        host=$(echo $jfr | cut -f 1 -d '.')
        app=$(echo $jfr | cut -f 2 -d '.')
        interval=$(echo $jfr | cut -f 3 -d '.')

        for collapsedfile in `ls $collapsedfiles` ; do

          profiling=$(echo $collapsedfile | cut -f 1 -d '.')
          httpresponse=$(zcat $collapsedfiles/$collapsedfile | grep -v Unsafe_Park | curl -fsS --write-out %{http_code} --noproxy '*' -X POST --data-binary @- "http://${pyroscopeserv}/ingest?name=${host}.${app}.${profiling}&units=samples&aggregationType=sum&sampleRate=${interval}&from=${startepoch}&until=${endepoch}&spyName=javaspy")

          if [ ! $httpresponse -eq 200 ] ; then
            control=1
          fi

          rm  $collapsedfiles/$collapsedfile

        done

      fi

    if [ $control -eq 0 ] ; then
      rm -f ${jfrfiles}/${jfr}
    else
      mv ${jfrfiles}/${jfr} $jfrerr
    fi

  done

 fi

sleep 10

done

Then you can put it as service.

[Unit]
Description=Java Profiling Pyroscope

[Service]
User=myuser
ExecStart=/opt/async-profiler/pyroscope.sh

[Install]
WantedBy=multi-user.target

Hopefully the pyroscope team will implement multiple events at some point.

abeaumont commented 2 years ago

We have added initial support for multiple concurrent event types in #22 and #23 (also https://github.com/pyroscope-io/pyroscope/pull/954 and https://github.com/pyroscope-io/pyroscope/pull/961), enabling cpu (one of cpu, itimer, wall events) and memory (alloc) profiling concurrently (lock profiling is yet pending). This is built on top of our own JFR parser, integrated into Pyroscope.

netamego commented 2 years ago

Wooow this is very good news!!! Thank you very much @abeaumont for your work and effort in this integration.

netamego commented 2 years ago

Hi,

I have been testing multievent support. Seems very promising.

Could be possible to add support to wall and cpu event at same time? At this moment we need to choose between cpu/itimer or wall but I think is important this two events are supported simultanoesly.

async-profiler supports that in jfr format.

Thanks a lot for your great work!!!

Best regards.

krzysztofslusarski commented 2 years ago

You cannot do it with async-profiler. Either wall or cpu. But you can use wall with jfr format and in the output you have a thread state. that's how my tool generates CPU collapsed file from wall jfr.

https://github.com/krzysztofslusarski/collapse-jfr/blob/650f28c6d3379c5ea5ec132b06f670f3a611b4f0/src/main/java/pl/ks/profiling/jft/converter/collapsed/JftToCollapseStacks.java#L269

The only big difference I notices is lack of kernel frames in wall mode.

abeaumont commented 2 years ago

Right, as @krzysztofslusarski says, only one cpu-like (CPU, wall, itimer) event is supported. But that's a nice trick you got there, I guess we could do something similar when wall is chosen as the event type :ok_hand:

netamego commented 2 years ago

Hi,

Understood. My mistake.

The challenge will be not to treat it by independent threads as collapse-jfr does. In large applications with thousands of threads, the flamegraph would be unmanageable.

Thanks a lot.

Best regards.

netamego commented 2 years ago

Hi,

Yes as @apangin in https://github.com/jvm-profiling-tools/async-profiler/issues/573 and @krzysztofslusarski said:

"Note that cpu and wall cannot be specified together in -e option. Only one type of an execution event at a time is supported. However, a wall-clock profile includes thread states, so it's possible to extract a CPU profile (samples in a running state only) from a wall-clock output"

Would be great to implement this in pyroscope java-agent.

Thanks a lot and much encouragement. You are doing an incredible job.

Best regards!!!

abeaumont commented 2 years ago

Support to generate both CPU and wall profile data when wall event is used has been added in https://github.com/pyroscope-io/pyroscope/pull/1002

netamego commented 2 years ago

Amazing!!! Thank you very much.

abeaumont commented 2 years ago

Lock profiling support has been added in #33. With this the support for multiple event types is complete!

grafana / pyroscope-java

Profiling multiple event types concurrently #3