Getting unable to parse errors

[charlesverdad]

Thanks for your work in getting jfr ingest support for Pyroscope.

I'm trying out that feature but I can't get the parser to work.

cat myjfr.jfr | curl --data-binary @- -X POST 'localhost:4040/ingest?name=foo'

I get an error that's coming from the parser.

time="2022-03-29T04:32:27.239297" level=error msg="error happened while parsing request body" file=" server/controller.go:559" error="unable to parse JFR format: unable to parse chunk: unable to parse chunk metadata: unable to parse metadata event's string: Unsupported string type :4"

I have two different jfr files: one created by Java Flight Recorder (from-jfr.jfr), and one created by libasyncprofiler (from-ap.jfr) with the following flags: event=itimer,lock,alloc,loop=60s.

Here are the error messages:

Attempting to parse Java Flight Recorder output

 ~/work/jfr-parser  main   6s 
15:33:33 ❯ go run . ~/from-jfr.jfr
2022/03/29 15:33:48 Unable to parse: unable to parse chunk: unable to parse chunk metadata: unable to parse metadata event's string: Unsupported string type :4
exit status 1

Attempting to parse Async Profiler output

 ~/work/jfr-parser  main  
15:33:48 ❯ go run . ~/from-ap.jfr
2022/03/29 15:34:13 Unable to parse: unable to parse chunk: unable to parse checkpoint event: unable to parse constant type 31: unknown type profiler.types.LogLevel
exit status 1

Note this error unable to parse constant type 31: unknown type profiler.types.LogLevel is different from the first one. I'm more interested in this async-profiler output.

[abeaumont]

Thanks for reporting the problems @charlesverdad!

Both problems are expected, as the JFR support is still partial. Pending work has a summary of the missing parts. In the cases you are reporting:

• The first problem is an unsupported string encoding, we currently don't support type 4.
• The second problem is due to unsupported types. We currently don't support some of the most recently introduced types by async-profiler and the lock-related event types, thus the error.

Fortunately, both are relatively simple to fix. Adding lock event support in Pyroscope requires some additional work, but we can add support for these types and encodings at the parsing level so that Pyroscope can at least ingest them fine.

Could you attach the generated JFRs? They would be pretty helpful to validate everything works fine.

[charlesverdad]

Could you attach the generated JFRs? They would be pretty helpful to validate everything works fine.

@abeaumont Sure thing! Here you go: jfr_archive.zip

In the archive is:

• from-jfr.jfr : output of Java Flight Recorder
• from-jfr.json : parsed using JMC's jfr print --json tool
• from-ap.jfr: output of libasyncProfiler
• from-ap.json : parsed using JMC's jfr print --json tool

The java program I profiled can be found here https://github.com/pyroscope-io/pyroscope/blob/main/examples/java/Main.java

---

Some other findings

unable to parse constant type 31: unknown type profiler.types.LogLevel exit status 1

I'm not entirely sure why this happens. If I look at the json file parsed by jfr tool, I don't see any Loglevel strings (that is, cat from-ap.json | grep Loglevel has no results). I assume it is not important - The profiler.types.LogLevel event type is probably emitted by the profiler https://github.com/jvm-profiling-tools/async-profiler/blob/master/src/flightRecorder.cpp#L1317 and so it is not necessary to parse it.

Below is the list of types in the jfr, parsed by jfr print --json.

13:34:51 ❯ cat from-ap.json | jq '.recording.events[] | {type}' | sort | uniq
  "type": "jdk.ActiveRecording"
  "type": "jdk.ActiveSetting"
  "type": "jdk.CPUInformation"
  "type": "jdk.CPULoad"
  "type": "jdk.ExecutionSample"
  "type": "jdk.InitialSystemProperty"
  "type": "jdk.JVMInformation"
  "type": "jdk.NativeLibrary"
  "type": "jdk.OSInformation"
  "type": "jdk.ObjectAllocationInNewTLAB"

Note in the above, jdk.NativeLibrary is not in the supported event_types https://github.com/pyroscope-io/jfr-parser/blob/main/parser/event_types.go#L9-L20

[abeaumont]

Thanks for the output files in both formats, they are really useful!

I'm not entirely sure why this happens. If I look at the json file parsed by jfr tool, I don't see any Loglevel strings (that is, cat from-ap.json | grep Loglevel has no results). I assume it is not important - The profiler.types.LogLevel event type is probably emitted by the profiler https://github.com/jvm-profiling-tools/async-profiler/blob/master/src/flightRecorder.cpp#L1317 and so it is not necessary to parse it.

LogLevel event looks like a custom event by async-profiler, not a standard JFR type, and thus, it's not supported by standard JFR tooling, but they can gracefully ignore unknown types without failing. I have added the same kind of support to our parser in #5, and also added support for jdk.NativeLibrary event type (this is not needed for Pyroscope once the support to ignore unknown event types is there, but useful for a generic JFR parser).

The Java Flight Recorder output file shows some additional problems that I'll address in a separate pull request.

[charlesverdad]

Thanks @abeaumont . I checked out that branch and I can confirm that it works for the async-profiler output 😄 . I even tried it out with the Pyroscope server and it seemd to work without any problems.

[abeaumont]

With the latest improvements in the parser, both provided JFR files should be properly parsed now :)