search

catapult-project / catapult

Deprecated Catapult GitHub. Please instead use http://crbug.com "Speed>Benchmarks" component for bugs and https://chromium.googlesource.com/catapult for downloading and editing source code..
https://chromium.googlesource.com/catapult
BSD 3-Clause "New" or "Revised" License
1.91k stars 562 forks source link

traceviewer: how to handle large traces? #4625

Open milianw opened 5 years ago

milianw commented 5 years ago

Hello!

I'm trying to convert LTTng trace points to the Chrome Trace Format and then load the data into chrome. The LTTng trace data is roughly ~100MB in its binary CTF (common trace format) representation. The JSON version of that easily takes four times as much and doesn't yet contain all information - mostly just the events names and timestamps without any additional optional metadata. That already is impossible to load, the tab gets killed once it reaches ~3GB of memory or so.

I saw that you seem to be using catapult for Android ftrace data, which should easily generate as much data as I'm seeing here with LTTng. How do you handle this?

milianw commented 5 years ago

a ~160MB binary LTTng trace gets converted to ~2.68GB of JSON text which then also completely fails to load:

While importing:
RangeError: Invalid typed array length: 2682325620
    at new Uint8Array (<anonymous>)
    at Object.flattenChunks (chrome://tracing/tracing.js:4696:6572)
    at i.onEnd (chrome://tracing/tracing.js:4696:5526)
    at i.push (chrome://tracing/tracing.js:4696:5263)
    at n (chrome://tracing/tracing.js:4696:3781)
    at Object.a.inflateRaw (chrome://tracing/tracing.js:4696:5679)
    at Function.GzipImporter.inflateGzipData_ (chrome://tracing/tracing.js:4710:137)
    at GzipImporter.extractSubtraces (chrome://tracing/tracing.js:4719:240)
    at chrome://tracing/tracing.js:1323:66
    at Task.run (chrome://tracing/tracing.js:2313:95)

could one use something more efficient than JSON for the trace format here? best would be something that's streamable and doesn't have a tree structure like JSON