search

jlfwong / speedscope

🔬 A fast, interactive web-based viewer for performance profiles.
https://www.speedscope.app
MIT License
5.58k stars 247 forks source link

10 minutes loading a 65MB json file #399

Closed aurlio closed 8 months ago

aurlio commented 2 years ago

hi,

I used speedscope to display unity frame's stack. In most cases, everything works fine. But when the json file is large enough, it can take a very long time before the json file finally loaded.

image

the sample json file: https://1drv.ms/u/s!AoIp7o1UyX8uonvPjDXmlPsGEume?e=VzDaMv

Is it possilbe to speed up the file loading?

Thanks a lot.

jlfwong commented 11 months ago

Extremely late following up on this, but I think the problem is that the shared.frames array has 481623 elements, with huge amount of duplication. The point of the shared.frames array is for the entries in it to de-duplicate information that shows up repeatedly in stack traces.

For example, here's a partial slice of the names of the frames entry in the shared.frames property of the given file:

  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',
  'Canvas.RenderSubBatch(total:0.07ms, calls:157)',

All of these should be a single entry in the frames list. I'm not 100% sure, but I suspect if these were de-duplicated, then the trace would import much faster. Most problematically, it seems there are 470,004 separate entries in the frames list for GC.Alloc(total:13.26ms, calls:470004).

Was this using some standard kind of Unity profile exporter, or was this something custom you wrote?

aurlio commented 8 months ago

Thanks for your reply！ This is collected by standard unity profiler, but only specific frames have this issue. I guess the duplication is due to logic errors or bugs.

Our project has been shut down last year, so this problem doesn't bother me any more.

Jamie Wong @.***> 于2023年12月26日周二 12:11写道：

Extremely late following up on this, but I think the problem is that the shared.frames array has 481623 elements, with huge amount of duplication. The point of the shared.frames array is for the entries in it to de-duplicate information that shows up repeatedly in stack traces.

For example, here's a partial slice of the names of the frames entry in the shared.frames property of the given file:

'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)', 'Canvas.RenderSubBatch(total:0.07ms, calls:157)',

All of these should be a single entry in the frames list. I'm not 100% sure, but I suspect if these were de-duplicated, then the trace would import much faster.

Was this using some standard kind of Unity profile exporter, or was this something custom you wrote?

— Reply to this email directly, view it on GitHub https://github.com/jlfwong/speedscope/issues/399#issuecomment-1869239368, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGPFXH3SQCD2WS7IBSSUJDYLJE6TAVCNFSM5ZXSADHKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBWHEZDGOJTGY4A . You are receiving this because you authored the thread.Message ID: @.***>

jlfwong commented 8 months ago

If anyone else runs into this, please let me know!