jlfwong / speedscope

🔬 A fast, interactive web-based viewer for performance profiles.
https://www.speedscope.app
MIT License
5.59k stars 247 forks source link

Support importing profiles whose contents exceed V8s maximum string size #385

Closed jlfwong closed 2 years ago

jlfwong commented 2 years ago

Browsers have a limit on how big you can make strings. In Chrome on a 64 bit machine, this is around 512MB, which explains why in #340, a 600MB file fails to load.

To work around this issue, we avoid making strings this large.

To do so, we need two core changes:

  1. Instead of sending large strings as the import mechanism to different file format importers, we introduce a new TextFileContent interface which exposes methods to get the lines in the file or the JSON representation. In the case of line splitting, we assume that no single line exceeds the 512MB limit.
  2. We introduce a dependency on https://github.com/evanw/uint8array-json-parser to allow us to parse JSON files contained in Uint8Array objects

To ensure that this code doesn't code rot without introducing 600MB test files or test file generation into the repository, we also re-run a small set of tests with a mocked maximum string size of 100 bytes. You can see that the chunked string representation code is getting executed via test coverage.

Fixes #340

coveralls commented 2 years ago

Coverage Status

Coverage increased (+0.6%) to 41.664% when pulling 5f851a437c1e6733b03a4d9a293e037e8e92ecc3 on jlfwong/big-string into e37f6fa7c38c110205e22081560b99cb89ce885e on main.