Failed to load file(linux perf script) with size lager than 600MB

[etaf]

Failed to load file(linux perf script) with size lager than 600MB, thrim the size to around 500MB works fine. Is this a bug or by design ?

[jlfwong]

Hi @etaf!

How does it fail to load? Does it crash the browser tab? Does it hang forever? Does it print something in the console?

[bjorn3]

Try looking in the browser console. There may be an error about a too big object or something like that.

[etaf]

Hi @etaf!

How does it fail to load? Does it crash the browser tab? Does it hang forever? Does it print something in the console?

@jlfwong Thanks for your reply :) The popup dialog shows "Unrecognized format! See documentation about supported formats." And the console prints: Importing as output of linux perf script

But After trim the file size to around 500MB it works perfect.

I thinks the content read from local file is abnormal when the file size is too large. And so the linux perf content is invalid.

My upload file name is example.linux-perf.txt And the content is valid linux perf script output format.

[etaf]

Try looking in the browser console. There may be an error about a too big object or something like that.

@bjorn3 Thanks for your reply. The popup dialog shows "Unrecognized format! See documentation about supported formats." And the console only prints: Importing as output of linux perf script, no other error information.

[jlfwong]

Hmm, unfortunately I don't think I can investigate any further without a copy of the file (and getting a 600MB file from you might be a big pain).

If you can write a simple script which generates either a 500MB file that's valid or a 600MB file that's invalid where the contents are the same except with a different number of repetitions, then I might be able to debug further.

It's possible that this is hitting a browser allocation limit, though if that was the case I would expect to see an error like "Allocation failed" or something along those lines in the browser console.

[etaf]

Hmm, unfortunately I don't think I can investigate any further without a copy of the file (and getting a 600MB file from you might be a big pain).

If you can write a simple script which generates either a 500MB file that's valid or a 600MB file that's invalid where the contents are the same except with a different number of repetitions, then I might be able to debug further.

It's possible that this is hitting a browser allocation limit, though if that was the case I would expect to see an error like "Allocation failed" or something along those lines in the browser console.

@jlfwong I've write a short python script as follow to generate the file :

with open("/tmp/example.linux-perf.txt", "w") as f:
    t = 00092.020421
    for i in xrange(300000):
        t+=30092.020421
        f.write("""Thread-example   2398 [000] {}:                247989 cpu-cycles:
    fd2678327564        libexample.so[+363564] (/data/app/com.a.to-0AZ8LUqpOd83Gou--PLoOQ==/lib/arm64/libexample.so)
    fd267825fc7a        libexample.so[+29bc7a] (/data/app/com.a.to-0AZ8LUqpOd83Gou--PLoOQ==/lib/arm64/libexample.so)
    fd26783e7bde        libexample.so[+423bde] (/data/app/com.a.to-0AZ8LUqpOd83Gou--PLoOQ==/lib/arm64/libexample.so)
    fd26783e6d12        libexample.so[+422d12] (/data/app/com.a.to-0AZ8LUqpOd83Gou--PLoOQ==/lib/arm64/libexample.so)
    fd26784f9806        libexample.so[+535806] (/data/app/com.a.to-0AZ8LUqpOd83Gou--PLoOQ==/lib/arm64/libexample.so)
    fd26784f970e        libexample.so[+53570e] (/data/app/com.a.to-0AZ8LUqpOd83Gou--PLoOQ==/lib/arm64/libexample.so)
    fd26784f976e        libexample.so[+53576e] (/data/app/com.a.to-0AZ8LUqpOd83Gou--PLoOQ==/lib/arm64/libexample.so)
    fd26781b643a        libexample.so[+1f243a] (/data/app/com.a.to-0AZ8LUqpOd83Gou--PLoOQ==/lib/arm64/libexample.so)
    fd26783f00b2        libexample.so[+42c0b2] (/data/app/com.a.to-0AZ8LUqpOd83Gou--PLoOQ==/lib/arm64/libexample.so)
    fd26781b5fb6        libexample.so[+1f1fb6] (/data/app/com.a.to-0AZ8LUqpOd83Gou--PLoOQ==/lib/arm64/libexample.so)
    fd26784f96fa        libexample.so[+5356fa] (/data/app/com.a.to-0AZ8LUqpOd83Gou--PLoOQ==/lib/arm64/libexample.so)
    fd26784f976e        libexample.so[+53576e] (/data/app/com.a.to-0AZ8LUqpOd83Gou--PLoOQ==/lib/arm64/libexample.so)
    fd26784c7a9a        libexample.so[+503a9a] (/data/app/com.a.to-0AZ8LUqpOd83Gou--PLoOQ==/lib/arm64/libexample.so)
    fd26783e17d6        libexample.so[+41d7d6] (/data/app/com.a.to-0AZ8LUqpOd83Gou--PLoOQ==/lib/arm64/libexample.so)
    fd2678257602        libexample.so[+293602] (/data/app/com.a.to-0AZ8LUqpOd83Gou--PLoOQ==/lib/arm64/libexample.so)
    fd2678407b96        libexample.so[+443b96] (/data/app/com.a.to-0AZ8LUqpOd83Gou--PLoOQ==/lib/arm64/libexample.so)
    fd267814ca5e        libexample.so[+188a5e] (/data/app/com.a.to-0AZ8LUqpOd83Gou--PLoOQ==/lib/arm64/libexample.so)
    fd267814cc4a        libexample.so[+188c4a] (/data/app/com.a.to-0AZ8LUqpOd83Gou--PLoOQ==/lib/arm64/libexample.so)
    fd26781459ca        libexample.so[+1819ca] (/data/app/com.a.to-0AZ8LUqpOd83Gou--PLoOQ==/lib/arm64/libexample.so)
    fd2695ac4196        __pthread_start(void*) (/system/lib64/libc.so)
    fd2695a643be        __start_thread (/system/lib64/libc.so)

""".format(t))

Speed scope failed to load the generated file. And change the iteration from 300000 to 200000 will generate file works with speedscope.

[jlfwong]

@etaf Awesome! This should be actionable now. I don't have any specific timeline for when I'll be able to look into this, but I think that should be all I need from you to investigate.

[alberdingk-thijm]

I also was unable to load a really big file (1.1GB) in speedscope. I generated the file from perf record --call-graph=dwarf on an OCaml binary that ran for 75 seconds and did quite a bit of work (perf collected over 300,000 samples); I then converted that perf data into a profile script using perf script -i, as described in the wiki.

It looks like the source of the problem was writing the buffer of the file to a string:

Error: Cannot create a string longer than 0x1fffffe8 characters
    at Object.slice (node:buffer:642:37)
    at Buffer.toString (node:buffer:811:14)
    at main (/usr/lib/node_modules/speedscope/bin/cli.js:69:39)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

This happens when speedscope is loading the file buffer into memory to then process using speedscope <filename>. 0x1fffffe8 is 512MB, so it seems like as currently implemented, speedscope has a hard cap in file size at 512MB, which is why a 600MB file wouldn't be usable. Perhaps there is a way to read the buffer in chunks, and then allow a larger file to be processed (assuming a user has enough memory to store such a file)?

[GGG-KILLER]

Ran into this issue while trying to load a 581 MiB file. The website just errors with an unrelated error since MaybeCompressedDataReader.readAsText() just returns an empty string when called from _importProfileGroup. The CLI tool errors with:

Error: Cannot create a string longer than 0x1fffffe8 characters
    at Object.slice (node:buffer:640:37)
    at Buffer.toString (node:buffer:809:14)
    at main (C:\Users\gusta\AppData\Roaming\npm\node_modules\speedscope\bin\cli.js:69:39)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)