search

josepdcs / kubectl-prof

kubectl-prof is a kubectl plugin to profile applications on kubernetes with minimum overhead
Apache License 2.0
48 stars 5 forks source link

flamegraphs for java with -e alloc and --interval don't download (Checksum does not match) error #27

Closed jlynch1 closed 9 months ago

jlynch1 commented 9 months ago

When I run with java async-profiler cpu events - all the flamegraphs are downloaded

--tool async-profiler -e cpu -l java -o flamegraph -t 2m --interval 60s

but with java async-profiler alloc events I get an error saying Checksum does not match

--tool async-profiler -e alloc -l java -o flamegraph -t 2m --interval 60s

Default profiling tool async-profiler will be used ... ✔
Verifying target pod ... ✔
Launching profiler ... ✔
Profiling ... ✔

Checksum does not match, retrying: /tmp/agent-flamegraph-1382909.html.gz...
Checksum does not match, retrying: /tmp/agent-flamegraph-1382909.html.gz...

with alloc events, when I remove --interval 60s a single flamegraph is produced and downloaded.

looks like a timing issue, I tried changing (perfDelayBetweenJobs) but it doesnt seem to introduce a delay at all as there is no delay between the jobs timestamps. I see the sleep in the code in ./internal/agent/profiler/jvm/async_profiler.go

{"type":"log","data":{"time":"2024-02-01T19:55:57.560460874Z","level":"debug","msg":"The target filesystem is: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9207e89a6d90b33dc9082d185a43c82a20c217e5b42bf42dd1dc33409829e9dc/rootfs"}}
{"type":"log","data":{"time":"2024-02-01T19:55:57.560837314Z","level":"debug","msg":"pgrep -P 3327888"}}
{"type":"log","data":{"time":"2024-02-01T19:55:57.580797179Z","level":"debug","msg":"pgrep -P 3328340"}}
{"type":"log","data":{"time":"2024-02-01T19:55:57.598762725Z","level":"debug","msg":"pgrep -P 3328373"}}
{"type":"log","data":{"time":"2024-02-01T19:55:57.615486974Z","level":"debug","msg":"The PIDs to be profiled: [3328373]"}}
{"type":"log","data":{"time":"2024-02-01T19:55:57.615520289Z","level":"debug","msg":"cp -r /app/async-profiler /tmp"}}
{"type":"log","data":{"time":"2024-02-01T19:55:57.618155249Z","level":"debug","msg":"/tmp/async-profiler/profiler.sh -o flamegraph -d 60 -f /tmp/agent-flamegraph-3328373.html -e alloc --fdtransfer 3328373"}}
{"type":"log","data":{"time":"2024-02-01T19:56:57.756881691Z","level":"debug","msg":"stat -c%s /tmp/agent-flamegraph-3328373.html.gz"}}
{"type":"result","data":{"time":"2024-02-01T19:56:57.757942168Z","result-type":"flamegraph","file":"/tmp/agent-flamegraph-3328373.html.gz","file-size-in-bytes":28507,"checksum":"2ddf9c65b8b12630eabfc2ffd6d9d61f","compressor-type":"gzip"}}
{"type":"log","data":{"time":"2024-02-01T19:56:57.758313068Z","level":"debug","msg":"/tmp/async-profiler/profiler.sh -o flamegraph -d 60 -f /tmp/agent-flamegraph-3328373.html -e alloc --fdtransfer 3328373"}}
{"type":"log","data":{"time":"2024-02-01T19:57:57.859065758Z","level":"debug","msg":"stat -c%s /tmp/agent-flamegraph-3328373.html.gz"}}
{"type":"result","data":{"time":"2024-02-01T19:57:57.860134251Z","result-type":"flamegraph","file":"/tmp/agent-flamegraph-3328373.html.gz","file-size-in-bytes":27931,"checksum":"385cf11bce4a902a073427e3b521f28b","compressor-type":"gzip"}}
{"type":"log","data":{"time":"2024-02-01T19:57:57.860312937Z","level":"debug","msg":"/tmp/async-profiler/profiler.sh -o flamegraph -d 60 -f /tmp/agent-flamegraph-3328373.html -e alloc --fdtransfer 3328373"}}
{"type":"log","data":{"time":"2024-02-01T19:58:57.961279147Z","level":"debug","msg":"stat -c%s /tmp/agent-flamegraph-3328373.html.gz"}}
{"type":"result","data":{"time":"2024-02-01T19:58:57.962588016Z","result-type":"flamegraph","file":"/tmp/agent-flamegraph-3328373.html.gz","file-size-in-bytes":34371,"checksum":"6673e8b7ae742ee567d29eb6dcc7d77c","compressor-type":"gzip"}}
{"type":"progress","data":{"time":"2024-02-01T19:58:57.962825035Z","stage":"ended"}}
{"type":"log","data":{"time":"2024-02-01T20:03:57.976570146Z","level":"warn","msg":"Maximum allowed time 5m0s surpassed. Cleaning up and auto-deleting the agent..."}}
{"type":"log","data":{"time":"2024-02-01T20:03:57.976643458Z","level":"debug","msg":"/tmp/async-profiler/profiler.sh stop 3328373"}}
{"type":"log","data":{"time":"2024-02-01T20:03:57.990292613Z","level":"debug","msg":"Trying to remove file: /tmp/agent-flamegraph-3328373.html"}}
{"type":"log","data":{"time":"2024-02-01T20:03:57.990424459Z","level":"debug","msg":"Trying to remove file: /tmp/agent-flamegraph-3328373.html.gz"}}

maybe if the filename had a timestamp or an interval counter rather than the same file filename agent-flamegraph-1382909.html for each interval, it would help. Also the perfRecordOutputFileName and perfScriptOutputFileName ?

josepdcs commented 9 months ago

Hi @jlynch1,

Checksum error can be experienced when some problem occurs downloading/copying the result file from the remote node. Maybe caused by an unexpected network problem.

I've unable to reproduce this case. I've been able to download all flamegraphs for any event type: alloc, cpu, itmer...

By the way, checksum is not related with manner of naming the result file.

killbulle commented 9 months ago

fyi experiment this also randomly on an aws cluster , independently of option , will dig more btw thks for this awesome tools !

jlynch1 commented 9 months ago

it happens for me every time I attempt to use alloc event with an interval

in retrieveFileOrRetry() I uncommented the debug line and added in filesize

-               //fmt.Printf("\nFile %s downloaded (local: %s | remote: %s)", remoteFile.FileName, checksum, remoteFile.Checksum)
+               fmt.Printf("\nFile %s downloaded (local: %s (%d bytes)| remote: %s (%d bytes))", remoteFile.FileName, checksum, len(fileBuff), remoteFile.Checksum, remoteFile.FileSizeInBytes)
File /tmp/agent-flamegraph-3328373.html.gz downloaded (local: 11678d5b6297b6e76b5b4eae90bdfda3 (36022 bytes)| remote: b032cfd534cb8a30e6a3c93eba2377a0 (36023 bytes))
Checksum does not match, retrying: /tmp/agent-flamegraph-3328373.html.gz...
File /tmp/agent-flamegraph-3328373.html.gz downloaded (local: 2fc118d516590948fbe14bd007fe79fa (72045 bytes)| remote: b032cfd534cb8a30e6a3c93eba2377a0 (36023 bytes))
Checksum does not match, retrying: /tmp/agent-flamegraph-3328373.html.gz...
File /tmp/agent-flamegraph-3328373.html.gz downloaded (local: e1d78beba971b5a7fe467d9365518b6d (108065 bytes)| remote: b032cfd534cb8a30e6a3c93eba2377a0 (36023 bytes))
Checksum does not match, retrying: /tmp/agent-flamegraph-3328373.html.gz...
File /tmp/agent-flamegraph-3328373.html.gz downloaded (local: a69629fbaea4c7f97934355cbd8ba4c6 (144087 bytes)| remote: b032cfd534cb8a30e6a3c93eba2377a0 (36023 bytes))
Checksum does not match, retrying: /tmp/agent-flamegraph-3328373.html.gz...❌
checksum does not match for file /tmp/agent-flamegraph-3328373.html.gz

and

type":"log","data":{"time":"2024-02-07T17:39:47.27282818Z","level":"debug","msg":"{\"Duration\":120000000000,\"Interval\":60000000000,\"UID\":\"e90460c6-a326-4a6b-bdfe-3b3bd0f8156f\",\"ContainerRuntime\":\"containerd\",\"ContainerRuntimePath\":\"/run/containerd\",\"ContainerID\":\"9207e89a6d90b33dc9082d185a43c82a20c217e5b42bf42dd1dc33409829e9dc\",\"PodUID\":\"f680eeab-f22b-4d8a-ae13-c3f76e8d85bd\",\"Language\":\"java\",\"Event\":\"alloc\",\"Compressor\":\"gzip\",\"Tool\":\"async-profiler\",\"OutputType\":\"flamegraph\",\"FileName\":\"\",\"HeapDumpSplitInChunkSize\":\"\",\"PID\":\"\",\"Pgrep\":\"\",\"AdditionalArguments\":null}"}}
{"type":"progress","data":{"time":"2024-02-07T17:39:47.27301402Z","stage":"started"}}
{"type":"log","data":{"time":"2024-02-07T17:39:47.273063807Z","level":"debug","msg":"The target filesystem is: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9207e89a6d90b33dc9082d185a43c82a20c217e5b42bf42dd1dc33409829e9dc/rootfs"}}
{"type":"log","data":{"time":"2024-02-07T17:39:47.273335554Z","level":"debug","msg":"pgrep -P 3327888"}}
{"type":"log","data":{"time":"2024-02-07T17:39:47.296582217Z","level":"debug","msg":"pgrep -P 3328340"}}
{"type":"log","data":{"time":"2024-02-07T17:39:47.315742591Z","level":"debug","msg":"pgrep -P 3328373"}}
{"type":"log","data":{"time":"2024-02-07T17:39:47.384352887Z","level":"debug","msg":"The PIDs to be profiled: [3328373]"}}
{"type":"log","data":{"time":"2024-02-07T17:39:47.384379558Z","level":"debug","msg":"cp -r /app/async-profiler /tmp"}}
{"type":"log","data":{"time":"2024-02-07T17:39:47.390612301Z","level":"debug","msg":"/tmp/async-profiler/profiler.sh -o flamegraph -d 60 -f /tmp/agent-flamegraph-3328373.html -e alloc --fdtransfer 3328373"}}
{"type":"log","data":{"time":"2024-02-07T17:40:47.463519037Z","level":"debug","msg":"stat -c%s /tmp/agent-flamegraph-3328373.html.gz"}}
{"type":"result","data":{"time":"2024-02-07T17:40:47.464583752Z","result-type":"flamegraph","file":"/tmp/agent-flamegraph-3328373.html.gz","file-size-in-bytes":36023,"checksum":"b032cfd534cb8a30e6a3c93eba2377a0","compressor-type":"gzip"}}
{"type":"log","data":{"time":"2024-02-07T17:40:47.464847254Z","level":"debug","msg":"/tmp/async-profiler/profiler.sh -o flamegraph -d 60 -f /tmp/agent-flamegraph-3328373.html -e alloc --fdtransfer 3328373"}}
{"type":"log","data":{"time":"2024-02-07T17:41:47.54847511Z","level":"debug","msg":"stat -c%s /tmp/agent-flamegraph-3328373.html.gz"}}
{"type":"result","data":{"time":"2024-02-07T17:41:47.549306282Z","result-type":"flamegraph","file":"/tmp/agent-flamegraph-3328373.html.gz","file-size-in-bytes":37935,"checksum":"7e9250b19d19ecf0cccec70016ab6e7c","compressor-type":"gzip"}}
{"type":"progress","data":{"time":"2024-02-07T17:41:47.549448578Z","stage":"ended"}}
{"type":"log","data":{"time":"2024-02-07T17:41:52.558181028Z","level":"debug","msg":"Received signal: terminated"}}
{"type":"log","data":{"time":"2024-02-07T17:41:52.558215876Z","level":"debug","msg":"/tmp/async-profiler/profiler.sh stop 3328373"}}
{"type":"log","data":{"time":"2024-02-07T17:41:52.579639461Z","level":"debug","msg":"Trying to remove file: /tmp/agent-flamegraph-3328373.html"}}
{"type":"log","data":{"time":"2024-02-07T17:41:52.579771066Z","level":"debug","msg":"Trying to remove file: /tmp/agent-flamegraph-3328373.html.gz"}}
{"type":"log","data":{"time":"2024-02-07T17:41:52.579909625Z","level":"debug","msg":"Profiling finished properly. Bye!"}}

So I see a few problems

josepdcs commented 9 months ago

I'll check it again. Seems you're right

jlynch1 commented 9 months ago

The only way I can see it worked for you, is if you downloaded the files with a single call to the loop with tail command in retrieveFileOrRetry().

Also the files do get overwritten - due to having the same name each time, so if they are been downloaded slower in chunks, a new file (same name, with different data and different checksum) overwrites the previous one, and the tail command is downloading from this new file - so the contents of the file downloaded will never match up and neither will checksum == remoteFile.Checksum

josepdcs commented 9 months ago

There was a problem calculating the offset of the remaining file to download when it couldn't be downloaded all at once. I've published a new release 1.2.1 which fix this one.

jlynch1 commented 9 months ago

I have pulled the latest changes but I still have the same underlying issue as described in the ticket. which is caused by having the same filenames e.g. /tmp/agent-flamegraph-3328373.html for each interval and the files get overwritten with the output for the next interval.

Notice the download for the first file failing and the 3 retries for the first file have the same file size and checksum for the 2nd file 47460 bytes and checksum "7ec5eb445db99e1abe5352b48d6e3806" because the first output was overwritten by the 2nd output with filename /tmp/agent-flamegraph-3328373.html.

File /tmp/agent-flamegraph-3328373.html.gz downloaded (local: b92a264e90f5b87044fab8ab7fe7da28 (47444 bytes) | remote: bf98cfe0ada19dfd919d09d48a91ef07 (34041 bytes))
Checksum does not match, retrying: /tmp/agent-flamegraph-3328373.html.gz...
File /tmp/agent-flamegraph-3328373.html.gz downloaded (local: 7ec5eb445db99e1abe5352b48d6e3806 (47460 bytes) | remote: bf98cfe0ada19dfd919d09d48a91ef07 (34041 bytes))
Checksum does not match, retrying: /tmp/agent-flamegraph-3328373.html.gz...
File /tmp/agent-flamegraph-3328373.html.gz downloaded (local: 7ec5eb445db99e1abe5352b48d6e3806 (47460 bytes) | remote: bf98cfe0ada19dfd919d09d48a91ef07 (34041 bytes))
Checksum does not match, retrying: /tmp/agent-flamegraph-3328373.html.gz...
File /tmp/agent-flamegraph-3328373.html.gz downloaded (local: 7ec5eb445db99e1abe5352b48d6e3806 (47460 bytes) | remote: bf98cfe0ada19dfd919d09d48a91ef07 (34041 bytes))
Checksum does not match, retrying: /tmp/agent-flamegraph-3328373.html.gz...❌
checksum does not match for file /tmp/agent-flamegraph-3328373.html.gz

File /tmp/agent-flamegraph-3328373.html.gz downloaded (local: 7ec5eb445db99e1abe5352b48d6e3806 (47460 bytes) | remote: 7ec5eb445db99e1abe5352b48d6e3806 (47460 bytes))
Remote profiling file downloaded in 3.520464 seconds. ✔
The profiling result file [agent-flamegraph-3328373-2024-02-08T17_41_31Z.html] was obtained in 137.597947 seconds.
{"type":"log","data":{"time":"2024-02-08T17:39:30.91778076Z","level":"debug","msg":"/tmp/async-profiler/profiler.sh -o flamegraph -d 60 -f /tmp/agent-flamegraph-3328373.html -e alloc --fdtransfer 3328373"}}
{"type":"log","data":{"time":"2024-02-08T17:40:31.049361339Z","level":"debug","msg":"stat -c%s /tmp/agent-flamegraph-3328373.html.gz"}}
{"type":"result","data":{"time":"2024-02-08T17:40:31.050328465Z","result-type":"flamegraph","file":"/tmp/agent-flamegraph-3328373.html.gz","file-size-in-bytes":34041,"checksum":"bf98cfe0ada19dfd919d09d48a91ef07","compressor-type":"gzip"}}
{"type":"log","data":{"time":"2024-02-08T17:40:31.050601405Z","level":"debug","msg":"/tmp/async-profiler/profiler.sh -o flamegraph -d 60 -f /tmp/agent-flamegraph-3328373.html -e alloc --fdtransfer 3328373"}}
{"type":"log","data":{"time":"2024-02-08T17:41:31.144286931Z","level":"debug","msg":"stat -c%s /tmp/agent-flamegraph-3328373.html.gz"}}
{"type":"result","data":{"time":"2024-02-08T17:41:31.145361315Z","result-type":"flamegraph","file":"/tmp/agent-flamegraph-3328373.html.gz","file-size-in-bytes":47460,"checksum":"7ec5eb445db99e1abe5352b48d6e3806","compressor-type":"gzip"}}
{"type":"progress","data":{"time":"2024-02-08T17:41:31.145509373Z","stage":"ended"}}
{"type":"log","data":{"time":"2024-02-08T17:41:50.355762225Z","level":"debug","msg":"Received signal: terminated"}}
{"type":"log","data":{"time":"2024-02-08T17:41:50.355802413Z","level":"debug","msg":"/tmp/async-profiler/profiler.sh stop 3328373"}}
{"type":"log","data":{"time":"2024-02-08T17:41:50.384676912Z","level":"debug","msg":"Trying to remove file: /tmp/agent-flamegraph-3328373.html"}}
{"type":"log","data":{"time":"2024-02-08T17:41:50.384779901Z","level":"debug","msg":"Trying to remove file: /tmp/agent-flamegraph-3328373.html.gz"}}
{"type":"log","data":{"time":"2024-02-08T17:41:50.38489262Z","level":"debug","msg":"Profiling finished properly. Bye!"}}
josepdcs commented 9 months ago

Ok. The problem is with the continuous mode. I got it. The file should be renamed according the iteration.

josepdcs commented 9 months ago

Hi, @jlynch1. Now, for continuous mode the result file is renamed after each iteration. It should resolve the issue experienced by you. Release: https://github.com/josepdcs/kubectl-prof/releases/tag/1.2.2

Default profiling tool async-profiler will be used ... ✔
Verifying target pod ... ✔
Launching profiler ... ✔
Profiling ... ✔
⚠Detected more than one PID to profile: [2737 2738]. It will be attempt to profile all of them. Use the --pid flag specifying the corresponding PID if you only want to profile one of them.
Profiling ... ✔
Remote profiling file downloaded in 0.137480 seconds. ✔
The profiling result file [/home/josepdcs/tmp/k-prof/agent-flamegraph-2737-1-2024-02-10T12_32_24Z.html] was obtained in 8.954445 seconds. 🔥
Remote profiling file downloaded in 0.132550 seconds. ✔
The profiling result file [/home/josepdcs/tmp/k-prof/agent-flamegraph-2738-1-2024-02-10T12_32_26Z.html] was obtained in 10.931881 seconds. 🔥
Remote profiling file downloaded in 0.122825 seconds. ✔
The profiling result file [/home/josepdcs/tmp/k-prof/agent-flamegraph-2737-2-2024-02-10T12_32_36Z.html] was obtained in 21.000285 seconds. 🔥
Remote profiling file downloaded in 0.099868 seconds. ✔
The profiling result file [/home/josepdcs/tmp/k-prof/agent-flamegraph-2738-2-2024-02-10T12_32_38Z.html] was obtained in 22.954586 seconds. 🔥
Remote profiling file downloaded in 0.101791 seconds. ✔
The profiling result file [/home/josepdcs/tmp/k-prof/agent-flamegraph-2737-3-2024-02-10T12_32_48Z.html] was obtained in 33.056469 seconds. 🔥
Remote profiling file downloaded in 0.111962 seconds. ✔
The profiling result file [/home/josepdcs/tmp/k-prof/agent-flamegraph-2738-3-2024-02-10T12_32_50Z.html] was obtained in 35.015531 seconds. 🔥
Remote profiling file downloaded in 0.143081 seconds. ✔
The profiling result file [/home/josepdcs/tmp/k-prof/agent-flamegraph-2737-4-2024-02-10T12_33_00Z.html] was obtained in 45.084805 seconds. 🔥
Remote profiling file downloaded in 0.086301 seconds. ✔
The profiling result file [/home/josepdcs/tmp/k-prof/agent-flamegraph-2738-4-2024-02-10T12_33_02Z.html] was obtained in 47.103090 seconds. 🔥
Remote profiling file downloaded in 0.149004 seconds. ✔
The profiling result file [/home/josepdcs/tmp/k-prof/agent-flamegraph-2737-5-2024-02-10T12_33_12Z.html] was obtained in 57.280298 seconds. 🔥
Remote profiling file downloaded in 0.194645 seconds. ✔
The profiling result file [/home/josepdcs/tmp/k-prof/agent-flamegraph-2738-5-2024-02-10T12_33_14Z.html] was obtained in 59.319846 seconds. 🔥
Remote profiling file downloaded in 0.122635 seconds. ✔
The profiling result file [/home/josepdcs/tmp/k-prof/agent-flamegraph-2737-6-2024-02-10T12_33_24Z.html] was obtained in 69.356782 seconds. 🔥
Remote profiling file downloaded in 0.125325 seconds. ✔
The profiling result file [/home/josepdcs/tmp/k-prof/agent-flamegraph-2738-6-2024-02-10T12_33_26Z.html] was obtained in 71.343668 seconds. 🔥

The agent logs:

{"type":"notice","data":{"time":"2024-02-10T12:32:14.48979419Z","msg":"Detected more than one PID to profile: [2737 2738]. It will be attempt to profile all of them. Use the --pid flag specifying the corresponding PID if you only want to profile one of them."}}
{"type":"log","data":{"time":"2024-02-10T12:32:14.489957799Z","level":"debug","msg":"The PIDs to be profiled: [2737 2738]"}}
{"type":"log","data":{"time":"2024-02-10T12:32:14.489968979Z","level":"debug","msg":"cp -r /app/async-profiler /tmp"}}
{"type":"log","data":{"time":"2024-02-10T12:32:14.492197107Z","level":"debug","msg":"/tmp/async-profiler/profiler.sh -o flamegraph -d 10 -f /tmp/agent-flamegraph-2737-1.html -e alloc --fdtransfer 2737"}}
{"type":"log","data":{"time":"2024-02-10T12:32:16.49441983Z","level":"debug","msg":"/tmp/async-profiler/profiler.sh -o flamegraph -d 10 -f /tmp/agent-flamegraph-2738-1.html -e alloc --fdtransfer 2738"}}
{"type":"log","data":{"time":"2024-02-10T12:32:24.528072629Z","level":"debug","msg":"stat -c%s /tmp/agent-flamegraph-2737-1.html.gz"}}
{"type":"result","data":{"time":"2024-02-10T12:32:24.539385288Z","result-type":"flamegraph","file":"/tmp/agent-flamegraph-2737-1.html.gz","file-size-in-bytes":3076,"checksum":"f53920b5bd81ef10234746b432bed775","compressor-type":"gzip"}}
{"type":"log","data":{"time":"2024-02-10T12:32:26.518939778Z","level":"debug","msg":"stat -c%s /tmp/agent-flamegraph-2738-1.html.gz"}}
{"type":"result","data":{"time":"2024-02-10T12:32:26.522493738Z","result-type":"flamegraph","file":"/tmp/agent-flamegraph-2738-1.html.gz","file-size-in-bytes":3218,"checksum":"ce70ecf8952a954321dd011aa0c46222","compressor-type":"gzip"}}
{"type":"log","data":{"time":"2024-02-10T12:32:26.527058575Z","level":"debug","msg":"/tmp/async-profiler/profiler.sh -o flamegraph -d 10 -f /tmp/agent-flamegraph-2737-2.html -e alloc --fdtransfer 2737"}}
{"type":"log","data":{"time":"2024-02-10T12:32:28.549467094Z","level":"debug","msg":"/tmp/async-profiler/profiler.sh -o flamegraph -d 10 -f /tmp/agent-flamegraph-2738-2.html -e alloc --fdtransfer 2738"}}
{"type":"log","data":{"time":"2024-02-10T12:32:36.596948528Z","level":"debug","msg":"stat -c%s /tmp/agent-flamegraph-2737-2.html.gz"}}
{"type":"result","data":{"time":"2024-02-10T12:32:36.599764961Z","result-type":"flamegraph","file":"/tmp/agent-flamegraph-2737-2.html.gz","file-size-in-bytes":3185,"checksum":"ccc87dcf06a25b7277acac65034da77a","compressor-type":"gzip"}}
{"type":"log","data":{"time":"2024-02-10T12:32:38.57433443Z","level":"debug","msg":"stat -c%s /tmp/agent-flamegraph-2738-2.html.gz"}}
{"type":"result","data":{"time":"2024-02-10T12:32:38.577940702Z","result-type":"flamegraph","file":"/tmp/agent-flamegraph-2738-2.html.gz","file-size-in-bytes":3185,"checksum":"ccc87dcf06a25b7277acac65034da77a","compressor-type":"gzip"}}
{"type":"log","data":{"time":"2024-02-10T12:32:38.580316165Z","level":"debug","msg":"/tmp/async-profiler/profiler.sh -o flamegraph -d 10 -f /tmp/agent-flamegraph-2737-3.html -e alloc --fdtransfer 2737"}}
{"type":"log","data":{"time":"2024-02-10T12:32:40.594551181Z","level":"debug","msg":"/tmp/async-profiler/profiler.sh -o flamegraph -d 10 -f /tmp/agent-flamegraph-2738-3.html -e alloc --fdtransfer 2738"}}
{"type":"log","data":{"time":"2024-02-10T12:32:48.674746878Z","level":"debug","msg":"stat -c%s /tmp/agent-flamegraph-2737-3.html.gz"}}
{"type":"result","data":{"time":"2024-02-10T12:32:48.67783395Z","result-type":"flamegraph","file":"/tmp/agent-flamegraph-2737-3.html.gz","file-size-in-bytes":3166,"checksum":"5c37d8aae7d291840b8c119c9eb0f494","compressor-type":"gzip"}}
{"type":"log","data":{"time":"2024-02-10T12:32:50.623320299Z","level":"debug","msg":"stat -c%s /tmp/agent-flamegraph-2738-3.html.gz"}}
{"type":"result","data":{"time":"2024-02-10T12:32:50.626918772Z","result-type":"flamegraph","file":"/tmp/agent-flamegraph-2738-3.html.gz","file-size-in-bytes":3118,"checksum":"c53dd29e105b836fb698decacc626aa5","compressor-type":"gzip"}}
{"type":"log","data":{"time":"2024-02-10T12:32:50.628412168Z","level":"debug","msg":"/tmp/async-profiler/profiler.sh -o flamegraph -d 10 -f /tmp/agent-flamegraph-2737-4.html -e alloc --fdtransfer 2737"}}
{"type":"log","data":{"time":"2024-02-10T12:32:52.651552765Z","level":"debug","msg":"/tmp/async-profiler/profiler.sh -o flamegraph -d 10 -f /tmp/agent-flamegraph-2738-4.html -e alloc --fdtransfer 2738"}}
{"type":"log","data":{"time":"2024-02-10T12:33:00.660429899Z","level":"debug","msg":"stat -c%s /tmp/agent-flamegraph-2737-4.html.gz"}}
{"type":"result","data":{"time":"2024-02-10T12:33:00.664559186Z","result-type":"flamegraph","file":"/tmp/agent-flamegraph-2737-4.html.gz","file-size-in-bytes":3253,"checksum":"75a240f19803150475b484165d70297a","compressor-type":"gzip"}}
{"type":"log","data":{"time":"2024-02-10T12:33:02.737658938Z","level":"debug","msg":"stat -c%s /tmp/agent-flamegraph-2738-4.html.gz"}}
{"type":"result","data":{"time":"2024-02-10T12:33:02.740126511Z","result-type":"flamegraph","file":"/tmp/agent-flamegraph-2738-4.html.gz","file-size-in-bytes":3195,"checksum":"5de9ab43240b91e7cdd0485b8d5a230d","compressor-type":"gzip"}}
{"type":"log","data":{"time":"2024-02-10T12:33:02.743609828Z","level":"debug","msg":"/tmp/async-profiler/profiler.sh -o flamegraph -d 10 -f /tmp/agent-flamegraph-2737-5.html -e alloc --fdtransfer 2737"}}
{"type":"log","data":{"time":"2024-02-10T12:33:04.766144545Z","level":"debug","msg":"/tmp/async-profiler/profiler.sh -o flamegraph -d 10 -f /tmp/agent-flamegraph-2738-5.html -e alloc --fdtransfer 2738"}}
{"type":"log","data":{"time":"2024-02-10T12:33:12.84812847Z","level":"debug","msg":"stat -c%s /tmp/agent-flamegraph-2737-5.html.gz"}}
{"type":"result","data":{"time":"2024-02-10T12:33:12.853989111Z","result-type":"flamegraph","file":"/tmp/agent-flamegraph-2737-5.html.gz","file-size-in-bytes":3184,"checksum":"3065ab4aa8efe399ca5d27d8e4a0fe04","compressor-type":"gzip"}}
{"type":"log","data":{"time":"2024-02-10T12:33:14.835274652Z","level":"debug","msg":"stat -c%s /tmp/agent-flamegraph-2738-5.html.gz"}}
{"type":"result","data":{"time":"2024-02-10T12:33:14.847132353Z","result-type":"flamegraph","file":"/tmp/agent-flamegraph-2738-5.html.gz","file-size-in-bytes":3208,"checksum":"2ae11334dec8c20bce1f9f2eb15fe977","compressor-type":"gzip"}}
{"type":"log","data":{"time":"2024-02-10T12:33:14.860894343Z","level":"debug","msg":"/tmp/async-profiler/profiler.sh -o flamegraph -d 10 -f /tmp/agent-flamegraph-2737-6.html -e alloc --fdtransfer 2737"}}
{"type":"log","data":{"time":"2024-02-10T12:33:16.875963017Z","level":"debug","msg":"/tmp/async-profiler/profiler.sh -o flamegraph -d 10 -f /tmp/agent-flamegraph-2738-6.html -e alloc --fdtransfer 2738"}}
{"type":"log","data":{"time":"2024-02-10T12:33:24.947702205Z","level":"debug","msg":"stat -c%s /tmp/agent-flamegraph-2737-6.html.gz"}}
{"type":"result","data":{"time":"2024-02-10T12:33:24.957726514Z","result-type":"flamegraph","file":"/tmp/agent-flamegraph-2737-6.html.gz","file-size-in-bytes":3202,"checksum":"1c55274763d3a007def56a41732f39db","compressor-type":"gzip"}}
{"type":"log","data":{"time":"2024-02-10T12:33:26.936852816Z","level":"debug","msg":"stat -c%s /tmp/agent-flamegraph-2738-6.html.gz"}}
{"type":"result","data":{"time":"2024-02-10T12:33:26.941434454Z","result-type":"flamegraph","file":"/tmp/agent-flamegraph-2738-6.html.gz","file-size-in-bytes":3239,"checksum":"02e6ea273dd403ff15fe55bdea107282","compressor-type":"gzip"}}
{"type":"progress","data":{"time":"2024-02-10T12:33:26.941530821Z","stage":"ended"}}
{"type":"log","data":{"time":"2024-02-10T12:33:27.132676216Z","level":"debug","msg":"Received signal: terminated"}}
{"type":"log","data":{"time":"2024-02-10T12:33:27.132727844Z","level":"debug","msg":"/tmp/async-profiler/profiler.sh stop 2737"}}
{"type":"log","data":{"time":"2024-02-10T12:33:27.175661715Z","level":"debug","msg":"/tmp/async-profiler/profiler.sh stop 2738"}}
{"type":"log","data":{"time":"2024-02-10T12:33:27.267339459Z","level":"debug","msg":"Trying to remove file: /tmp/agent-flamegraph-2737-1.html"}}
{"type":"log","data":{"time":"2024-02-10T12:33:27.2675347Z","level":"debug","msg":"Trying to remove file: /tmp/agent-flamegraph-2737-1.html.gz"}}
{"type":"log","data":{"time":"2024-02-10T12:33:27.267589655Z","level":"debug","msg":"Trying to remove file: /tmp/agent-flamegraph-2737-2.html"}}
{"type":"log","data":{"time":"2024-02-10T12:33:27.267686007Z","level":"debug","msg":"Trying to remove file: /tmp/agent-flamegraph-2737-2.html.gz"}}
{"type":"log","data":{"time":"2024-02-10T12:33:27.267737353Z","level":"debug","msg":"Trying to remove file: /tmp/agent-flamegraph-2737-3.html"}}
{"type":"log","data":{"time":"2024-02-10T12:33:27.267829135Z","level":"debug","msg":"Trying to remove file: /tmp/agent-flamegraph-2737-3.html.gz"}}
{"type":"log","data":{"time":"2024-02-10T12:33:27.267871652Z","level":"debug","msg":"Trying to remove file: /tmp/agent-flamegraph-2737-4.html"}}
{"type":"log","data":{"time":"2024-02-10T12:33:27.267907182Z","level":"debug","msg":"Trying to remove file: /tmp/agent-flamegraph-2737-4.html.gz"}}
{"type":"log","data":{"time":"2024-02-10T12:33:27.267939422Z","level":"debug","msg":"Trying to remove file: /tmp/agent-flamegraph-2737-5.html"}}
{"type":"log","data":{"time":"2024-02-10T12:33:27.267978649Z","level":"debug","msg":"Trying to remove file: /tmp/agent-flamegraph-2737-5.html.gz"}}
{"type":"log","data":{"time":"2024-02-10T12:33:27.268011715Z","level":"debug","msg":"Trying to remove file: /tmp/agent-flamegraph-2737-6.html"}}
{"type":"log","data":{"time":"2024-02-10T12:33:27.268046987Z","level":"debug","msg":"Trying to remove file: /tmp/agent-flamegraph-2737-6.html.gz"}}
{"type":"log","data":{"time":"2024-02-10T12:33:27.268097433Z","level":"debug","msg":"Trying to remove file: /tmp/agent-flamegraph-2738-1.html"}}
{"type":"log","data":{"time":"2024-02-10T12:33:27.268448036Z","level":"debug","msg":"Trying to remove file: /tmp/agent-flamegraph-2738-1.html.gz"}}
{"type":"log","data":{"time":"2024-02-10T12:33:27.268562686Z","level":"debug","msg":"Trying to remove file: /tmp/agent-flamegraph-2738-2.html"}}
{"type":"log","data":{"time":"2024-02-10T12:33:27.268603807Z","level":"debug","msg":"Trying to remove file: /tmp/agent-flamegraph-2738-2.html.gz"}}
{"type":"log","data":{"time":"2024-02-10T12:33:27.268639829Z","level":"debug","msg":"Trying to remove file: /tmp/agent-flamegraph-2738-3.html"}}
{"type":"log","data":{"time":"2024-02-10T12:33:27.268686509Z","level":"debug","msg":"Trying to remove file: /tmp/agent-flamegraph-2738-3.html.gz"}}
{"type":"log","data":{"time":"2024-02-10T12:33:27.268730626Z","level":"debug","msg":"Trying to remove file: /tmp/agent-flamegraph-2738-4.html"}}
{"type":"log","data":{"time":"2024-02-10T12:33:27.268824059Z","level":"debug","msg":"Trying to remove file: /tmp/agent-flamegraph-2738-4.html.gz"}}
{"type":"log","data":{"time":"2024-02-10T12:33:27.268867889Z","level":"debug","msg":"Trying to remove file: /tmp/agent-flamegraph-2738-5.html"}}
{"type":"log","data":{"time":"2024-02-10T12:33:27.269103104Z","level":"debug","msg":"Trying to remove file: /tmp/agent-flamegraph-2738-5.html.gz"}}
{"type":"log","data":{"time":"2024-02-10T12:33:27.269617293Z","level":"debug","msg":"Trying to remove file: /tmp/agent-flamegraph-2738-6.html"}}
{"type":"log","data":{"time":"2024-02-10T12:33:27.269771517Z","level":"debug","msg":"Trying to remove file: /tmp/agent-flamegraph-2738-6.html.gz"}}
{"type":"log","data":{"time":"2024-02-10T12:33:27.271571119Z","level":"debug","msg":"Profiling finished properly. Bye!"}}
jlynch1 commented 9 months ago

Hi @josepdcs, I can confirm it is now downloading correctly the flamegraphs for me in normal and continuous mode. Thanks for all your work in maintaining this project.