Closed mithkr closed 3 years ago
Oh damn. I'm going to investigate this as soon as possible and release a fixed version. I'll let you know once I was able to replicate this.
Ok I just had a look at this and did a bit of testing. I wasn't able to reproduce this on the dev system. The JS heap size always stayed in the 5-8MB area and also the memory profile doesn't show any obvious signs of a memory leak:
Could yo maybe elaborate on what you did to cause OctoDash to fail? Was it just idleing? According to your logs OctoDash is filling up around half of the memory in about 45 minutes, so it would crash after around 70-80 minutes?
The problem is on my system seen immediately after reboot. System is fresh rebooted and octoprint and octodash is started. Octoprint is connected to 3d printer and only temperature sensor values are exchange between printer, octoprint and octodash. Nothing else is started. I made new top measurement after reboot and there are two processes (pid: 510, 576) of octodash are continuous increasing in the RES memory. I also add at the end the list of the octodash processes with their args and attached strace files of this two processes (strace -o octodash-510 -tt -ff -p 510; strace -o octodash-576 -tt -ff -p 576) octodash-strace.zip
top - 19:06:10 up 3 min, 2 users, load average: 1,92, 1,47, 0,63
Tasks: 131 total, 1 running, 130 sleeping, 0 stopped, 0 zombie
%Cpu(s): 1,6 us, 1,2 sy, 0,0 ni, 97,2 id, 0,1 wa, 0,0 hi, 0,0 si, 0,0 st
MiB Mem : 460,7 total, 38,3 free, 197,6 used, 224,7 buff/cache
MiB Swap: 100,0 total, 50,5 free, 49,5 used. 206,7 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
431 pi 20 0 410264 76412 8008 S 8,3 16,2 1:18.95 octoprint
510 pi 20 0 580912 43900 7692 S 0,0 9,3 0:05.87 octodash
576 pi 20 0 309048 42876 18456 S 0,7 9,1 0:11.68 octodash
592 pi 20 0 215040 18748 8544 S 0,0 4,0 0:02.19 octodash
557 pi 20 0 209476 10924 5056 S 0,3 2,3 0:00.59 octodash
top - 19:08:34 up 6 min, 2 users, load average: 0,27, 0,93, 0,54
Tasks: 125 total, 1 running, 124 sleeping, 0 stopped, 0 zombie
%Cpu(s): 2,2 us, 0,7 sy, 0,0 ni, 97,0 id, 0,1 wa, 0,0 hi, 0,0 si, 0,0 st
MiB Mem : 460,7 total, 33,2 free, 198,8 used, 228,7 buff/cache
MiB Swap: 100,0 total, 50,5 free, 49,5 used. 205,4 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
431 pi 20 0 410264 76412 8008 S 8,3 16,2 1:31.23 octoprint
510 pi 20 0 580912 52180 15808 S 0,3 11,1 0:06.36 octodash
576 pi 20 0 309048 43180 18840 S 0,3 9,2 0:12.53 octodash
592 pi 20 0 215040 18748 8544 S 0,0 4,0 0:02.19 octodash
557 pi 20 0 209476 10924 5056 S 0,3 2,3 0:00.78 octodash
top - 19:11:36 up 9 min, 2 users, load average: 0,08, 0,53, 0,45
Tasks: 124 total, 1 running, 123 sleeping, 0 stopped, 0 zombie
%Cpu(s): 2,0 us, 0,7 sy, 0,0 ni, 97,3 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st
MiB Mem : 460,7 total, 31,3 free, 199,2 used, 230,2 buff/cache
MiB Swap: 100,0 total, 50,5 free, 49,5 used. 205,0 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
431 pi 20 0 410264 76412 8008 S 8,6 16,2 1:46.65 octoprint
510 pi 20 0 580912 52180 15808 S 0,3 11,1 0:06.84 octodash
576 pi 20 0 311096 44420 19012 S 0,7 9,4 0:13.63 octodash
592 pi 20 0 215040 18748 8544 S 0,0 4,0 0:02.19 octodash
557 pi 20 0 209476 10924 5056 S 0,3 2,3 0:01.03 octodash
top - 19:24:26 up 22 min, 2 users, load average: 0,26, 0,20, 0,27
Tasks: 122 total, 1 running, 121 sleeping, 0 stopped, 0 zombie
%Cpu(s): 3,7 us, 1,1 sy, 0,0 ni, 95,2 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st
MiB Mem : 460,7 total, 30,1 free, 192,6 used, 238,0 buff/cache
MiB Swap: 100,0 total, 50,5 free, 49,5 used. 211,6 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
431 pi 20 0 410264 76416 8008 S 9,3 16,2 2:51.60 octoprint
510 pi 20 0 580912 53892 17468 S 0,3 11,4 0:08.72 octodash
576 pi 20 0 311096 52464 28260 S 0,7 11,1 0:18.08 octodash
592 pi 20 0 215040 18748 8544 S 0,0 4,0 0:02.19 octodash
557 pi 20 0 209476 11380 5512 S 0,0 2,4 0:02.07 octodash
root@octopi:~# ps aux | grep octodash
pi 510 0.6 11.4 580912 53892 tty1 Sl 19:02 0:08 /opt/OctoDash/octodash
pi 515 0.0 0.9 156208 4316 tty1 S 19:02 0:00 /opt/OctoDash/octodash --type=zygote --no-zygote-sandbox
pi 516 0.0 0.7 156208 3332 tty1 S 19:02 0:00 /opt/OctoDash/octodash --type=zygote
pi 518 0.0 0.0 156208 336 tty1 S 19:02 0:00 /opt/OctoDash/octodash --type=zygote
pi 557 0.1 2.4 209476 11380 tty1 Sl 19:02 0:02 /opt/OctoDash/octodash --type=utility --utility-sub-type=network.mojom.NetworkService --field-trial-handle=8698878411638238832,2576688415711229123,131072 --enable-features=WebComponentsV0Enabled --disable-features=CertVerifierService,CookiesWithoutSameSiteMustBeSecure,SameSiteByDefaultCookies,SpareRendererForSitePerProcess --lang=de --service-sandbox-type=network --standard-schemes=app --secure-schemes --bypasscsp-schemes --cors-schemes --fetch-schemes --service-worker-schemes --streaming-schemes --shared-files=v8_context_snapshot_data:100
pi 576 1.3 11.1 311096 52728 tty1 Sl 19:02 0:18 /opt/OctoDash/octodash --type=renderer --touch-events=enabled --field-trial-handle=8698878411638238832,2576688415711229123,131072 --enable-features=WebComponentsV0Enabled --disable-features=CertVerifierService,CookiesWithoutSameSiteMustBeSecure,SameSiteByDefaultCookies,SpareRendererForSitePerProcess --lang=de --standard-schemes=app --secure-schemes --bypasscsp-schemes --cors-schemes --fetch-schemes --service-worker-schemes --streaming-schemes --app-path=/opt/OctoDash/resources/app.asar --no-sandbox --no-zygote --num-raster-threads=2 --enable-main-frame-before-activation --renderer-client-id=4 --no-v8-untrusted-code-mitigations --shared-files=v8_context_snapshot_data:100
pi 592 0.1 3.9 215040 18748 tty1 Sl 19:02 0:02 /opt/OctoDash/octodash --type=gpu-process --field-trial-handle=8698878411638238832,2576688415711229123,131072 --enable-features=WebComponentsV0Enabled --disable-features=CertVerifierService,CookiesWithoutSameSiteMustBeSecure,SameSiteByDefaultCookies,SpareRendererForSitePerProcess --gpu-preferences=UAAAAAAAAAAgAAAQAAAAAAAAAAAAAAAAAABgAAAAAAAwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAgAAAAAAAAACAAAAAAAAAAAAAAAAAAAAAIAAAAAAAAAAgAAAAAAAAACAAAAAAAAAA= --use-gl=swiftshader-webgl --shared-files
pi 594 0.0 0.6 171084 3228 tty1 S 19:02 0:00 /opt/OctoDash/octodash --type=broker
root 1631 0.0 0.1 7360 504 pts/1 S+ 19:24 0:00 grep octodash
Thanks for the additional info! I'll do some further investigation and probably provide a dev build here to test.
I have made a further investigation and looked how octodash behave if printer is disconnected from octoprint. If printer is disconnected, than octodash show sleep screen and memory leak is not seen. So memory leak maybe somehow related to the printer status update messages (e.g. temp sensors) between octodash and octoprint. Here the top measurement with disconnected printer:
top - 22:43:06 up 1 day, 12:27, 2 users, load average: 0,04, 0,06, 0,05
Tasks: 8 total, 0 running, 8 sleeping, 0 stopped, 0 zombie
%Cpu(s): 1,4 us, 0,6 sy, 0,0 ni, 97,9 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st
MiB Mem : 460,7 total, 44,6 free, 210,3 used, 205,8 buff/cache
MiB Swap: 100,0 total, 66,5 free, 33,5 used. 191,0 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
580 pi 20 0 339864 76724 36328 S 0,3 16,3 6:59.93 /opt/OctoDash/octodash --type=renderer --touch-events=enabled --field-trial-handle=12013433723393411160,11906847453383645420,131072 --disable-features=CookiesWithoutSameSiteMustBeSecure,+
510 pi 20 0 573872 61488 31660 S 0,2 13,0 4:22.77 /opt/OctoDash/octodash
591 pi 20 0 217524 28016 14388 S 0,0 5,9 0:02.39 /opt/OctoDash/octodash --type=gpu-process --field-trial-handle=12013433723393411160,11906847453383645420,131072 --disable-features=CookiesWithoutSameSiteMustBeSecure,SameSiteByDefaultCoo+
558 pi 20 0 209580 17900 12248 S 0,1 3,8 2:15.92 /opt/OctoDash/octodash --type=utility --utility-sub-type=network.mojom.NetworkService --field-trial-handle=12013433723393411160,11906847453383645420,131072 --disable-features=CookiesWith+
515 pi 20 0 158712 9484 4640 S 0,0 2,0 0:00.13 /opt/OctoDash/octodash --type=zygote
514 pi 20 0 158712 9392 4576 S 0,0 2,0 0:00.14 /opt/OctoDash/octodash --type=zygote --no-zygote-sandbox
598 pi 20 0 173568 6992 1756 S 0,0 1,5 0:00.00 /opt/OctoDash/octodash --type=broker
517 pi 20 0 158712 4984 144 S 0,0 1,1 0:00.00 /opt/OctoDash/octodash --type=zygote
top - 01:43:08 up 2 days, 15:27, 2 users, load average: 0,00, 0,03, 0,05
Tasks: 8 total, 0 running, 8 sleeping, 0 stopped, 0 zombie
%Cpu(s): 1,4 us, 0,6 sy, 0,0 ni, 98,0 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st
MiB Mem : 460,7 total, 49,7 free, 212,6 used, 198,3 buff/cache
MiB Swap: 100,0 total, 63,5 free, 36,5 used. 185,5 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
580 pi 20 0 339864 79320 36344 S 0,3 16,8 11:59.53 /opt/OctoDash/octodash --type=renderer --touch-events=enabled --field-trial-handle=12013433723393411160,11906847453383645420,131072 --disable-features=CookiesWithoutSameSiteMustBeSecure,+
510 pi 20 0 573872 59028 31652 S 0,2 12,5 7:33.04 /opt/OctoDash/octodash
591 pi 20 0 217524 28012 14384 S 0,0 5,9 0:02.67 /opt/OctoDash/octodash --type=gpu-process --field-trial-handle=12013433723393411160,11906847453383645420,131072 --disable-features=CookiesWithoutSameSiteMustBeSecure,SameSiteByDefaultCoo+
558 pi 20 0 225972 18856 13016 S 0,1 4,0 3:56.48 /opt/OctoDash/octodash --type=utility --utility-sub-type=network.mojom.NetworkService --field-trial-handle=12013433723393411160,11906847453383645420,131072 --disable-features=CookiesWith+
515 pi 20 0 158712 9480 4636 S 0,0 2,0 0:00.13 /opt/OctoDash/octodash --type=zygote
514 pi 20 0 158712 9388 4572 S 0,0 2,0 0:00.14 /opt/OctoDash/octodash --type=zygote --no-zygote-sandbox
598 pi 20 0 173568 6992 1756 S 0,0 1,5 0:00.00 /opt/OctoDash/octodash --type=broker
517 pi 20 0 158712 4984 144 S 0,0 1,1 0:00.00 /opt/OctoDash/octodash --type=zygote
top - 15:43:09 up 3 days, 5:27, 2 users, load average: 0,02, 0,08, 0,03
Tasks: 8 total, 0 running, 8 sleeping, 0 stopped, 0 zombie
%Cpu(s): 1,4 us, 0,6 sy, 0,0 ni, 98,0 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st
MiB Mem : 460,7 total, 49,5 free, 212,1 used, 199,1 buff/cache
MiB Swap: 100,0 total, 63,5 free, 36,5 used. 186,0 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
580 pi 20 0 339864 79616 36340 S 0,3 16,9 14:35.27 /opt/OctoDash/octodash --type=renderer --touch-events=enabled --field-trial-handle=12013433723393411160,11906847453383645420,131072 --disable-features=CookiesWithoutSameSiteMustBeSecure,+
510 pi 20 0 573872 58928 31612 S 0,2 12,5 9:11.75 /opt/OctoDash/octodash
591 pi 20 0 217524 28012 14384 S 0,0 5,9 0:02.82 /opt/OctoDash/octodash --type=gpu-process --field-trial-handle=12013433723393411160,11906847453383645420,131072 --disable-features=CookiesWithoutSameSiteMustBeSecure,SameSiteByDefaultCoo+
558 pi 20 0 225972 18972 13016 S 0,1 4,0 4:48.81 /opt/OctoDash/octodash --type=utility --utility-sub-type=network.mojom.NetworkService --field-trial-handle=12013433723393411160,11906847453383645420,131072 --disable-features=CookiesWith+
515 pi 20 0 158712 9480 4636 S 0,0 2,0 0:00.13 /opt/OctoDash/octodash --type=zygote
514 pi 20 0 158712 9388 4572 S 0,0 2,0 0:00.14 /opt/OctoDash/octodash --type=zygote --no-zygote-sandbox
598 pi 20 0 173568 6992 1756 S 0,0 1,5 0:00.00 /opt/OctoDash/octodash --type=broker
517 pi 20 0 158712 4984 144 S 0,0 1,1 0:00.00 /opt/OctoDash/octodash --type=zygote
Interesting find. I'm going to post a dev build here sometime tomorrow, please try that one as well and report whether the memory leak is gone. Thanks!
I also don't know what you changed with your top command in your latest comment, but having the arguments for each process actually is super useful, so please include them for further tests if possible! :)
https://drive.google.com/file/d/1M1_Ksvj49Naxz-Yijxj2pmalApb07Bl0/view?usp=sharing please try this version and report whether this fixes the issue.
Also i noticed, that your raspberry only has 512MB of memory, which model are you running this on?
Thanks, I will test it.
I have a raspberry pi 3A+
I have run your test version for 40 hour and capture memory usage every 10 minutes with following command:
top -o RES -p $(pidof -f %d, octodash)0 -b -c -d 600 | tee top.log
unluckily mem leak is still there. See attached top.log and generated graphs from that log file. top_plot.txt contains the script I used to generate the graphs from the top.log file.
top.log
top_plot.txt
Could you try this version: https://drive.google.com/file/d/1uTu9q4XjojOWRfrwU4-win52v4AuwVY0/view?usp=sharing and report whether that fixed the issue?
I have tested. I looks like that after some time memory increase stops now, see charts. I will further test after my vacation in 3 weeks. top.log
Ok that looks like we're finally making progress! Thanks for the all graphs. It's really weird that it's allocating memory for a while and then just stops. Might need to do some further digging.
Enjoy your vacation!
I have also been sealing with this. Any idea when the fix will go main stream?
Once @mithkr has confirmed that this has been fixed for him :) But for now he should enjoy his vacation :) Will definitely be included in the next release though.
I have now started a long time recording of top, to see memory consumption for 1 week. I will then upload the result.
Thanks!
I have now started a long time recording of top, to see memory consumption for 1 week. I will then upload the result.
Here are the test results. It looks good. Measurement period is about 9 days. No OOM happened. I only see a very little increase in the virtual memory chart (light blue). Due to the jumps in the RES memory chart, due to enabled swap, it is not possible to proof this trend there. I will repeat measurement with disabled swap. But it can be concluded, that the fixes you already provided helped a lot. top.log
Ok great thanks for the update! If anything weird shows up again feel free to re-open this issue :)
Describe the bug System runs out of memory after a while also if idle (no print jobs running). OOM killer kills octodash, as it has leaked the memory.
[289804.365120] Out of memory: Killed process 509 (octodash) total-vm:817456kB, anon-rss:249080kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:632kB oom_score_adj:0
To Reproduce Boot Raspberry Pi with auto stared octodash and wait until system runs out of memory. Monitoring memory usage with top shows memory usage increase of octodash.
Expected behavior System should not run out of memory.
Screenshots
General Information:
Additional context