guysoft / FullPageOS

A raspberrypi distro to display a full page browser on boot
GNU General Public License v3.0
3.89k stars 240 forks source link

Chromium crashes periodically #291

Open ilyai opened 5 years ago

ilyai commented 5 years ago

I'm running a Chrome app on Raspberry Pi 3 and 4 that plays media content using HTML5 (video, images, web widgets) and after some time the web page crashes (Aw, Snap! error is displayed). System log contains the following:

Aug  1 14:20:36 fullpageos kernel: [  601.273226] Compositor invoked oom-killer: gfp_mask=0x7000c0(GFP_KERNEL_ACCOUNT), nodemask=(null), order=0, oom_score_adj=300
Aug  1 14:20:36 fullpageos kernel: [  601.273232] Compositor cpuset=/ mems_allowed=0
Aug  1 14:20:36 fullpageos kernel: [  601.273248] CPU: 0 PID: 1042 Comm: Compositor Tainted: G         C        4.19.57-v7l+ #1244
Aug  1 14:20:36 fullpageos kernel: [  601.273251] Hardware name: BCM2835
Aug  1 14:20:36 fullpageos kernel: [  601.273269] [<c0212c8c>] (unwind_backtrace) from [<c020d49c>] (show_stack+0x20/0x24)
Aug  1 14:20:36 fullpageos kernel: [  601.273279] [<c020d49c>] (show_stack) from [<c0976220>] (dump_stack+0xd4/0x118)
Aug  1 14:20:36 fullpageos kernel: [  601.273289] [<c0976220>] (dump_stack) from [<c033f134>] (dump_header+0x80/0x250)
Aug  1 14:20:36 fullpageos kernel: [  601.273298] [<c033f134>] (dump_header) from [<c033e4ac>] (oom_kill_process+0x358/0x3a8)
Aug  1 14:20:36 fullpageos kernel: [  601.273306] [<c033e4ac>] (oom_kill_process) from [<c033eddc>] (out_of_memory+0x134/0x36c)
Aug  1 14:20:36 fullpageos kernel: [  601.273313] [<c033eddc>] (out_of_memory) from [<c03451a0>] (__alloc_pages_nodemask+0xfc0/0x1180)
Aug  1 14:20:36 fullpageos kernel: [  601.273323] [<c03451a0>] (__alloc_pages_nodemask) from [<c084c0a8>] (alloc_skb_with_frags+0xe4/0x1b4)
Aug  1 14:20:36 fullpageos kernel: [  601.273333] [<c084c0a8>] (alloc_skb_with_frags) from [<c08448b8>] (sock_alloc_send_pskb+0x214/0x23c)
Aug  1 14:20:36 fullpageos kernel: [  601.273345] [<c08448b8>] (sock_alloc_send_pskb) from [<c0933f50>] (unix_stream_sendmsg+0x218/0x390)
Aug  1 14:20:36 fullpageos kernel: [  601.273354] [<c0933f50>] (unix_stream_sendmsg) from [<c083fa10>] (sock_sendmsg+0x24/0x34)
Aug  1 14:20:36 fullpageos kernel: [  601.273360] [<c083fa10>] (sock_sendmsg) from [<c0840e28>] (__sys_sendto+0xd4/0x114)
Aug  1 14:20:36 fullpageos kernel: [  601.273366] [<c0840e28>] (__sys_sendto) from [<c0840ec4>] (sys_send+0x28/0x30)
Aug  1 14:20:36 fullpageos kernel: [  601.273373] [<c0840ec4>] (sys_send) from [<c0201198>] (__sys_trace_return+0x0/0x10)
Aug  1 14:20:36 fullpageos kernel: [  601.273377] Exception stack(0xd1441fa8 to 0xd1441ff0)
Aug  1 14:20:36 fullpageos kernel: [  601.273382] 1fa0:                   00004000 00002750 0000001c b0065e58 00002750 00004000
Aug  1 14:20:36 fullpageos kernel: [  601.273387] 1fc0: 00004000 00002750 b0065e58 00000121 00000000 ae5faa70 00000000 00000000
Aug  1 14:20:36 fullpageos kernel: [  601.273391] 1fe0: 00000002 ae5fa9c8 00000000 b6aad6e0
Aug  1 14:20:36 fullpageos kernel: [  601.273395] Mem-Info:
Aug  1 14:20:36 fullpageos kernel: [  601.273406] active_anon:51018 inactive_anon:98637 isolated_anon:0
Aug  1 14:20:36 fullpageos kernel: [  601.273406]  active_file:3309 inactive_file:5117 isolated_file:0
Aug  1 14:20:36 fullpageos kernel: [  601.273406]  unevictable:4 dirty:72 writeback:0 unstable:0
Aug  1 14:20:36 fullpageos kernel: [  601.273406]  slab_reclaimable:3807 slab_unreclaimable:7864
Aug  1 14:20:36 fullpageos kernel: [  601.273406]  mapped:104056 shmem:109691 pagetables:1729 bounce:0
Aug  1 14:20:36 fullpageos kernel: [  601.273406]  free:57657 free_pcp:0 free_cma:53016
Aug  1 14:20:36 fullpageos kernel: [  601.273413] Node 0 active_anon:204072kB inactive_anon:394548kB active_file:13236kB inactive_file:20468kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:416224kB dirty:288kB writeback:0kB shmem:438764kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
Aug  1 14:20:36 fullpageos kernel: [  601.273423] DMA free:226120kB min:16384kB low:20480kB high:24576kB active_anon:166036kB inactive_anon:290260kB active_file:3316kB inactive_file:296kB unevictable:0kB writepending:156kB present:786432kB managed:762348kB mlocked:0kB kernel_stack:1760kB pagetables:4348kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:212064kB
Aug  1 14:20:36 fullpageos kernel: [  601.273426] lowmem_reserve[]: 0 0 180 180
Aug  1 14:20:36 fullpageos kernel: [  601.273451] HighMem free:4508kB min:180kB low:1168kB high:2156kB active_anon:38036kB inactive_anon:104288kB active_file:9848kB inactive_file:19912kB unevictable:16kB writepending:132kB present:184320kB managed:184320kB mlocked:16kB kernel_stack:0kB pagetables:2568kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Aug  1 14:20:36 fullpageos kernel: [  601.273454] lowmem_reserve[]: 0 0 0 0
Aug  1 14:20:36 fullpageos kernel: [  601.273474] DMA: 426*4kB (UMEH) 234*8kB (UMEH) 61*16kB (UMEH) 11*32kB (UMEHC) 14*64kB (UEC) 8*128kB (UC) 8*256kB (C) 11*512kB (EHC) 9*1024kB (UEHC) 5*2048kB (EHC) 47*4096kB (C) = 226472kB
Aug  1 14:20:36 fullpageos kernel: [  601.273557] HighMem: 445*4kB (UM) 195*8kB (UM) 30*16kB (UM) 2*32kB (UM) 0*64kB 3*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 4524kB
Aug  1 14:20:36 fullpageos kernel: [  601.273622] 118489 total pagecache pages
Aug  1 14:20:36 fullpageos kernel: [  601.273627] 316 pages in swap cache
Aug  1 14:20:36 fullpageos kernel: [  601.273631] Swap cache stats: add 64861, delete 64545, find 13772/17523
Aug  1 14:20:36 fullpageos kernel: [  601.273635] Free swap  = 0kB
Aug  1 14:20:36 fullpageos kernel: [  601.273638] Total swap = 102396kB
Aug  1 14:20:36 fullpageos kernel: [  601.273642] 242688 pages RAM
Aug  1 14:20:36 fullpageos kernel: [  601.273646] 46080 pages HighMem/MovableOnly
Aug  1 14:20:36 fullpageos kernel: [  601.273649] 6021 pages reserved
Aug  1 14:20:36 fullpageos kernel: [  601.273653] 65536 pages cma reserved
Aug  1 14:20:36 fullpageos kernel: [  601.273656] Tasks state (memory values in pages):
Aug  1 14:20:36 fullpageos kernel: [  601.273660] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Aug  1 14:20:36 fullpageos kernel: [  601.273671] [    112]     0   112     4734      446   143360       11             0 systemd-journal
Aug  1 14:20:36 fullpageos kernel: [  601.273677] [    142]     0   142     4500      201    49152       79         -1000 systemd-udevd
Aug  1 14:20:36 fullpageos kernel: [  601.273683] [    315]   100   315     5592      133    61440       30             0 systemd-timesyn
Aug  1 14:20:36 fullpageos kernel: [  601.273689] [    357] 65534   357     1080       36    36864        7             0 thd
Aug  1 14:20:36 fullpageos kernel: [  601.273694] [    358]     0   358      923       86    32768       15             0 alsactl
Aug  1 14:20:36 fullpageos kernel: [  601.273700] [    359]   104   359     1640       93    40960       32          -900 dbus-daemon
Aug  1 14:20:36 fullpageos kernel: [  601.273705] [    361]     0   361     3275      139    53248       33             0 systemd-logind
Aug  1 14:20:36 fullpageos kernel: [  601.273710] [    366]     0   366     1993       76    40960        3             0 cron
Aug  1 14:20:36 fullpageos kernel: [  601.273715] [    368]   108   368     1443       91    36864        6             0 avahi-daemon
Aug  1 14:20:36 fullpageos kernel: [  601.273720] [    372]     0   372     6914       11    36864        9             0 rngd
Aug  1 14:20:36 fullpageos kernel: [  601.273725] [    376]     0   376     6378      140    57344       18             0 rsyslogd
Aug  1 14:20:36 fullpageos kernel: [  601.273730] [    382]     0   382     2675      122    49152        1             0 wpa_supplicant
Aug  1 14:20:36 fullpageos kernel: [  601.273736] [    383]   108   383     1443       48    32768       15             0 avahi-daemon
Aug  1 14:20:36 fullpageos kernel: [  601.273741] [    446]     0   446     2740      139    45056       25             0 wpa_supplicant
Aug  1 14:20:36 fullpageos kernel: [  601.273746] [    451]    33   451     2180      216    45056        8             0 lighttpd
Aug  1 14:20:36 fullpageos kernel: [  601.273751] [    469]    33   469    46618     1022    94208       78             0 php-cgi
Aug  1 14:20:36 fullpageos kernel: [  601.273757] [    470]    33   470    46657     1058    86016       78             0 php-cgi
Aug  1 14:20:36 fullpageos kernel: [  601.273762] [    471]    33   471    46618      416    73728       75             0 php-cgi
Aug  1 14:20:36 fullpageos kernel: [  601.273767] [    472]    33   472    46618      416    73728       75             0 php-cgi
Aug  1 14:20:36 fullpageos kernel: [  601.273772] [    473]    33   473    46618      416    73728       75             0 php-cgi
Aug  1 14:20:36 fullpageos kernel: [  601.273777] [    489]     0   489      535       27    28672        5             0 hciattach
Aug  1 14:20:36 fullpageos kernel: [  601.273782] [    493]     0   493     2384      165    45056        2             0 bluetoothd
Aug  1 14:20:36 fullpageos kernel: [  601.273788] [    547]     0   547      724       68    28672       74             0 dhcpcd
Aug  1 14:20:36 fullpageos kernel: [  601.273793] [    555]     0   555     9375      179    61440       67             0 lightdm
Aug  1 14:20:36 fullpageos kernel: [  601.273798] [    559]     0   559     2671      147    45056       64         -1000 sshd
Aug  1 14:20:36 fullpageos kernel: [  601.273803] [    569]     0   569    32634     5951   278528      206             0 Xorg
Aug  1 14:20:36 fullpageos kernel: [  601.273808] [    576]     0   576     1077       19    36864        3             0 agetty
Aug  1 14:20:36 fullpageos kernel: [  601.273813] [    602]     0   602     7476      212    57344        2             0 lightdm
Aug  1 14:20:36 fullpageos kernel: [  601.273818] [    610]  1000   610     3670      266    53248       30             0 systemd
Aug  1 14:20:36 fullpageos kernel: [  601.273824] [    613]  1000   613     4196      254    57344      182             0 (sd-pam)
Aug  1 14:20:36 fullpageos kernel: [  601.273829] [    624]  1000   624     1919       82    36864        0             0 start_gui
Aug  1 14:20:36 fullpageos kernel: [  601.273834] [    633]  1000   633     1604       63    40960        4             0 dbus-daemon
Aug  1 14:20:36 fullpageos kernel: [  601.273839] [    650]  1000   650     1120       78    32768        0             0 ssh-agent
Aug  1 14:20:36 fullpageos kernel: [  601.273845] [    655]  1000   655    21309     1295    90112        0             0 compton
Aug  1 14:20:36 fullpageos kernel: [  601.273850] [    656]  1000   656     3291      142    53248      207             0 matchbox-window
Aug  1 14:20:36 fullpageos kernel: [  601.273855] [    657]  1000   657     1919       69    40960       13             0 run_onepageos
Aug  1 14:20:36 fullpageos kernel: [  601.273860] [    658]  1000   658     8017      495    73728     1010             0 x11vnc
Aug  1 14:20:36 fullpageos kernel: [  601.273865] [    664]  1000   664     1919       45    40960       38             0 start_chromium_
Aug  1 14:20:36 fullpageos kernel: [  601.273871] [    672]  1000   672   188250     7187   823296     7701             0 chromium-browse
Aug  1 14:20:36 fullpageos kernel: [  601.273876] [    693]  1000   693    49582      344   303104     1074             0 chromium-browse
Aug  1 14:20:36 fullpageos kernel: [  601.273881] [    695]  1000   695    49582      433   176128     1060             0 chromium-browse
Aug  1 14:20:36 fullpageos kernel: [  601.273886] [    734]  1000   734    64054     1944   344064     1423           200 chromium-browse
Aug  1 14:20:36 fullpageos kernel: [  601.273891] [    772]  1000   772    85246    21500   614400     3128           200 chromium-browse
Aug  1 14:20:36 fullpageos kernel: [  601.273897] [    905]  1000   905    44048       48   208896     2020           200 chromium-browse
Aug  1 14:20:36 fullpageos kernel: [  601.273902] [    937]  1000   937   218985   107440  1851392     4107           300 chromium-browse
Aug  1 14:20:36 fullpageos kernel: [  601.273907] [   1028]  1000  1028    84986     4123   610304     3894           300 chromium-browse
Aug  1 14:20:36 fullpageos kernel: [  601.273913] [   1234]     0  1234     1405        0    36864      107             0 login
Aug  1 14:20:36 fullpageos kernel: [  601.273918] [   1244]  1000  1244     2123       51    40960      226             0 bash
Aug  1 14:20:36 fullpageos kernel: [  601.273923] [   1285]     0  1285     3090      281    49152        7             0 sshd
Aug  1 14:20:36 fullpageos kernel: [  601.273929] [   1301]  1000  1301     3090      204    45056       20             0 sshd
Aug  1 14:20:36 fullpageos kernel: [  601.273934] [   1304]  1000  1304     2126      267    45056        9             0 bash
Aug  1 14:20:36 fullpageos kernel: [  601.273939] [   1317]  1000  1317     2199      412    45056        9             0 htop
Aug  1 14:20:36 fullpageos kernel: [  601.294339] oom_reaper: reaped process 937 (chromium-browse), now anon-rss:0kB, file-rss:0kB, shmem-rss:314028kB

FullPageOS version: 0.11.0

Screenshot of htop running just before the crash: Screen Shot 2019-08-01 at 6 52 57 PM

And the one after it: Screen Shot 2019-08-01 at 6 53 07 PM

I also tried to take memory heap snapshots using Chrome Inspector and they where not much different in size, i.e. heap was not growing.

rabzu commented 5 years ago

Any news on this?

guysoft commented 5 years ago

Hey, I would need a webpage to test against. You could uncomment the auto refresh on crash script that is in start_chromium_browser

ilyai commented 5 years ago

Actually I was able to reproduce this with playing youtube videos only.

I set fullpageos.txt to https://www.youtube.com/watch?v=Pkh8UtuejGw&list=PLFgquLnL59alCl_2TQvOiD5Vgm1hCaGSI, switched to fullscreen and let youtube autoplay videos. It has been playing for almost 2 hours, but memory was keep growing:

Screen Shot 2019-08-06 at 7 18 33 PM

And finally chrome tab crashed. The last message before the tab was killed: [675:738:0806/162008.827460:WARNING:audio_sync_reader.cc(187)] ASR: No room in socket buffer.: Broken pipe (32)

Full browser's log: chrome_debug.log

ilyai commented 5 years ago

Regarding crash script (/home/pi/scripts/start_chromium_browser) I commented two lines according to the instruction but it didn't seem to be working:

# Standard behavior - runs chrome
#chromium-browser --kiosk --touch-events=enabled --disable-pinch --noerrdialogs --disable-session-crashed-bubble --app=$(head -n 1 /boot/fullpageos.txt | sed -e "s/{serial}/${SERIAL}/g")
#exit;

# Remove the two lines above to enable signage mode - refresh the browser whenever errors are seen in log
BasilHall commented 5 years ago

I can confirm that I have experienced something similar on another Raspbian based Kiosk OS distro with this version of Chromium. We saw it when playing our own internal web page with simple HTML5 Video tags and H264 content on a RPi 3. The video could loop for around 30-45 minutes and then the Chromium tab would crash (and sometimes all of Chromium).

Much like FullPageOS, we had enabled hardware video acceleration in Chromium (overriding the GPU blacklist), and I wonder if this had an effect on things.

I say this to point out that this may be a Chromium issue common to all the Raspbian based distros. I am in the process of migrating our main Kiosk image over to FullPageOS and will retest the issue again and report back.

ilyai commented 5 years ago

@BasilHall I tried testing with --disable-accelerated-video-decode option enabled but nothing changed, Chromium tab crashed as usual.

guysoft commented 5 years ago

You can also try the no-acceleration variant.

ilyai commented 5 years ago

@guysoft I tested no-acceleration version (2019-06-20-fullpageos-no-acceleration-buster-lite-0.11.0.img) and it worked only on RPi 3, and not on RPi 4. On the later it just booted up and showed a grey screen, no chromium or mouse cursor.

guysoft commented 5 years ago

@ilyai Sounds like something has changed with the video memory, I will have to test when I get to it, unless you want to.

The difference between the acceleration and no-acceleration is:

echo "gpu_mem=128" >> /boot/config.txt

located at: https://github.com/guysoft/CustomPiOS/blob/devel/src/modules/gui/start_chroot_script#L39

BasilHall commented 5 years ago

@ilyai You might want to try removing the following flag from Chromium's startup and see if this issue still persists: "--ignore-gpu-blacklist" This will disable a significant portion of the GPU's acceleration in Chromium, but it would be informative to see if the crash still happens or not. This would help us figure out if it is a video driver instability issue in Chromium, or just a more general issue in this version of Chromium in Raspbian.

ilyai commented 5 years ago

@BasilHall I tested with this option (added in /etc/chromium-browser/default) there was no difference. Is there anything else I can try?

guysoft commented 5 years ago

You can try reporting the bug to chromium, I can't see how we can fix it on our end.

ilyai commented 5 years ago

@guysoft Most likely this is due to this bug ( https://bugs.chromium.org/p/chromium/issues/detail?id=969049 ) which is reproducible in v74, the one FullPageOS has, and was fixed in more recent versions of Chromium. So updating Chromium package within FullPageOS should fix this.

guysoft commented 5 years ago

@ilyai Good catch. I guess this means that just restarting on a crash is the only workaround we can do at the moment.

ilyai commented 5 years ago

@guysoft And updating Chromium package to v76 (current) in FullPageOS image isn't planned?

guysoft commented 5 years ago

The Rpi foundation added a Debian package, but they did not update the base image. I guess we could add to the build a stage that updates chromium.

kleintorres commented 4 years ago

Has this been resolved? We plan to use FullPageOS for a Kiosk showing some videos and every distro has the same problem with Chromium

guysoft commented 4 years ago

@kleintorres There are three workarounds that AFAIK fixes all issues I ever encountered. You might not need any of them, it depends on the usage you have with Chromium.

  1. Uncomment the lines explained above
  2. If you are using a heavy page use the non-acceleration variant.
  3. If its possible for you - eliminating cases where internet failes or over situations you can make the Pi restart every night. Removes memory leaking issues.

I have dashboards running for days here.