Black screen after a day and a half

billgod commented 5 years ago

I have looked around and found many black screen issues but non of them seem to match what I have. My screenly would run for about a day and half sometimes 2 days then black screen. I just downloaded the latest 2 days ago, new sd card, different pi 3 and a different power supply. After about a day and a half screen went black again. The screen is on and back lit. just black. Has anyone else suffered this? I find it odd that 2 version, 2 pies same problem?

I am only displaying an image and 2 urls to google calendars.

ALSO.. PLEASE add a reboot option to the web interface!

When filing a bug, please include the following headings if possible. Any example text in this template can be deleted.

Overview of the Issue

A paragraph or two about the issue you're experiencing.

Reproduction Steps

Steps to reproduce this issue, eg:

Press Add Asset
Enter the Asset URL ...
Press "On"
Wait for the asset to show up on the screen

Environment

Raspberry Pi Hardware Version:
Screenly OSE Version:

rusko124 commented 5 years ago

Hey @billgod ! Could you provide the logs, please? Command: journalctl -u screenly-viewer.service

mobilebob commented 5 years ago

I am having the same problem. Running the command for logs shows no logs. A reboot fixes the problem.

Started out with a fresh copy and ran the upgrade process on 9-20-19

ealmonte32 commented 5 years ago

@billgod

Yeah i agree with rusko, without running the journalctl command and showing logs, there is no way to tell what is causing the black screen.. could be so many things..

Regarding the web interface and reboot button.. Screenly has a reboot option, have you installed the latest version?

ealmonte32 commented 5 years ago

@billgod I am seeing this as well on production branch, I have logs from my own Pi and will troubleshoot and see what i find.. the only thing i think i've done to my Pi differently is i use ZRAM to try and optimize the memory usage on the Pi, but i noticed an out of memory error on the log and that could maybe be the cause on mine.. maybe, maybe not, just saying this because it said it closed the parent process of uzbl browser (uzbl-core)..

Sep 24 08:47:15 rpi300we kernel: Out of memory: Kill process 1111 (uzbl-core) score 518 or sacrifice child
Sep 24 08:47:15 rpi300we kernel: Killed process 1111 (uzbl-core) total-vm:1155412kB, anon-rss:234688kB, file-rss:0kB, shmem-rss:0kB

will see what i find through the logs.

TehBeef commented 5 years ago

I'm having this same issue, I have it deployed on BalenaCloud and I get a black screen. The log gives me this:

27.09.19 15:32:47 (-0400) screenly-viewer Showing asset 1204743365.mp4 (video) 27.09.19 15:32:47 (-0400) screenly-viewer current url is file:///tmp/screenly_html/black_page.html 27.09.19 15:32:48 (-0400) screenly-viewer Resource URI is not correct, remote host is not responding or request was rejected.

I've re-installed everything and PNG's and a few url's seem to work, but other then that most things seem to not want to load and I'm stuck. I've also had this issue installing screenly directly on the Pi as the sole software. I'm on the Pi 3 1gb ram.

ealmonte32 commented 5 years ago

27.09.19 15:32:47 (-0400) screenly-viewer Showing asset 1204743365.mp4 (video) 27.09.19 15:32:47 (-0400) screenly-viewer current url is file:///tmp/screenly_html/black_page.html 27.09.19 15:32:48 (-0400) screenly-viewer Resource URI is not correct, remote host is not responding or request was rejected.

Just to confirm: So if you dont add the video it doesn't go black? Can you try and look through the logs more for any errors that are not explicitly from screenly since it does interact with other services of the raspbian OS.. maybe we might have some in common..

I'm assuming you are looking at the System info logs right? if so, can you ssh into the Pi and run journalctl , you will most likely need to press space bar to go through the logs one page at a time, but once you reach the end, you should be able to use a mouse to scroll up and see if you can find any errors or something before it started doing it..

TehBeef commented 5 years ago

Looked through all of logs and I couldn't find anything. It just seems like it's not pulling from the right directory when I add the video. Maybe it's me being stupid and not configuring something right but so far it doesn't seem like that. (It's been a year since the last time I played with anything like this or linux)

billgod commented 5 years ago

Sorry I have been out of town. Can you please provide me with a link on how to ssh into the box? There is no SSH port open and not sure how to enable it.

rusko124 commented 5 years ago

Hey @TehBeef ! Did you increase gpu_mem to 64(min) in device settings ?

rusko124 commented 5 years ago

@billgod I am seeing this as well on production branch, I have logs from my own Pi and will troubleshoot and see what i find.. the only thing i think i've done to my Pi differently is i use ZRAM to try and optimize the memory usage on the Pi, but i noticed an out of memory error on the log and that could maybe be the cause on mine.. maybe, maybe not, just saying this because it said it closed the parent process of uzbl browser (uzbl-core)..
Sep 24 08:47:15 rpi300we kernel: Out of memory: Kill process 1111 (uzbl-core) score 518 or sacrifice child
Sep 24 08:47:15 rpi300we kernel: Killed process 1111 (uzbl-core) total-vm:1155412kB, anon-rss:234688kB, file-rss:0kB, shmem-rss:0kB
will see what i find through the logs.

Good find! I think It's also related to https://github.com/Screenly/screenly-ose/issues/1211

TehBeef commented 5 years ago

Hey @TehBeef ! Did you increase gpu_mem to 64(min) in device settings ?

I had it set to 124 at one point and I had the same issues, I'm not that education in the full extent of that does so I thought I'd leave it off as of now for the sake of troubleshooting.

Edit: also have my log from the other day, I didn't see anything. But again I'm not that educated in this. https://gist.github.com/TehBeef/2d649d1bc06e16d9059052d5c3f442e1

ealmonte32 commented 5 years ago

gpu_mem just assigns the amount of memory to the graphics card of the Pi, 124 is not really a good size to assign because it is not an exponent of 2..I suggest 128 as rusko mentions 64 is minimum, this can be easily done either via the boot config file or through the memory split section raspi-config..

ealmonte32 commented 5 years ago

@TehBeef I see you are using balena, the steps to change gpu_mem are different when running a docker container.. here are the instructions: https://www.balena.io/docs/reference/OS/advanced/

Also, I didnt see any memory errors on your logs, which is interesting because that would rule out it being a memory error.. hmm..

@billgod here are instructions for SSH into a Pi: https://github.com/Screenly/screenly-ose/wiki#enabling-ssh

ealmonte32 commented 5 years ago

Looked through all of logs and I couldn't find anything. It just seems like it's not pulling from the right directory when I add the video. Maybe it's me being stupid and not configuring something right but so far it doesn't seem like that. (It's been a year since the last time I played with anything like this or linux)

What do you mean pulling from the right directory, how are you uploading/assigning the video assets?

TehBeef commented 5 years ago

gpu_mem just assigns the amount of memory to the graphics card of the Pi, 124 is not really a good size to assign because it is not an exponent of 2..I suggest 128 as rusko mentions 64 is minimum, this can be easily done either via the boot config file or through the memory split section raspi-config..

In all honestly I think it was 128, I just can't remember for sure. I'm out of town so my mind is elsewhere.

rusko124 commented 5 years ago

@vpetersson A little bit of research: 1. memory_1 2. memory_3

For 3 hours uzbl-core ate 100MB

Guys, This issue should be resolved after this https://github.com/Screenly/screenly-ose/issues/1093 This is the main task of the current sprint https://github.com/Screenly/screenly-ose/projects/4

vpetersson commented 5 years ago

Yeah, I guess we could in theory add a memory watchdog, but given that we're deprecating UZBL i don't think it's worth the effort. Good research tho @rusko124.

TehBeef commented 5 years ago

I'm a bit uneducated here, mind pointing me in the right direction on how to switch to the experimental browser? I'll see if that was the issue once switched out.

ealmonte32 commented 5 years ago

I'm a bit uneducated here, mind pointing me in the right direction on how to switch to the experimental browser? I'll see if that was the issue once switched out.

@TehBeef

One way, you can just re-run the screenly install script and when it asks if you want to install the experimental branch you say yes by typing y..

bash <(curl -sL https://www.screenly.io/install-ose.sh)

@rusko124 correct me if I'm wrong, but as far as I know the default browser will still be uzbl, which is why you said that you are working on changing the default browser on that experimental branch.. (i believe it will be Qt WebView)

ealmonte32 commented 5 years ago

@rusko124 After 4 days and removed zram use there has still been out of memory errors, but no black screen so far... although I did notice rabbitmq showing process taking up 127M of memory and the time spent in kernel to be very long (time in red)... any ideas?

rusko124 commented 5 years ago

@ealmonte32 Yes, I also ran into a memory issue with celery. In search found that at people solved the issue by celery updating to 4.3.0. Currently I'm testing this, I'll let you know if it helps.

ealmonte32 commented 5 years ago

Oct 02 08:36:42 screenlyatrium kernel: Out of memory: Kill process 11531 (uzbl-core) score 567 or sacrifice child
Oct 02 23:25:41 screenlyatrium kernel: [<8023c358>] (oom_kill_process) from [<8023cc88>] (out_of_memory+0x134/0x36c)
Oct 02 23:25:41 screenlyatrium kernel: [<8023cc88>] (out_of_memory) from [<802430e0>] (__alloc_pages_nodemask+0x1024/0x1178)
Oct 02 23:25:41 screenlyatrium kernel: Tasks state (memory values in pages):
Oct 02 23:25:41 screenlyatrium kernel: Out of memory: Kill process 23997 (uzbl-core) score 569 or sacrifice child
Oct 03 06:55:26 screenlyatrium celery[1062]: [2019-10-03 06:55:25,971: ERROR/ForkPoolWorker-4] Task server.cleanup[5df510bb-e7b9-4826-a463-c75db831ba82] raised unexpected: OSError(12, 'Cannot allocate memory')
Oct 03 06:55:26 screenlyatrium celery[1062]: OSError: [Errno 12] Cannot allocate memory
pi@screenlyatrium:~ $

Now the actual celery worker reported not being able to allocate enough memory.. I'll update to see..

@TehBeef try the following command to see if it also helps you:

sudo pip install --upgrade --no-deps celery

ealmonte32 commented 5 years ago

updated celery to 4.3.0 and still getting memory eaten up by uzbl-core:

pi@screenly600w:~ $ journalctl | grep memory
Oct 03 11:52:35 screenly600w kernel: Virtual kernel memory layout:
Oct 03 11:52:35 screenly600w kernel: vc-sm: Videocore shared memory driver
Oct 03 11:52:35 screenly600w kernel: Freeing unused kernel memory: 1024K
Oct 03 11:52:35 screenly600w kernel: bcm2835_vc_sm_cma_probe: Videocore shared memory driver
Oct 03 13:39:34 screenly600w celery[1107]: [2019-10-03 13:39:32,688: ERROR/ForkPoolWorker-5] Task server.cleanup[76971243-fdaa-455f-b04b-8c13906af81f] raised unexpected: OSError(12, 'Cannot allocate memory')
Oct 03 13:39:34 screenly600w celery[1107]: OSError: [Errno 12] Cannot allocate memory
Oct 03 14:37:25 screenly600w celery[1107]: [2019-10-03 14:37:25,612: ERROR/ForkPoolWorker-5] Task server.cleanup[182921c6-1f3b-46b5-bd42-c37ef019b808] raised unexpected: OSError(12, 'Cannot allocate memory')
Oct 03 14:37:25 screenly600w celery[1107]: OSError: [Errno 12] Cannot allocate memory
Oct 03 15:37:25 screenly600w celery[1107]: [2019-10-03 15:37:25,543: ERROR/ForkPoolWorker-5] Task server.cleanup[30e1c42f-9639-437e-ad02-b97a543d8469] raised unexpected: OSError(12, 'Cannot allocate memory')
Oct 03 15:37:25 screenly600w celery[1107]: OSError: [Errno 12] Cannot allocate memory
Oct 03 16:37:25 screenly600w celery[1107]: [2019-10-03 16:37:25,809: ERROR/ForkPoolWorker-5] Task server.cleanup[c5ac6d38-992e-4a5f-9922-b4dadcf5d79c] raised unexpected: OSError(12, 'Cannot allocate memory')
Oct 03 16:37:25 screenly600w celery[1107]: OSError: [Errno 12] Cannot allocate memory

pi@screenly600w:~ $ pip list
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Package                 Version    
----------------------- -----------
celery                  4.3.0

ealmonte32 commented 5 years ago

Just an update, the Xorg process seems to also be affected which indicates issue with video drivers or video rendering, and I thought, hmm.. since uzbl does not do GPU rendering by default, maybe it needs to be enabled in the variables by adding: set enable_3d_acceleration = 1 in the ./.config/uzbl/config-screenly file.. Maybe this can help?

vpetersson commented 5 years ago

ping @rusko124

ealmonte32 commented 5 years ago

I thought about it and maybe you're right, there shouldn't be time spent on tweaking uzbl anyway.. I think for anyone that wants to have a quick workaround a simple cron job of systemctl restart screenly-viewer.service will keep it running throughout the day/week.. I have been testing and have all screenly pi's restarting the service every 6 hours and I have not seen a black screen since.

I was also seeing rabbitmq(amqp) causing high memory usage so I disabled that service as well..

rusko124 commented 5 years ago

@vpetersson I'm waiting #1239 About RabbitMQ I will try to replace RabbitMQ in favor of Redis and set the expire time for results to half a day. What do you think?

vpetersson commented 5 years ago

@rusko124 I prefer redis over rabbit due to it's smaller footprint in this environment.

rusko124 commented 5 years ago

Hey @ealmonte32 In recent changes Screenly uses Redis instead of Rabbitmq. After 2 days, I doesn't notice any memory leak.

ealmonte32 commented 5 years ago

@rusko124 thanks, will begin testing today..

dingetje commented 5 years ago

For what it's worth, just want to report the same issue. Black screen after random amount of time. The journal log shows frequent Celery errors:

Nov 08 18:27:25 screenly celery[20560]: [2019-11-08 18:27:25,127: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused. Nov 08 18:27:25 screenly celery[20560]: Trying again in 32.00 seconds...

I have Screenly running on a Rpi-0, so both CPU and memory challenged hardware...

For now I've added a cronjob to restart the service every 6 hours

vpetersson commented 5 years ago

@dingetje Are you running on latest master changes?

dingetje commented 5 years ago

@vpetersson no, I'm running the latest release version. Not sure where to find the version number...

vpetersson commented 5 years ago

I see. While it might break things, you might want to switch to the development version as we are working on resolving this issue. However, if you're on a Pi Zero, there is a known issue right now so you might need to wait a bit.

ealmonte32 commented 5 years ago

@rusko124 When users re-run the install screenly-ose script, and they are running on top of previous install, maybe we need to remind them to sudo apt purge rabbitmq-server -y so that it removes the previous service..

This is of course low priority but I am trying to think of a way that re-running the install script via terminal or upgrading via web can remove previous services that were changed to something else and are no longer used.. but then again this will require constant keeping up with packages removed ... ugh no easy way to do this.. =[

rusko124 commented 5 years ago

Hey @ealmonte32 ! Yes, that's a good point. But it is already provided :) https://github.com/Screenly/screenly-ose/blob/master/ansible/roles/system/tasks/main.yml#L177

ealmonte32 commented 5 years ago

@rusko124 But when I ran the install script and kept the production version, the rabbitmq service was still running (after installation finished and reboot).. so something didnt happen during the ansible task..

ealmonte32 commented 5 years ago

I could test by manually installing the rabbitmq-server package now, and re-running the ./screenly/bin/run_upgrade.sh file to show you that after installation finishes, upon reboot you can still find the rabbitmq.service running..

ealmonte32 commented 5 years ago

ugh, i keep getting confused with version named production vs one named development aka master branch... doh, can we add the version name of master next to the development question?

when it asks if we want to use the latest development version, can we add in parentheses (master branch)?

vpetersson commented 5 years ago

Sensible. I've pushed a fix.

ealmonte32 commented 5 years ago

@dingetje If you run the install script again and select yes for development version, it has been tested, and it fixes the memory error.

Maybe we can close this issue now.. unless you notice that it breaks any of the other assets you have?..

P.S. I notice you said you have old rpi, can i suggest you install ZRAM on it?

Running install script again: ./screenly/bin/install.sh

ealmonte32 commented 5 years ago

@rusko124

the master branch still shows uzbl-core memory errors:

pi@screenlypi:~ $ journalctl | grep -i memory
Nov 13 13:05:17 screenlypi kernel: Memory: 886300K/917504K available (8192K kernel code, 653K rwdata, 2224K rodata, 1024K init, 822K bss, 23012K reserved, 8192K cma-reserved)
Nov 13 13:05:17 screenlypi kernel: Virtual kernel memory layout:
Nov 13 13:05:17 screenlypi kernel: vc-sm: Videocore shared memory driver
Nov 13 13:05:17 screenlypi kernel: Freeing unused kernel memory: 1024K
Nov 13 13:05:18 screenlypi kernel: bcm2835_vc_sm_cma_probe: Videocore shared memory driver
Nov 13 13:40:16 screenlypi kernel: [<802605b4>] (oom_kill_process) from [<80260ee4>] (out_of_memory+0x134/0x37c)
Nov 13 13:40:16 screenlypi kernel: [<80260ee4>] (out_of_memory) from [<8026734c>] (__alloc_pages_nodemask+0x1024/0x1178)
Nov 13 13:40:16 screenlypi kernel: Tasks state (memory values in pages):
Nov 13 13:40:16 screenlypi kernel: Out of memory: Kill process 625 (uzbl-core) score 657 or sacrifice child
Nov 13 14:32:45 screenlypi kernel: [<802605b4>] (oom_kill_process) from [<80260ee4>] (out_of_memory+0x134/0x37c)
Nov 13 14:32:45 screenlypi kernel: [<80260ee4>] (out_of_memory) from [<8026734c>] (__alloc_pages_nodemask+0x1024/0x1178)
Nov 13 14:32:45 screenlypi kernel: Tasks state (memory values in pages):
Nov 13 14:32:45 screenlypi kernel: Out of memory: Kill process 1764 (uzbl-core) score 660 or sacrifice child
Nov 13 15:07:11 screenlypi kernel: [<802605b4>] (oom_kill_process) from [<80260ee4>] (out_of_memory+0x134/0x37c)
Nov 13 15:07:11 screenlypi kernel: [<80260ee4>] (out_of_memory) from [<8026734c>] (__alloc_pages_nodemask+0x1024/0x1178)
Nov 13 15:07:11 screenlypi kernel: Tasks state (memory values in pages):
Nov 13 15:07:11 screenlypi kernel: Out of memory: Kill process 3041 (uzbl-core) score 638 or sacrifice child
Nov 13 15:38:30 screenlypi kernel: [<802605b4>] (oom_kill_process) from [<80260ee4>] (out_of_memory+0x134/0x37c)
Nov 13 15:38:30 screenlypi kernel: [<80260ee4>] (out_of_memory) from [<8026734c>] (__alloc_pages_nodemask+0x1024/0x1178)
Nov 13 15:38:30 screenlypi kernel: Tasks state (memory values in pages):
Nov 13 15:38:30 screenlypi kernel: Out of memory: Kill process 4501 (uzbl-core) score 632 or sacrifice child

The production version with removed celery service does not show me any memory errors at all, which i have set this way in 3 more Pi's. I will test now by disabling the celery worker from this one:

pi@screenlypi:~ $ cat ./version.md
Screenly version: master@bc64b08

which will then determine if the celery worker is in fact causing the memory error, but what does celery worker have to do with uzbl? 🤦‍♂

ealmonte32 commented 5 years ago

@rusko124

on this screenshot Xorg is invoking the oom-killer because of low memory, and of course uzbl-core is always going to be the process that is using the most memory and this will be why it gets killed.. i assume that when celery was also consuming all the memory, the system did the same and killed uzl-core..

i am comparing two screenly on the same pi but with different branches, and the Xorg (same package version) is spending a lot of time in the CPU, whether on the master branch or production.. in the master branch i get the following:

Nov 13 18:34:58 screenlypi kernel: Xorg invoked oom-killer: gfp_mask=0x6000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=0
Nov 13 18:34:58 screenlypi kernel: Xorg cpuset=/ mems_allowed=0
Nov 13 18:34:58 screenlypi kernel: CPU: 0 PID: 461 Comm: Xorg Tainted: G         C        4.19.83-v7+ #1277
Nov 13 18:34:58 screenlypi kernel: Hardware name: BCM2835
Nov 13 18:34:58 screenlypi kernel: [<801120c0>] (unwind_backtrace) from [<8010d5f4>] (show_stack+0x20/0x24)
Nov 13 18:34:58 screenlypi kernel: [<8010d5f4>] (show_stack) from [<80844280>] (dump_stack+0xd4/0x118)
Nov 13 18:34:58 screenlypi kernel: [<80844280>] (dump_stack) from [<8026124c>] (dump_header+0x80/0x250)
Nov 13 18:34:58 screenlypi kernel: [<8026124c>] (dump_header) from [<802605b4>] (oom_kill_process+0x358/0x3a8)
Nov 13 18:34:58 screenlypi kernel: [<802605b4>] (oom_kill_process) from [<80260ee4>] (out_of_memory+0x134/0x37c)
Nov 13 18:34:58 screenlypi kernel: [<80260ee4>] (out_of_memory) from [<8026734c>] (__alloc_pages_nodemask+0x1024/0x1178)
Nov 13 18:34:58 screenlypi kernel: [<8026734c>] (__alloc_pages_nodemask) from [<802674c4>] (__get_free_pages+0x24/0x70)
Nov 13 18:34:58 screenlypi kernel: [<802674c4>] (__get_free_pages) from [<80347824>] (proc_pid_cmdline_read+0x1d0/0x3c8)
Nov 13 18:34:58 screenlypi kernel: [<80347824>] (proc_pid_cmdline_read) from [<802cfd9c>] (__vfs_read+0x48/0x16c)
Nov 13 18:34:58 screenlypi kernel: [<802cfd9c>] (__vfs_read) from [<802cff5c>] (vfs_read+0x9c/0x168)
Nov 13 18:34:58 screenlypi kernel: [<802cff5c>] (vfs_read) from [<802d05a4>] (ksys_read+0x74/0xe8)
Nov 13 18:34:58 screenlypi kernel: [<802d05a4>] (ksys_read) from [<802d0630>] (sys_read+0x18/0x1c)
Nov 13 18:34:58 screenlypi kernel: [<802d0630>] (sys_read) from [<80101000>] (ret_fast_syscall+0x0/0x28)
Nov 13 18:34:58 screenlypi kernel: Exception stack(0xb4ef7fa8 to 0xb4ef7ff0)
Nov 13 18:34:58 screenlypi kernel: 7fa0:                   7ea83af0 011c5550 00000019 7ea83af0 00001001 00000000
Nov 13 18:34:58 screenlypi kernel: 7fc0: 7ea83af0 011c5550 76f40ce8 00000003 7ea83af8 00000001 00000019 00201a00
Nov 13 18:34:58 screenlypi kernel: 7fe0: 00000000 7ea83ae0 00000000 76a86414
Nov 13 18:34:58 screenlypi kernel: Mem-Info:
Nov 13 18:34:58 screenlypi kernel: active_anon:30398 inactive_anon:30435 isolated_anon:0
                                    active_file:328 inactive_file:1071 isolated_file:32
                                    unevictable:3 dirty:0 writeback:0 unstable:0
                                    slab_reclaimable:2372 slab_unreclaimable:4770
                                    mapped:48389 shmem:48029 pagetables:1164 bounce:0
                                    free:4221 free_pcp:34 free_cma:791
Nov 13 18:34:58 screenlypi kernel: Node 0 active_anon:121592kB inactive_anon:121740kB active_file:1312kB inactive_file:4284kB unevictable:12kB isolated(anon):0kB isolated(file):128kB mapped:193556k
Nov 13 18:34:58 screenlypi kernel: Normal free:16884kB min:16384kB low:20480kB high:24576kB active_anon:121592kB inactive_anon:121740kB active_file:612kB inactive_file:4452kB unevictable:12kB write
Nov 13 18:34:58 screenlypi kernel: lowmem_reserve[]: 0 0
Nov 13 18:34:58 screenlypi kernel: Normal: 200*4kB (UMEHC) 251*8kB (UMEHC) 197*16kB (UMEHC) 86*32kB (UMEHC) 53*64kB (UEHC) 17*128kB (UEHC) 3*256kB (UEH) 3*512kB (UE) 1*1024kB (C) 0*2048kB 0*4096kB
Nov 13 18:34:58 screenlypi kernel: 49511 total pagecache pages
Nov 13 18:34:58 screenlypi kernel: 40 pages in swap cache
Nov 13 18:34:58 screenlypi kernel: Swap cache stats: add 1278737, delete 1278657, find 8904/603264
Nov 13 18:34:58 screenlypi kernel: Free swap  = 0kB
Nov 13 18:34:58 screenlypi kernel: Total swap = 895504kB
Nov 13 18:34:58 screenlypi kernel: 229376 pages RAM
Nov 13 18:34:58 screenlypi kernel: 0 pages HighMem/MovableOnly
Nov 13 18:34:58 screenlypi kernel: 5497 pages reserved
Nov 13 18:34:58 screenlypi kernel: 2048 pages cma reserved
Nov 13 18:34:58 screenlypi kernel: Tasks state (memory values in pages):
Nov 13 18:34:58 screenlypi kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Nov 13 18:34:58 screenlypi kernel: [    103]     0   103     3112        2    18432      102             0 systemd-journal
Nov 13 18:34:58 screenlypi kernel: [    290]     0   290     1918        0    16384      771             0 haveged
Nov 13 18:34:58 screenlypi kernel: [    294]   100   294     4300        0    20480      120             0 systemd-timesyn
Nov 13 18:34:58 screenlypi kernel: [    349]   105   349     1640        2    18432      137          -900 dbus-daemon
Nov 13 18:34:58 screenlypi kernel: [    354]     0   354     6900        0    18432       56             0 rngd
Nov 13 18:34:58 screenlypi kernel: [    361]     0   361    13256        0    38912      347             0 ModemManager
Nov 13 18:34:58 screenlypi kernel: [    362]     0   362     1845        0    16384      121             0 systemd-logind
Nov 13 18:34:58 screenlypi kernel: [    363]  1000   363    14934        0    45056     2523             0 python
Nov 13 18:34:58 screenlypi kernel: [    364]     0   364     1324        0    14336       66             0 cron
Nov 13 18:34:58 screenlypi kernel: [    365] 65534   365     1324        0    12288       59             0 thd
Nov 13 18:34:58 screenlypi kernel: [    366]     0   366     5939        0    24576      249             0 rsyslogd
Nov 13 18:34:58 screenlypi kernel: [    369]     0   369     2500        0    18432      108             0 wpa_supplicant
Nov 13 18:34:58 screenlypi kernel: [    402]     0   402     9418        0    28672      152             0 polkitd
Nov 13 18:34:58 screenlypi kernel: [    412]     0   412     2643        0    18432      138             0 wpa_supplicant
Nov 13 18:34:58 screenlypi kernel: [    449]     0   449      734        0    10240      116             0 dhcpcd
Nov 13 18:34:58 screenlypi kernel: [    461]     0   461   104227    28332   290816     2057             0 Xorg
Nov 13 18:34:58 screenlypi kernel: [    462]  1000   462     3046        0    18432      266             0 matchbox-window
Nov 13 18:34:58 screenlypi kernel: [    463]     0   463     1423        0    16384       60             0 vncserver-x11-s
Nov 13 18:34:58 screenlypi kernel: [    471]     0   471     3690        0    20480      183         -1000 systemd-udevd
Nov 13 18:34:58 screenlypi kernel: [    479]     0   479     5814      109    30720      914             0 vncserver-x11-c
Nov 13 18:34:58 screenlypi kernel: [    491]  1000   491    14565      292    65536     9041             0 python
Nov 13 18:34:58 screenlypi kernel: [    496]     0   496     1684        0    12288      134             0 nginx
Nov 13 18:34:58 screenlypi kernel: [    497]    33   497     1730        0    14336      212             0 nginx
Nov 13 18:34:58 screenlypi kernel: [    498]    33   498     1730       16    14336      161             0 nginx
Nov 13 18:34:58 screenlypi kernel: [    499]    33   499     1730       19    14336      158             0 nginx
Nov 13 18:34:58 screenlypi kernel: [    500]    33   500     1730       17    14336      160             0 nginx
Nov 13 18:34:58 screenlypi kernel: [    508]     0   508     1046        0    14336       31             0 agetty
Nov 13 18:34:58 screenlypi kernel: [    509]     0   509      991        0    12288       32             0 agetty
Nov 13 18:34:58 screenlypi kernel: [    512]     0   512     2551        0    16384      136         -1000 sshd
Nov 13 18:34:58 screenlypi kernel: [    516]   108   516     7423      109    16384      166             0 redis-server
Nov 13 18:34:58 screenlypi kernel: [    531]     0   531     2131        2    14336      115             0 vncagent
Nov 13 18:34:58 screenlypi kernel: [    549]  1000   549    32026      207    81920     6306             0 python
Nov 13 18:34:58 screenlypi kernel: [    572]  1000   572    23796      184    75776     9631             0 python
Nov 13 18:34:58 screenlypi kernel: [    581]  1000   581   332058    51113  1189888   183240             0 uzbl-core
Nov 13 18:34:58 screenlypi kernel: [    586]  1000   586     3382      101    22528      853             0 uzbl-event-mana
Nov 13 18:34:58 screenlypi kernel: [    666]     0   666     2881        0    18432      198             0 sshd
Nov 13 18:34:58 screenlypi kernel: [    671]  1000   671     2414        0    16384      177             0 systemd
Nov 13 18:34:58 screenlypi kernel: [    674]  1000   674     2820        0    20480      310             0 (sd-pam)
Nov 13 18:34:58 screenlypi kernel: [    681]  1000   681     2914        0    18432      208             0 sshd
Nov 13 18:34:58 screenlypi kernel: [    684]  1000   684     1720        2    16384      482             0 bash
Nov 13 18:34:58 screenlypi kernel: Out of memory: Kill process 581 (uzbl-core) score 524 or sacrifice child
Nov 13 18:34:58 screenlypi kernel: Killed process 581 (uzbl-core) total-vm:1328232kB, anon-rss:32980kB, file-rss:148kB, shmem-rss:171324kB
Nov 13 18:34:59 screenlypi kernel: oom_reaper: reaped process 581 (uzbl-core), now anon-rss:0kB, file-rss:0kB, shmem-rss:148792kB

but on the production i do not get the oom-killer invoked, but the Xorg does stay in CPU for long time and becomes red.

On a raspberry pi that is not running screenly, i have it running chromium on the solaredge site, and i just noticed the GPU flag process on chromium spending time on CPU as well.. so this is all probably GPU related.. but that rpi is running buster and not stretch, the xserver-xorg version is just +b8 of difference.. i am trying to think of how to best approach this issue since it is not specifically screenly related.. mostly to do with handling of memory.. 🤦‍♂

vpetersson commented 5 years ago

What memory split are you using?

ealmonte32 commented 5 years ago

so for testing I changed from my normal default of 128 to 256, it crashed even faster, went down to 64, has been much stabler... still not seen black screen in master branch.

getting python invoking oom-killer but the kernel takes care of restarting all processes so the user would not see anything unless they go looking through logs..

i am pasting the following logs just for documenting, i dont expect anyone to read through them all but just for documentation..

Nov 14 20:30:59 screenlypi python[621]: current url is https://docs.google.com/presentation/d/e/XXXXXXXX/pub?start=true
Nov 14 20:30:59 screenlypi python[621]: Sleeping for 60
Nov 14 20:32:20 screenlypi kernel: python invoked oom-killer: gfp_mask=0x7080c0(GFP_KERNEL_ACCOUNT|__GFP_ZERO), nodemask=(null), order=1, oom_score_adj=0
Nov 14 20:32:20 screenlypi kernel: python cpuset=/ mems_allowed=0
Nov 14 20:32:20 screenlypi kernel: CPU: 3 PID: 621 Comm: python Tainted: G         C        4.19.83-v7+ #1277
Nov 14 20:32:20 screenlypi kernel: Hardware name: BCM2835
Nov 14 20:32:20 screenlypi kernel: [<801120c0>] (unwind_backtrace) from [<8010d5f4>] (show_stack+0x20/0x24)
Nov 14 20:32:20 screenlypi kernel: [<8010d5f4>] (show_stack) from [<80844280>] (dump_stack+0xd4/0x118)
Nov 14 20:32:20 screenlypi kernel: [<80844280>] (dump_stack) from [<8026124c>] (dump_header+0x80/0x250)
Nov 14 20:32:20 screenlypi kernel: [<8026124c>] (dump_header) from [<802605b4>] (oom_kill_process+0x358/0x3a8)
Nov 14 20:32:20 screenlypi kernel: [<802605b4>] (oom_kill_process) from [<80260ee4>] (out_of_memory+0x134/0x37c)
Nov 14 20:32:20 screenlypi kernel: [<80260ee4>] (out_of_memory) from [<8026734c>] (__alloc_pages_nodemask+0x1024/0x1178)
Nov 14 20:32:20 screenlypi kernel: [<8026734c>] (__alloc_pages_nodemask) from [<8011e008>] (copy_process.part.5+0x1f8/0x1b08)
Nov 14 20:32:20 screenlypi kernel: [<8011e008>] (copy_process.part.5) from [<8011fae0>] (_do_fork+0xd8/0x438)
Nov 14 20:32:20 screenlypi kernel: [<8011fae0>] (_do_fork) from [<8011ff64>] (sys_clone+0x34/0x3c)
Nov 14 20:32:20 screenlypi kernel: [<8011ff64>] (sys_clone) from [<80101000>] (ret_fast_syscall+0x0/0x28)
Nov 14 20:32:20 screenlypi kernel: Exception stack(0xb61ddfa8 to 0xb61ddff0)
Nov 14 20:32:20 screenlypi kernel: dfa0:                   76f5d068 7ea11638 01200011 00000000 00000000 00000000
Nov 14 20:32:20 screenlypi kernel: dfc0: 76f5d068 7ea11638 00000000 00000078 00000001 76e14000 00000000 7ea11674
Nov 14 20:32:20 screenlypi kernel: dfe0: 76f5d000 7ea11638 76d752e8 76d75178
Nov 14 20:32:20 screenlypi kernel: Mem-Info:
Nov 14 20:32:20 screenlypi kernel: active_anon:28454 inactive_anon:28460 isolated_anon:0
                                    active_file:119 inactive_file:195 isolated_file:27
                                    unevictable:3 dirty:0 writeback:0 unstable:0
                                    slab_reclaimable:2321 slab_unreclaimable:4982
                                    mapped:11248 shmem:39916 pagetables:1175 bounce:0
                                    free:6037 free_pcp:424 free_cma:1439
Nov 14 20:32:20 screenlypi kernel: Node 0 active_anon:113816kB inactive_anon:113840kB active_file:476kB inactive_file:780kB unevictable:12kB isolated(anon):0kB isolated(file):0kB mapped:44992kB dir
Nov 14 20:32:20 screenlypi kernel: Normal free:24148kB min:16384kB low:20480kB high:24576kB active_anon:113816kB inactive_anon:113840kB active_file:44kB inactive_file:384kB unevictable:12kB writepe
Nov 14 20:32:20 screenlypi kernel: lowmem_reserve[]: 0 0
Nov 14 20:32:20 screenlypi kernel: Normal: 319*4kB (UMEHC) 626*8kB (UMEHC) 181*16kB (UMEHC) 63*32kB (UMEHC) 33*64kB (UMEHC) 10*128kB (UEH) 5*256kB (UHC) 3*512kB (UHC) 2*1024kB (UE) 0*2048kB 1*4096k
Nov 14 20:32:20 screenlypi kernel: 40321 total pagecache pages
Nov 14 20:32:20 screenlypi kernel: 53 pages in swap cache
Nov 14 20:32:20 screenlypi kernel: Swap cache stats: add 1165333, delete 1165248, find 8615/520494
Nov 14 20:32:20 screenlypi kernel: Free swap  = 0kB
Nov 14 20:32:20 screenlypi kernel: Total swap = 960464kB
Nov 14 20:32:20 screenlypi kernel: 245760 pages RAM
Nov 14 20:32:20 screenlypi kernel: 0 pages HighMem/MovableOnly
Nov 14 20:32:20 screenlypi kernel: 5641 pages reserved
Nov 14 20:32:20 screenlypi kernel: 2048 pages cma reserved
Nov 14 20:32:20 screenlypi kernel: Tasks state (memory values in pages):
Nov 14 20:32:20 screenlypi kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Nov 14 20:32:20 screenlypi kernel: [    104]     0   104    10315        0    24576      108             0 systemd-journal
Nov 14 20:32:20 screenlypi kernel: [    274]     0   274     1918        2    16384      771             0 haveged
Nov 14 20:32:20 screenlypi kernel: [    276]   100   276     4300        0    18432      111             0 systemd-timesyn
Nov 14 20:32:20 screenlypi kernel: [    322]     0   322     5939        0    22528      241             0 rsyslogd
Nov 14 20:32:20 screenlypi kernel: [    325]   105   325     1624        2    16384      114          -900 dbus-daemon
Nov 14 20:32:20 screenlypi kernel: [    350]  1000   350    14954      253    40960     2249             0 python
Nov 14 20:32:20 screenlypi kernel: [    351]     0   351     1324        0    12288       57             0 cron
Nov 14 20:32:20 screenlypi kernel: [    355]     0   355     1845        0    16384      112             0 systemd-logind
Nov 14 20:32:20 screenlypi kernel: [    356]     0   356    13228        0    36864      346             0 ModemManager
Nov 14 20:32:20 screenlypi kernel: [    362]     0   362     2500       12    18432       84             0 wpa_supplicant
Nov 14 20:32:20 screenlypi kernel: [    364] 65534   364     1324        0    14336       61             0 thd
Nov 14 20:32:20 screenlypi kernel: [    369]     0   369     6900        0    16384       48             0 rngd
Nov 14 20:32:20 screenlypi kernel: [    407]     0   407     9418        0    24576      152             0 polkitd
Nov 14 20:32:20 screenlypi kernel: [    416]     0   416     2644       17    16384      120             0 wpa_supplicant
Nov 14 20:32:20 screenlypi kernel: [    492]     0   492      734        0    10240      123             0 dhcpcd
Nov 14 20:32:20 screenlypi kernel: [    495]     0   495    71687     6790   241664     1956             0 Xorg
Nov 14 20:32:20 screenlypi kernel: [    496]  1000   496     3046       12    22528      234             0 matchbox-window
Nov 14 20:32:20 screenlypi kernel: [    502]     0   502     1423        0    12288       60             0 vncserver-x11-s
Nov 14 20:32:20 screenlypi kernel: [    510]     0   510     3690        0    16384      204         -1000 systemd-udevd
Nov 14 20:32:20 screenlypi kernel: [    517]     0   517     5814       46    30720      911             0 vncserver-x11-c
Nov 14 20:32:20 screenlypi kernel: [    532]  1000   532    14565      161    65536     9049             0 python
Nov 14 20:32:20 screenlypi kernel: [    540]     0   540     1684        0    12288      134             0 nginx
Nov 14 20:32:20 screenlypi kernel: [    541]    33   541     1730        0    14336      176             0 nginx
Nov 14 20:32:20 screenlypi kernel: [    542]    33   542     1730       19    14336      158             0 nginx
Nov 14 20:32:20 screenlypi kernel: [    543]    33   543     1730       19    14336      158             0 nginx
Nov 14 20:32:20 screenlypi kernel: [    544]    33   544     1730       19    14336      158             0 nginx
Nov 14 20:32:20 screenlypi kernel: [    562]     0   562     2551        0    20480      135         -1000 sshd
Nov 14 20:32:20 screenlypi kernel: [    566]     0   566     1046        0    10240       31             0 agetty
Nov 14 20:32:20 screenlypi kernel: [    567]     0   567      991        0    10240       32             0 agetty
Nov 14 20:32:20 screenlypi kernel: [    577]   108   577     7423       28    18432      166             0 redis-server
Nov 14 20:32:20 screenlypi kernel: [    603]     0   603     2131        2    16384      115             0 vncagent
Nov 14 20:32:20 screenlypi kernel: [    621]  1000   621    31962      612    81920     5793             0 python
Nov 14 20:32:20 screenlypi kernel: [    644]  1000   644    14682      222    67584     8998             0 python
Nov 14 20:32:20 screenlypi kernel: [    653]  1000   653   330902    22330  1292288   199708             0 uzbl-core
Nov 14 20:32:20 screenlypi kernel: [    658]  1000   658     3382      374    22528      576             0 uzbl-event-mana
Nov 14 20:32:20 screenlypi kernel: [    690]     0   690     2881        0    18432      180             0 sshd
Nov 14 20:32:20 screenlypi kernel: [    695]  1000   695     2414        0    18432      181             0 systemd
Nov 14 20:32:20 screenlypi kernel: [    698]  1000   698     2805        0    16384      304             0 (sd-pam)
Nov 14 20:32:20 screenlypi kernel: [    705]  1000   705     2916        0    18432      205             0 sshd
Nov 14 20:32:20 screenlypi kernel: [    708]  1000   708     1522        2    12288      306             0 bash
Nov 14 20:32:20 screenlypi kernel: Out of memory: Kill process 653 (uzbl-core) score 463 or sacrifice child
Nov 14 20:32:20 screenlypi kernel: Killed process 653 (uzbl-core) total-vm:1323608kB, anon-rss:44496kB, file-rss:0kB, shmem-rss:44824kB
Nov 14 20:32:20 screenlypi kernel: oom_reaper: reaped process 653 (uzbl-core), now anon-rss:0kB, file-rss:0kB, shmem-rss:28584kB
Nov 14 20:32:21 screenlypi python[621]: Generating asset-list...
Nov 14 20:32:21 screenlypi python[621]: Showing asset 600w slides (webpage)
Nov 14 20:32:21 screenlypi python[621]: browser found dead, restarting
Nov 14 20:32:21 screenlypi python[621]: Loading browser...
Nov 14 20:32:21 screenlypi python[621]: Viewer crashed.
Nov 14 20:32:21 screenlypi python[621]: Traceback (most recent call last):
Nov 14 20:32:21 screenlypi python[621]:   File "/home/pi/screenly/viewer.py", line 580, in <module>
Nov 14 20:32:21 screenlypi python[621]:     main()
Nov 14 20:32:21 screenlypi python[621]:   File "/home/pi/screenly/viewer.py", line 575, in main
Nov 14 20:32:21 screenlypi python[621]:     asset_loop(scheduler)
Nov 14 20:32:21 screenlypi python[621]:   File "/home/pi/screenly/viewer.py", line 456, in asset_loop
Nov 14 20:32:21 screenlypi python[621]:     browser_url(uri)
Nov 14 20:32:21 screenlypi python[621]:   File "/home/pi/screenly/viewer.py", line 328, in browser_url
Nov 14 20:32:21 screenlypi python[621]:     browser_send('uri ' + escaped_url, cb=cb)
Nov 14 20:32:21 screenlypi python[621]:   File "/home/pi/screenly/viewer.py", line 307, in browser_send
Nov 14 20:32:21 screenlypi python[621]:     load_browser()
Nov 14 20:32:21 screenlypi python[621]:   File "/home/pi/screenly/viewer.py", line 262, in load_browser
Nov 14 20:32:21 screenlypi python[621]:     if browser:
Nov 14 20:32:21 screenlypi python[621]:   File "/usr/local/lib/python2.7/dist-packages/sh.py", line 334, in __len__
Nov 14 20:32:21 screenlypi python[621]:     return len(str(self))
Nov 14 20:32:21 screenlypi python[621]:   File "/usr/local/lib/python2.7/dist-packages/sh.py", line 370, in __str__
Nov 14 20:32:21 screenlypi python[621]:     else: return unicode(self).encode(self.call_args["encoding"])
Nov 14 20:32:21 screenlypi python[621]:   File "/usr/local/lib/python2.7/dist-packages/sh.py", line 373, in __unicode__
Nov 14 20:32:21 screenlypi python[621]:     if self.process and self.stdout:
Nov 14 20:32:21 screenlypi python[621]:   File "/usr/local/lib/python2.7/dist-packages/sh.py", line 316, in stdout
Nov 14 20:32:21 screenlypi python[621]:     self.wait()
Nov 14 20:32:21 screenlypi python[621]:   File "/usr/local/lib/python2.7/dist-packages/sh.py", line 295, in wait
Nov 14 20:32:21 screenlypi python[621]:     self._handle_exit_code(self.process.wait())
Nov 14 20:32:21 screenlypi python[621]:   File "/usr/local/lib/python2.7/dist-packages/sh.py", line 309, in _handle_exit_code
Nov 14 20:32:21 screenlypi python[621]:     self.process.stderr
Nov 14 20:32:21 screenlypi python[621]: SignalException_9:
Nov 14 20:32:21 screenlypi python[621]:   RAN: '/usr/bin/uzbl-browser --config=- --uri=file:///tmp/screenly_html/black_page.html --print-events'
Nov 14 20:32:21 screenlypi python[621]:   STDOUT:
Nov 14 20:32:21 screenlypi python[621]: EVENT [653] INSTANCE_START 653
Nov 14 20:32:21 screenlypi python[621]: EVENT [653] BUILTINS 'back forward scroll reload reload_ign_cache stop zoom_in zoom_out toggle_zoom_type uri js script toggle_status spawn sync_spawn sync_sp
Nov 14 20:32:21 screenlypi python[621]: EVENT [653] VARIABLE_SET status_format str '<b>@[@TITLE]@</b> - @[@uri]@ - <span foreground="#bbb">@NAME</span>'
Nov 14 20:32:21 screenlypi python[621]: EVENT [653] VARIABLE_S... (2395915 more, please see e.stdout)
Nov 14 20:32:21 screenlypi python[621]:   STDERR:
Nov 14 20:32:21 screenlypi python[621]: ** Message: console message:  @0: Unrecognized Content-Security-Policy directive 'base-uri'.
Nov 14 20:32:21 screenlypi python[621]: ** Message: console message: https://docs.google.com/presentation/d/e/XXXXXXXX/
Nov 14 20:32:21 screenlypi python[621]: ** Message: console message:  @0: Unrecognized Content-Security-Policy directive 'base-uri'.
Nov 14 20:32:21 screenlypi python[621]: ** Message: console message: https://docs.google.com/presentation/d/e/XXXXXXXX-XXXXXXXX/
Nov 14 20:32:21 screenlypi python[621]: ** Messa... (60013 more, please see e.stderr)
Nov 14 20:32:21 screenlypi python[621]: Traceback (most recent call last):
Nov 14 20:32:21 screenlypi python[621]:   File "/home/pi/screenly/viewer.py", line 580, in <module>
Nov 14 20:32:21 screenlypi python[621]:     main()
Nov 14 20:32:21 screenlypi python[621]:   File "/home/pi/screenly/viewer.py", line 575, in main
Nov 14 20:32:21 screenlypi python[621]:     asset_loop(scheduler)
Nov 14 20:32:21 screenlypi python[621]:   File "/home/pi/screenly/viewer.py", line 456, in asset_loop
Nov 14 20:32:21 screenlypi python[621]:     browser_url(uri)
Nov 14 20:32:21 screenlypi python[621]:   File "/home/pi/screenly/viewer.py", line 328, in browser_url
Nov 14 20:32:21 screenlypi python[621]:     browser_send('uri ' + escaped_url, cb=cb)
Nov 14 20:32:21 screenlypi python[621]:   File "/home/pi/screenly/viewer.py", line 307, in browser_send
Nov 14 20:32:21 screenlypi python[621]:     load_browser()
Nov 14 20:32:21 screenlypi python[621]:   File "/home/pi/screenly/viewer.py", line 262, in load_browser
Nov 14 20:32:21 screenlypi python[621]:     if browser:
Nov 14 20:32:21 screenlypi python[621]:   File "/usr/local/lib/python2.7/dist-packages/sh.py", line 334, in __len__
Nov 14 20:32:21 screenlypi python[621]:     return len(str(self))
Nov 14 20:32:21 screenlypi python[621]:   File "/usr/local/lib/python2.7/dist-packages/sh.py", line 370, in __str__
Nov 14 20:32:21 screenlypi python[621]:     else: return unicode(self).encode(self.call_args["encoding"])
Nov 14 20:32:21 screenlypi python[621]:   File "/usr/local/lib/python2.7/dist-packages/sh.py", line 373, in __unicode__
Nov 14 20:32:21 screenlypi python[621]:     if self.process and self.stdout:
Nov 14 20:32:21 screenlypi python[621]:   File "/usr/local/lib/python2.7/dist-packages/sh.py", line 316, in stdout
Nov 14 20:32:21 screenlypi python[621]:     self.wait()
Nov 14 20:32:21 screenlypi python[621]:   File "/usr/local/lib/python2.7/dist-packages/sh.py", line 295, in wait
Nov 14 20:32:21 screenlypi python[621]:     self._handle_exit_code(self.process.wait())
Nov 14 20:32:21 screenlypi python[621]:   File "/usr/local/lib/python2.7/dist-packages/sh.py", line 309, in _handle_exit_code
Nov 14 20:32:21 screenlypi python[621]:     self.process.stderr
Nov 14 20:32:21 screenlypi python[621]: sh.SignalException_9:
Nov 14 20:32:21 screenlypi python[621]:   RAN: '/usr/bin/uzbl-browser --config=- --uri=file:///tmp/screenly_html/black_page.html --print-events'
Nov 14 20:32:21 screenlypi python[621]:   STDOUT:
Nov 14 20:32:21 screenlypi python[621]: EVENT [653] INSTANCE_START 653
Nov 14 20:32:21 screenlypi python[621]: EVENT [653] BUILTINS 'back forward scroll reload reload_ign_cache stop zoom_in zoom_out toggle_zoom_type uri js script toggle_status spawn sync_spawn sync_sp
Nov 14 20:32:21 screenlypi python[621]: EVENT [653] VARIABLE_SET status_format str '<b>@[@TITLE]@</b> - @[@uri]@ - <span foreground="#bbb">@NAME</span>'
Nov 14 20:32:21 screenlypi python[621]: EVENT [653] VARIABLE_S... (2395915 more, please see e.stdout)
Nov 14 20:32:21 screenlypi python[621]:   STDERR:
Nov 14 20:32:21 screenlypi python[621]: ** Message: console message:  @0: Unrecognized Content-Security-Policy directive 'base-uri'.
Nov 14 20:32:21 screenlypi python[621]: ** Message: console message: https://docs.google.com/presentation/d/e/XXXXXXXX-XXXXXXXX/
Nov 14 20:32:21 screenlypi python[621]: ** Message: console message:  @0: Unrecognized Content-Security-Policy directive 'base-uri'.
Nov 14 20:32:21 screenlypi python[621]: ** Message: console message: https://docs.google.com/presentation/d/e/XXXXXXXX-XXXXXXXX/
Nov 14 20:32:21 screenlypi python[621]: ** Messa... (60013 more, please see e.stderr)
Nov 14 20:32:22 screenlypi systemd[1]: screenly-viewer.service: Main process exited, code=exited, status=1/FAILURE
Nov 14 20:32:22 screenlypi systemd[1]: screenly-viewer.service: Unit entered failed state.
Nov 14 20:32:22 screenlypi systemd[1]: screenly-viewer.service: Failed with result 'exit-code'.
Nov 14 20:32:22 screenlypi systemd[1]: screenly-viewer.service: Service hold-off time over, scheduling restart.
Nov 14 20:32:22 screenlypi systemd[1]: Stopped Screenly Viewer.
Nov 14 20:32:22 screenlypi systemd[1]: Starting Screenly Viewer...
Nov 14 20:32:22 screenlypi systemd[1]: Started Screenly Viewer.
Nov 14 20:32:26 screenlypi python[4354]: /usr/local/lib/python2.7/dist-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.25.1) or chardet (3.0.4) doesn't match a supported ver
Nov 14 20:32:26 screenlypi python[4354]:   RequestsDependencyWarning)
Nov 14 20:32:28 screenlypi python[4354]: Generating asset-list...
Nov 14 20:32:28 screenlypi python[4354]: browser found dead, restarting
Nov 14 20:32:28 screenlypi python[4354]: Loading browser...
Nov 14 20:32:28 screenlypi python[4354]: Browser loading file:///tmp/screenly_html/black_page.html. Running as PID 4372.
Nov 14 20:32:30 screenlypi python[4354]: current url is file:///tmp/screenly_html/black_page.html
Nov 14 20:32:31 screenlypi python[4354]: Generating asset-list...

ealmonte32 commented 5 years ago

@Jrdnkasparek A few questions:

when you look at the logs, are you looking after the black screen happens, or since you are saying it becomes unresponsive and cant ssh, are you manually rebooting and then checking the logs? because as far as i know unless you are doing something different, that would of course not show previous logs due to reboot.

(i can suggest a simple checkwifi.sh script that would check if the pi has internet connectivity, and if not, it restarts the networking.service and screenly-viewer.service so that it sort of tries to repair itself)

did you try at least on one to run screenly install (./screenly/bin/install.sh) and select the development branch/version?
if you run a systemctl status do you see any of the following there: rabbitmq, celery, running?

thx

ealmonte32 commented 5 years ago

Black screen on master branch still happening..

Pressing Reboot in WebUI did not restart screenly viewer which should have temporarily resolved the issue, and checking syslog only showed this:

Nov 18 08:59:23 screenlypi sudo[30305]:       pi : TTY=unknown ; PWD=/home/pi/screenly ; USER=root ; COMMAND=/bin/systemctl status screenly-viewer.service -n 20
Nov 18 08:59:23 screenlypi sudo[30305]: pam_unix(sudo:session): session opened for user root by (uid=0)
Nov 18 08:59:23 screenlypi sudo[30305]: pam_unix(sudo:session): session closed for user root
Nov 18 09:00:08 screenlypi sudo[30347]:       pi : TTY=unknown ; PWD=/home/pi/screenly ; USER=root ; COMMAND=/bin/systemctl status screenly-viewer.service -n 20
Nov 18 09:00:11 screenlypi sudo[30347]: pam_unix(sudo:session): session opened for user root by (uid=0)
Nov 18 09:00:31 screenlypi sudo[30347]: pam_unix(sudo:session): session closed for user root

Will try disabling swap memory to see if system flushes out when needed.

Gifford47 commented 4 years ago

is there already a fix for the stable version?

vpetersson commented 4 years ago

@Gifford47 No, we don't do releases until the end of the sprint, but please verify that it solves the issue for you by switching to the development version.

Gifford47 commented 4 years ago

i have the version master@e301484 and after one day i still get a black screen... the ram is also continuously fully described. free ram at screenly start: 724mb free ram after 1 day: 84mb ram

Screenly / Anthias