Closed tobias-grasse closed 1 year ago
Saviq
commented
1 year ago Hi Tobias, PID 1 is systemd… so something must be causing systemd to try and restart Frame. The obvious candidate would be the snap refreshing, can you see if things in snap changes correlate with when Frame restarts?
or sometimes halts completely
That one's more worrying. Is the error the same when this happens? What's systemctl status snap.ubuntu-frame.daemon say?
AlanGriffiths
commented
1 year ago this may also be a better fit for the mir repo
No, terminating on receiving SIGTERM is correct behaviour. Any problem lies in not understanding why systemd is sends SIGTERM, nor why snapd doesn't restart Frame after it exits.
tobias-grasse
commented
1 year ago Thanks @Saviq @AlanGriffiths for the quick feedback.
can you see if things in
snap changescorrelate with when Frame restarts?
Service-related things in snap changes do correlate with Frame restarts, but not always. I have to admit it's a bit of a puzzle because the timestamps are off: My appliance needs to reboot once after initialization (i.e. seeding) so that NetworkManager reports the correct connectivity. After the reboot, system time is ~7 minutes behind again because the device didn't get an NTP sync and the RasPi does not have a RTC. So all log entries from the initial boot have “future” timestamps. I guess systemd does not parse previous log entries and hiccup on those, so they should “just” affect log entry order?
That one's more worrying. Is the error the same when this happens? What's
systemctl status snap.ubuntu-frame.daemonsay?
I have not been able to reproduce this one yet, so I can't provide further details right now. Will update here if I observe this again.
tobias-grasse
commented
1 year ago One thing I noticed in the logs:
~$ snap logs -nall ubuntu-frame.daemon | grep -a6 -b1 -i fatal
126110-2023-01-16T16:23:02Z ubuntu-frame.daemon[1335]: [2023-01-16 16:23:02.969727] < - debug - > mirserver: Handling Terminated from pid=1
126243:2023-01-16T16:23:03Z ubuntu-frame.daemon[1335]: !!! Fatal signal received. Attempting cleanup, but deadlock may occur
126361:2023-01-16T16:23:03Z ubuntu-frame.daemon[1335]: Mir fatal error: Unsupported attempt to continue after a fatal signal: SIGSEGV
126488:2023-01-16T16:23:03Z ubuntu-frame.daemon[1335]: !!! Fatal signal received. Attempting cleanup, but deadlock may occur
126606:2023-01-16T16:23:03Z ubuntu-frame.daemon[1335]: Mir fatal error: Unsupported attempt to continue after a fatal signal: SIGABRT
126733-2023-01-16T16:23:03Z systemd[1]: snap.ubuntu-frame.daemon.service: Main process exited, code=killed, status=6/ABRTThis only occurred once, but I can't tell if this was during an initial boot (i.e. frame was setting up the first time).
AlanGriffiths
commented
1 year ago This only occurred once
That looks somewhat like https://github.com/MirServer/mir/issues/2799. (But without a stack trace it is hard to be sure if it is the same thing - there might be clues earlier in the log)
tobias-grasse
commented
1 year ago I checked my test installations for these log lines, below is the full log from a ubuntu-frame.daemon service start that ended in a fatal error. note the ~2min delay between the last startup log line at 2023-01-19T15:42:58Z (scaling factor) and the Handling Terminated from pid=1, so honestly, I can't remember if I shut down the service myself as part of a system reboot.
Three things I noticed, but don't know if they are relevant. This is on a Raspberry Pi 3B with the config provided in my first post.
The VC4 driver seems weirdly out of date and has no proper version:
2023-01-19T15:42:57Z ubuntu-frame.daemon[15577]: [2023-01-19 15:42:57.270685] <information> gbm-kms: /dev/dri/card0: using driver vc4 [Broadcom VC4 graphics] (version: 0.0.0 driver date: 20140616)This warning. Could be caused by prevent_firmware_kms_setup enabled in config.txt?
2023-01-19T15:42:55Z ubuntu-frame.daemon[15577]: [2023-01-19 15:42:55.165026] < -warning- > gbm-kms: Failed to detect whether device /dev/dri/card0 supports KMS, but continuing anywayThis warning, the attached screen is 1080x1920 and frame proceeds to list supported modes for the connected HDMI-A-1 output.
2023-01-19T15:42:57Z ubuntu-frame.daemon[15577]: [2023-01-19 15:42:57.265275] < -warning- > gbm-kms: Unable to determine the current display mode.I did not observe another instance of unexpected restarts or fatal errors since yesterday, so unless you spot something worth investigating, this can be closed for now.
Saviq
commented
1 year ago Will close, please report back if you see anything new. Thanks!
On my Ubuntu Core appliance, Frame sometimes receives SIGTERM from the init process:
I cannot reproduce this reliably yet. I observed both a single automatic restart of the frame daemon service and a completely black screen where I would need to force-restart Frame.
Expected behavior
Ubuntu Frame should stay up and running.
Actual behavior
Ubuntu Frame daemon service restarts, or sometimes halts completely. Error message in
ubuntu-frame.daemonlog:I'm providing as much debug information as possible below in
<details>tags to keep this readable 🙂 Based on the debug message, this may also be a better fit for the mir repo, please move it if necessary.Device / OS setup
Ubuntu Core 22 custom model image on a Raspberry Pi 3B rev 1.2,
armhfarchitecture.The gadget snap is based on the reference
pi-gadgetsnap (22-armhfbranch) and only adds minor modifications to config.txt, two kernel command line parameters and default config / connection for seeded snaps.The appliance is set up via a WiFi captive portal without an initial internet connection, so the system time is usually way behind until users have configured their WiFi credentials and NTP could sync the correct time.
Installed snaps:
Ubuntu Frame configuration
```shell ~$ snap get ubuntu-frame Key Value config wallpaper-top=0x000000 wallpaper-bottom=0x000000 display-layout=default cursor none daemon true display layouts: default: cards: - card-id: 0 HDMI-A-1: orientation: right normal: cards: - card-id: 0 HDMI-A-1: orientation: normal right: cards: - card-id: 0 HDMI-A-1: orientation: right left: cards: - card-id: 0 HDMI-A-1: orientation: left inverted: cards: - card-id: 0 HDMI-A-1: orientation: inverted ```Kernel command line in /proc/cmdline
```shell ~$ for cmd in $(cat /proc/cmdline); do echo "$cmd"; done coherent_pool=1M 8250.nr_uarts=1 snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.enable_hdmi=1 vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000 console=ttyS0,115200 dwc_otg.lpm_enable=0 rng_core.default_quality=700 vt.handoff=2 quiet splash snapd_recovery_mode=run ```My config.txt is pretty much the default pi-gadget version from
22-armhf, note the### BEGIN/END customizationscomment:contents of config.txt
```shell ~$ cat /run/mnt/ubuntu-seed/config.txt [all] kernel=kernel.img cmdline=cmdline.txt initramfs initrd.img followkernel os_prefix=/piboot/ubuntu/pi-kernel_569.snap/ [pi4] max_framebuffers=2 arm_boost=1 [all] # Enable the audio output, I2C and SPI interfaces on the GPIO header. As these # parameters related to the base device-tree they must appear *before* any # other dtoverlay= specification dtparam=audio=on dtparam=i2c_arm=on dtparam=spi=on # Comment out the following line if the edges of the desktop appear outside # the edges of your display #disable_overscan=1 # If you have issues with audio, you may try uncommenting the following line # which forces the HDMI output into HDMI mode instead of DVI (which doesn't # support audio output) #hdmi_drive=2 # Enable the serial pins enable_uart=1 [cm4] # Enable the USB2 outputs on the IO board (assuming your CM4 is plugged into # such a board) dtoverlay=dwc2,dr_mode=host [all] # Enable the FKMS ("Fake" KMS) graphics overlay, and allocate 128Mb to the # GPU memory dtoverlay=vc4-fkms-v3d,cma-128 # Uncomment the following to enable the Raspberry Pi camera module firmware #start_x=1 #gpu_mem=128 ### BEGIN customizations for mirr.OS one ### # NOTE: On some displays, ubuntu-frame fails to start properly which results in a boot to console. # If that is the case, add a `#` in front of the line `dtoverlay=vc4-fkms-v3d,cma-128` and remove # the `#` from the following line: #dtoverlay=vc4-kms-v3d,cma-256 # Completely disable rainbow splash at boot disable_splash=1 # If your display doesn't get the correct mode delivered (shows "no signal"), enable this line. # See https://www.raspberrypi.com/documentation/computers/configuration.html#command-line-options disable_fw_kms_setup=1 ### END customizations for mirr.OS one ### # Prior versions of Ubuntu Core customized the on-board LEDs so that the green # ACT LED was a heartbeat, and the red PWR LED represented SD card activity. # Uncomment the following lines if you wish to restore these customizations #dtparam=act_led_trigger=heartbeat #dtparam=pwr_led_trigger=mmc0 ```