random crashes (futex-related)

[bdantas]

About once a day sfwbar will crash after I close a window. Sometimes the window that was closed was the only existing window (as in the screenshot below), sometimes one of many. The number of windows does not seem to have any correlation. How long my laptop has been running also seems to have no correlation (today it happened within 15 minutes of a fresh reboot).

After the crash, pkill sfwbar; sfwbar & results in two bars, because the crashed instance does not respond to SIGTERM:

To make the crashed instance disappear it is necessary to send it a SIGKILL. My system is a ThinkPad laptop running a very lean Tiny Core Linux x86_64. Compositor is labwc. I'd be happy to help find root cause if given some troubleshooting guidance.

EDIT: Software version details: sfwbar: 1.0_beta13 labwc: 0.6.5 wayland: 1.22.0 linux: 6.1.2

[LBCrion]

Ouch. I suspect this may have been introduced with paged taskbar (aka multidesktop tint2 mode). Am I right in understanding that the issue is sporadic, always happens when window closes and the bar freezes?

Can you post your config file please?

Thank you for flagging this!

On Tue, 7 Nov 2023, 13:46 GNUser, @.***> wrote:

About once a day sfwbar will crash after I close a window. Sometimes the window that was closed was the only existing window (as in the screenshot below), sometimes one of many. The number of windows does not seem to have any correlation.

After the crash, pkill sfwbar; sfwbar & results in two bars, because the crashed instance does not respond to SIGTERM: [image: screenshot001] https://user-images.githubusercontent.com/5084088/281059000-513b778b-236a-45f1-9219-1e5560a05585.jpg

To make the crashed instance disappear it is necessary to send it a SIGKILL. I'd be happy to help find root cause if given some troubleshooting guidance.

— Reply to this email directly, view it on GitHub https://github.com/LBCrion/sfwbar/issues/121, or unsubscribe https://github.com/notifications/unsubscribe-auth/ASHPFFE7EIKQQ5RBRMJPXY3YDI3SVAVCNFSM6AAAAAA7BG5HROVHI2DSMVQWIX3LMV43ASLTON2WKOZRHE4DCNBQGEZDSOI . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[bdantas]

Am I right in understanding that the issue is sporadic, always happens when window closes and the bar freezes?

Precisely.

My config file is below as you requested. Anything I should change or eliminate that would reduce the chances of sfwbar crashing?

Set Term = "xterm"

# triggers for eject_widget
TriggerAction "SIGRTMIN+3", SetStyle "usb_widget", "hidden"
TriggerAction "SIGRTMIN+4", SetStyle "usb_widget", "launcher"

# triggers for vpn_widget (used by: vpn-monitor)
TriggerAction "SIGRTMIN+5", SetValue "vpn_widget", "/opt/scripts/media/spikey.png"
TriggerAction "SIGRTMIN+6", SetValue "vpn_widget", "/opt/scripts/media/lock.png"

# triggers for misc_widget (used by: wifi-connect, easyprint, ep-monitor, hidden-files-toggler)
TriggerAction "SIGRTMIN+10", SetValue "misc_widget", ""     
TriggerAction "SIGRTMIN+16", SetValue "misc_widget", "/opt/scripts/media/radar.png"            
TriggerAction "SIGRTMIN+17", SetValue "misc_widget", "/opt/scripts/media/error.png"            
TriggerAction "SIGRTMIN+18", SetValue "misc_widget", "/opt/scripts/media/gear.png"            
TriggerAction "SIGRTMIN+19", SetValue "misc_widget", "/opt/scripts/media/printer-white.png"            
TriggerAction "SIGRTMIN+20", SetValue "misc_widget", "/opt/scripts/media/printer-green.png"
TriggerAction "SIGRTMIN+21", SetValue "misc_widget", "/opt/scripts/media/printer-red.png"
TriggerAction "SIGRTMIN+22", SetValue "misc_widget", "/opt/scripts/media/eye.png"        

# Window Placer
placer {
  xorigin = 5  # place the first window at X% from the left
  yorigin = 5  # place the first window at X% from the top
  xstep = 5    # step by X% of desktop horizontally
  ystep = 5    # step by X% of desktop vertically
  children = true
}

# Task Switcher
switcher {
  interval = 700
  icons = true
  labels = false
  cols = 5
}

include("winops.widget")

# Panel layout

# Window Placer
placer {
  xorigin = 5  # place the first window at X% from the left
  yorigin = 5  # place the first window at X% from the top
  xstep = 5    # step by X% of desktop horizontally
  ystep = 5    # step by X% of desktop vertically
  children = true
}

# Task Switcher
switcher {
  interval = 700
  icons = true
  labels = false
  cols = 5
}

function("SfwbarInit") {
  SetBarId "bar-0"
  SetLayer "bottom"
  SetExclusiveZone "auto"
}

include("winops.widget")

# Panel layout

layout {
  include("showdesktop.widget")
  #include("startmenu.widget")

  #button {
  #  style = "launcher"
  #  value = $Term
  #  action = Exec $Term
  #}

  #button {
  #  style = "launcher"
  #  value = "brave-browser"
  #  action = "brave-browser"
  #}

  pager {
    style = "pager"
    rows = 1
    preview = true
    numeric = true
  }

  taskbar {
    rows = 1
    css = "* { -GtkWidget-hexpand: false; }" # stretch horizontally
    title_width = 15
    icons = true
    labels = true
    action[3] = Menu "winops"
    action[2] = Close
  }

  label {
    css = "* { -GtkWidget-hexpand: true; }"
  }

  #include("idle.widget")

  tray {
    rows = 1
  }

  button "usb_widget" {
    style = "hidden"
    value = "/opt/scripts/media/eject.png"
    tooltip = "eject|unmount usb drive"
    action[LeftClick] = Exec "usb-unmount eject"
    action[RightClick] = Exec "usb-unmount"
  }
  include("bluez.widget")
  include("network-module.widget")
  button "vpn_widget" { style = "launcher" tooltip = "vpn status" value = "/opt/scripts/media/wireguard.png" }
  include("battery-svg.widget")
# Select pulse-module or alsa-module based on your sound configuration
# include("pulse-module.widget")
  include("alsa-module.widget")
  include("mpd-intmod.widget")
  include("sway-lang.widget")
  button "misc_widget" { style = "launcher" tooltip = "misc" }
  grid {
    css = "* { padding-left: 5px; padding-right: 5px; }"
    label {
      value = Time("%a %b %e")
      loc(1,1)
      action[LeftClick] = Exec "calendar-toggle"
    }
    label {
      value = Time("%l:%M %p")
      loc(1,2)
      action[LeftClick] = Exec "calendar-toggle"
    }
  }
  #include("cpu.widget")
  #include("memory.widget")
}

#CSS
@define-color theme_text_color white;
@define-color theme_bg_color black;
@define-color theme_fg_color white;
@define-color borders rgba(256,256,256,0.3);

window#sfwbar {
  -GtkWidget-direction: top;
  background-color: rgba(0,0,0,0.6);
}

grid#layout {
  -GtkWidget-direction: right;
  min-height: 30px;
}

grid {
  padding: 0px;
  margin: 0px;
}

tooltip {
  background-color: @theme_bg_color;
}

image {
  min-width: 18px;
  min-height: 18px;
  box-shadow: none;
  border: none;
  border-image: none;
  background-image: none;
  background: none;
  -gtk-icon-shadow: none;
}

button {
  box-shadow: none;
  border-image: none;
  background-image: none;
  background: none;
  outline-style: none;
}

menu {
  background-color: black;
  border: 1px solid @borders;
  padding: 3px;
}

menu arrow {
  background-color: rgba(0,0,0,0);
  color: @theme_text_color;
}

menuitem {
  color: @theme_text_color;
  padding: 2px;
}

menu image {
  padding-right: 5px;
}

progressbar trough {
  background-color: rgba(256, 256, 256, 0.5);
  border-color: rgba(256,256,256,0.5);
}

#hidden {
  -GtkWidget-visible: false;
}

button#taskbar_normal grid {
  -GtkWidget-hexpand: false;
}

button#taskbar_normal image,
button#taskbar_active image,
button#taskbar_normal:hover image {
  min-width: 24px;
  min-height: 24px;
}

button#taskbar_normal label,
button#taskbar_active label,
button#taskbar_normal:hover label {
  padding: 0px 0px 0px 5px;
  -GtkWidget-hexpand: false;
}

button#taskbar_normal,
button#taskbar_active,
button#taskbar_normal:hover {
  padding: 0px 5px 0px 5px;
  border-radius: 4px;
  background-color: rgba(119,119,119,0.2);
  border-color: rgba(119,119,119,0.3);
  -GtkWidget-hexpand: false;
}

button#taskbar_active {
  background-color: rgba(255,255,255,0.2);
  border-color: rgba(255,255,255,0.4);
}

button#taskbar_normal:hover,
button#taskbar_active:hover {
  background-color: rgba(176,176,176,0.22);
  border-color: rgba(234,234,234,0.44);
}

button#pager_normal,
button#pager_visible,
button#pager_focused {
  padding: 0px 5px 0px 5px;
  border-radius: 4px;
  background-color: rgba(119,119,119,0);
  border-color: rgba(119,119,119,0);
}

button#pager_focused {
  background-color: rgba(119,119,119,0.2);
  border: none;
}

button#pager_preview {
  border-radius: 0;
  border-color: #000000;
  border-width: 0.25mm;
  color: #777777;
  min-width: 5cm;
  min-height: 2.8125cm;
}

grid#switcher_active image,
grid#switcher_active {
  min-width: 60px;
  min-height: 60px;
  border-image: none;
  padding: 6px;
  background-color: #777777;
  border: 0px;
  box-shadow: none;
  border-radius: 1.25mm;
  -GtkWidget-hexpand: true;
}

grid#switcher_normal image,
grid#switcher_normal {
  min-width: 60px;
  min-height: 60px;
  padding: 6px;
  -GtkWidget-direction: right;
  -GtkWidget-hexpand: true;
}

window#switcher { 
  background-color: rgba(0,0,0,0);
  padding: 6px; 
  -GtkWidget-hexpand: true;
}

grid#switcher { 
  border-radius: 6px;
  background-color: @theme_bg_color;
  border: 1px solid @borders;
  padding: 6px;
  box-shadow: none;
  -GtkWidget-hexpand: true;
}

button#tray_active,
button#tray_passive,
button#tray_attention {
  padding: 0px 1px 0px 0px;
  border: none;
  margin: 0px;
  -GtkWidget-valign: center;
}

button#startmenu image,
button#module image{
  -ScaleImage-color: @theme_text_color;
}

button#module,
button#launcher,
button#showdesktop,
button#startmenu {
  padding: 0px 3px 0px 3px;
  border: none;
  -GtkWidget-valign: center;
}

button#startmenu {
  border: 1px solid;
  border-radius: 4px;
}

button#startmenu image {
  min-width: 18px;
  min-height: 18px;
  padding: 1px;
}

chart#cpu_chart {
  background: rgba(127,127,127,0.3);
  min-width: 9px;
  margin: 2px;
  border: 1px solid @borders;
  color: red;
}

progressbar#memory {
  -GtkWidget-direction: top;
  min-width: 9px;
  border: 1px solid @borders;
  margin: 2px;
}

progressbar#memory trough {
  min-height: 2px;
  min-width: 9px;
  border: none;
  border-radius: 0px;
  background: rgba(127,127,127,0.3);
}

progressbar#memory progress {
  -GtkWidget-hexpand: true;
  min-width: 9px;
  border-radius: 0px;
  border: none;
  margin: 0px;
  background-color: alpha(green,0.9);
}

label#sway_lang {
  margin-left: 3px;
  margin-right: 5px;
}

label {
  font: 12px Sans;
  color: @theme_text_color;
  text-shadow: none;
}

* {
  -GtkWidget-vexpand: true;
}

#taskbar_normal,                   
#taskbar_active {                                                               
  min-width: 150px;                                                
}     

[LBCrion]

No need to change config, I just wanted to check which codepath causes this error. Are you running the latest git version of sfwbar or one of the betas?

On Tue, Nov 7, 2023 at 9:09 PM GNUser @.***> wrote:

Am I right in understanding that the issue is sporadic, always happens when window closes and the bar freezes?

Precisely.

My config file is below as you requested. Anything I should change or eliminate that would reduce the chances of sfwbar crashing?

Set Term = "xterm"

triggers for eject_widget

TriggerAction "SIGRTMIN+3", SetStyle "usb_widget", "hidden" TriggerAction "SIGRTMIN+4", SetStyle "usb_widget", "launcher"

triggers for vpn_widget (used by: vpn-monitor)

TriggerAction "SIGRTMIN+5", SetValue "vpn_widget", "/opt/scripts/media/spikey.png" TriggerAction "SIGRTMIN+6", SetValue "vpn_widget", "/opt/scripts/media/lock.png"

triggers for misc_widget (used by: wifi-connect, easyprint, ep-monitor, hidden-files-toggler)

TriggerAction "SIGRTMIN+10", SetValue "misc_widget", "" TriggerAction "SIGRTMIN+16", SetValue "misc_widget", "/opt/scripts/media/radar.png" TriggerAction "SIGRTMIN+17", SetValue "misc_widget", "/opt/scripts/media/error.png" TriggerAction "SIGRTMIN+18", SetValue "misc_widget", "/opt/scripts/media/gear.png" TriggerAction "SIGRTMIN+19", SetValue "misc_widget", "/opt/scripts/media/printer-white.png" TriggerAction "SIGRTMIN+20", SetValue "misc_widget", "/opt/scripts/media/printer-green.png" TriggerAction "SIGRTMIN+21", SetValue "misc_widget", "/opt/scripts/media/printer-red.png" TriggerAction "SIGRTMIN+22", SetValue "misc_widget", "/opt/scripts/media/eye.png"

Window Placer

placer { xorigin = 5 # place the first window at X% from the left yorigin = 5 # place the first window at X% from the top xstep = 5 # step by X% of desktop horizontally ystep = 5 # step by X% of desktop vertically children = true }

Task Switcher

switcher { interval = 700 icons = true labels = false cols = 5 }

include("winops.widget")

Panel layout

Window Placer

placer { xorigin = 5 # place the first window at X% from the left yorigin = 5 # place the first window at X% from the top xstep = 5 # step by X% of desktop horizontally ystep = 5 # step by X% of desktop vertically children = true }

Task Switcher

switcher { interval = 700 icons = true labels = false cols = 5 }

function("SfwbarInit") { SetBarId "bar-0" SetLayer "bottom" SetExclusiveZone "auto" }

include("winops.widget")

Panel layout

layout { include("showdesktop.widget")

include("startmenu.widget")

button {

style = "launcher"

value = $Term

action = Exec $Term

}

button {

style = "launcher"

value = "brave-browser"

action = "brave-browser"

}

pager { style = "pager" rows = 1 preview = true numeric = true }

taskbar { rows = 1 css = "* { -GtkWidget-hexpand: false; }" # stretch horizontally title_width = 15 icons = true labels = true action[3] = Menu "winops" action[2] = Close }

label { css = "* { -GtkWidget-hexpand: true; }" }

include("idle.widget")

tray { rows = 1 }

button "usb_widget" { style = "hidden" value = "/opt/scripts/media/eject.png" tooltip = "eject|unmount usb drive" action[LeftClick] = Exec "usb-unmount eject" action[RightClick] = Exec "usb-unmount" } include("bluez.widget") include("network-module.widget") button "vpn_widget" { style = "launcher" tooltip = "vpn status" value = "/opt/scripts/media/wireguard.png" } include("battery-svg.widget")

Select pulse-module or alsa-module based on your sound configuration

include("pulse-module.widget")

include("alsa-module.widget") include("mpd-intmod.widget") include("sway-lang.widget") button "misc_widget" { style = "launcher" tooltip = "misc" } grid { css = "* { padding-left: 5px; padding-right: 5px; }" label { value = Time("%a %b %e") loc(1,1) action[LeftClick] = Exec "calendar-toggle" } label { value = Time("%l:%M %p") loc(1,2) action[LeftClick] = Exec "calendar-toggle" } }

include("cpu.widget")

include("memory.widget")

}

CSS

@define-color theme_text_color white; @define-color theme_bg_color black; @define-color theme_fg_color white; @define-color borders rgba(256,256,256,0.3);

window#sfwbar { -GtkWidget-direction: top; background-color: rgba(0,0,0,0.6); }

grid#layout { -GtkWidget-direction: right; min-height: 30px; }

grid { padding: 0px; margin: 0px; }

tooltip { background-color: @theme_bg_color; }

image { min-width: 18px; min-height: 18px; box-shadow: none; border: none; border-image: none; background-image: none; background: none; -gtk-icon-shadow: none; }

button { box-shadow: none; border-image: none; background-image: none; background: none; outline-style: none; }

menu { background-color: black; border: 1px solid @borders; padding: 3px; }

menu arrow { background-color: rgba(0,0,0,0); color: @theme_text_color; }

menuitem { color: @theme_text_color; padding: 2px; }

menu image { padding-right: 5px; }

progressbar trough { background-color: rgba(256, 256, 256, 0.5); border-color: rgba(256,256,256,0.5); }

hidden {

-GtkWidget-visible: false; }

button#taskbar_normal grid { -GtkWidget-hexpand: false; }

button#taskbar_normal image, button#taskbar_active image, button#taskbar_normal:hover image { min-width: 24px; min-height: 24px; }

button#taskbar_normal label, button#taskbar_active label, button#taskbar_normal:hover label { padding: 0px 0px 0px 5px; -GtkWidget-hexpand: false; }

button#taskbar_normal, button#taskbar_active, button#taskbar_normal:hover { padding: 0px 5px 0px 5px; border-radius: 4px; background-color: rgba(119,119,119,0.2); border-color: rgba(119,119,119,0.3); -GtkWidget-hexpand: false; }

button#taskbar_active { background-color: rgba(255,255,255,0.2); border-color: rgba(255,255,255,0.4); }

button#taskbar_normal:hover, button#taskbar_active:hover { background-color: rgba(176,176,176,0.22); border-color: rgba(234,234,234,0.44); }

button#pager_normal, button#pager_visible, button#pager_focused { padding: 0px 5px 0px 5px; border-radius: 4px; background-color: rgba(119,119,119,0); border-color: rgba(119,119,119,0); }

button#pager_focused { background-color: rgba(119,119,119,0.2); border: none; }

button#pager_preview { border-radius: 0; border-color: #000000; border-width: 0.25mm; color: #777777; min-width: 5cm; min-height: 2.8125cm; }

grid#switcher_active image, grid#switcher_active { min-width: 60px; min-height: 60px; border-image: none; padding: 6px; background-color: #777777; border: 0px; box-shadow: none; border-radius: 1.25mm; -GtkWidget-hexpand: true; }

grid#switcher_normal image, grid#switcher_normal { min-width: 60px; min-height: 60px; padding: 6px; -GtkWidget-direction: right; -GtkWidget-hexpand: true; }

window#switcher { background-color: rgba(0,0,0,0); padding: 6px; -GtkWidget-hexpand: true; }

grid#switcher { border-radius: 6px; background-color: @theme_bg_color; border: 1px solid @borders; padding: 6px; box-shadow: none; -GtkWidget-hexpand: true; }

button#tray_active, button#tray_passive, button#tray_attention { padding: 0px 1px 0px 0px; border: none; margin: 0px; -GtkWidget-valign: center; }

button#startmenu image, button#module image{ -ScaleImage-color: @theme_text_color; }

button#module, button#launcher, button#showdesktop, button#startmenu { padding: 0px 3px 0px 3px; border: none; -GtkWidget-valign: center; }

button#startmenu { border: 1px solid; border-radius: 4px; }

button#startmenu image { min-width: 18px; min-height: 18px; padding: 1px; }

chart#cpu_chart { background: rgba(127,127,127,0.3); min-width: 9px; margin: 2px; border: 1px solid @borders; color: red; }

progressbar#memory { -GtkWidget-direction: top; min-width: 9px; border: 1px solid @borders; margin: 2px; }

progressbar#memory trough { min-height: 2px; min-width: 9px; border: none; border-radius: 0px; background: rgba(127,127,127,0.3); }

progressbar#memory progress { -GtkWidget-hexpand: true; min-width: 9px; border-radius: 0px; border: none; margin: 0px; background-color: alpha(green,0.9); }

label#sway_lang { margin-left: 3px; margin-right: 5px; }

label { font: 12px Sans; color: @theme_text_color; text-shadow: none; }

• { -GtkWidget-vexpand: true; }

taskbar_normal,

taskbar_active {

min-width: 150px; }

— Reply to this email directly, view it on GitHub https://github.com/LBCrion/sfwbar/issues/121#issuecomment-1800123842, or unsubscribe https://github.com/notifications/unsubscribe-auth/ASHPFFEVRQVQBXQY2DXUNVLYDKPRNAVCNFSM6AAAAAA7BG5HROVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBQGEZDGOBUGI . You are receiving this because you commented.Message ID: @.***>

[bdantas]

I'm running 1.0_beta13

[LBCrion]

Would you be able to check if you see this error with the latest git version please? I fixed a few bugs (at least one of them relate to closing the last window on a workspace) since the last release. It's possible that this is a manifestation of one of them.

[bdantas]

Yes, I'll be happy to test latest git for this issue. Today I'll compile it and start using it. I'll report back after 2-3 days to make sure I've stressed it enough.

[bdantas]

I have put latest git version through the wringer for >24 hours and there has not been a single crash due to window closing. It seems commits since 1.0_beta13 took care of the problem.

If issue recurs I'll reopen this issue so that we can investigate more, but it seems to be fixed :+1:

[bdantas]

Unfortunately it happened again (closing a window -> unresponsive sfwbar).

I'll be away from computer this Friday-Sunday. If you are interested in investigating this, I'd be happy to help next week.

[LBCrion]

Doh! I was hoping this was one of the bugs I already fixed. Would you mind trying the latest git version? I added some safeguard that I suspect may help with any race condition that may occur on window close.

If it does happen again, can you try killing it with signal 11 (SIGSEGV) please? This should generate a core dump. Can you try getting a stack trace from it? (if you run systemd and have gdb installed, you can run coredumpctl gdb and once in gdb type bt full to get a stack trace).

[bdantas]

I'll be happy to try latest git and send SIGSEGV if it happens again. Is there a particular directory where I should look for the core dump, or is that OS- dependent?

I'm on Tiny Core Linux (busybox init) so systemd not applicable.

[Consolatis]

Should be fine to just start with gdb --args sfwbar .. then pressing r to start instead. Then after killing sfwbar it should allow you to do the bt full there.

[bdantas]

It just happened again with latest git version (which includes the mutex commit). When it locks up, sfwbar does not respond to SIGTERM or SIGSEGV, only SIGKILL.

I do occasionally run Xwayland application as root (e.g., gparted or file manager), which is made possible by xhost +SI:localuser:root (I know it is insecure, ill-advised, etc). The sfwbar crashes seem to happen with greater frequency when there is a GUI application running as root.

[LBCrion]

In this case the method recommended by @Consolatis may work better. It should catch termination even if you are killing sfwbar using SIGKILL. I would recommend pulling the latest git again (I added a couple more fixes, although I don't think they will affect this issue directly).

I've never seen a lockup like you mention so far, which makes your testing really valuable here (THANK YOU!). I don't think running xwayland applications as root should be able to cause this (since sfwbar only gets information about windows from labwc anyway), but if they tend to cause the issue to appear faster, it may help with testing.

[bdantas]

sfwbar is very valuable to me, so I'm more than happy to help with testing :smile:

I pulled the latest git again and am running sfwbar with your latest changes.

Unfortunately, Consolatis's method fails to start sfwbar:

bruno@x230:~/Downloads$ gdb --args sfwbar ..
GNU gdb (GDB) 11.2
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from sfwbar...
(gdb) r
Starting program: /usr/local/bin/sfwbar ..
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".
[New Thread 0x7ffff5e996c0 (LWP 10662)]
[New Thread 0x7ffff56986c0 (LWP 10663)]
[New Thread 0x7ffff4e976c0 (LWP 10664)]

(sfwbar:10658): GLib-GIO-CRITICAL **: 14:02:58.759: This application can not open files.
[Thread 0x7ffff6449e40 (LWP 10658) exited]
[Thread 0x7ffff56986c0 (LWP 10663) exited]
[Thread 0x7ffff5e996c0 (LWP 10662) exited]
[Inferior 1 (process 10658) exited with code 01]
(gdb) 

Maybe I need a more minimalistic way to capture information about the crash when it occurs--I'm on Tiny Core Linux, which provides a very barebones environment.

EDIT: bruno@x230:~/Downloads$ gdb --args sfwbar then r works fine.

[bdantas]

Hi LBCrion. I'm trying to create a testing environment that will capture something helpful for you when the crash happens next time. I tried Consolatis's suggestion but, after sending SIGKILL (as a test), there is no coredump or any other information:

$ gdb --args sfwbar
GNU gdb (GDB) 11.2
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from sfwbar...
(gdb) r
Starting program: /usr/local/bin/sfwbar 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".
[New Thread 0x7ffff5e996c0 (LWP 14786)]
[New Thread 0x7ffff56986c0 (LWP 14787)]
[New Thread 0x7ffff4e976c0 (LWP 14788)]
[New Thread 0x7fffe7fff6c0 (LWP 14789)]

(sfwbar:14782): GLib-CRITICAL **: 14:39:56.478: g_hash_table_remove_all: assertion 'hash_table != NULL' failed

(sfwbar:14782): GLib-GIO-CRITICAL **: 14:39:56.478: g_dbus_connection_signal_unsubscribe: assertion 'G_IS_DBUS_CONNECTION (connection)' failed

(sfwbar:14782): GLib-GIO-CRITICAL **: 14:39:56.478: g_dbus_connection_signal_unsubscribe: assertion 'G_IS_DBUS_CONNECTION (connection)' failed

(sfwbar:14782): GLib-GIO-CRITICAL **: 14:39:56.478: g_dbus_connection_signal_unsubscribe: assertion 'G_IS_DBUS_CONNECTION (connection)' failed
i3-ipc*input type:keyboard xkb_switch_layout nexti3-ipc*input type:keyboard xkb_switch_layout prev[Detaching after vfork from child process 14790]
[New Thread 0x7fffe6c226c0 (LWP 14792)]
[New Thread 0x7fffe64216c0 (LWP 14793)]
[New Thread 0x7fffe64216c0 (LWP 14794)]
[Thread 0x7fffe64216c0 (LWP 14793) exited]
[New Thread 0x7fffe5c206c0 (LWP 14795)]
[Thread 0x7fffe64216c0 (LWP 14794) exited]
[Thread 0x7fffe5c206c0 (LWP 14795) exited]
[New Thread 0x7fffe5c206c0 (LWP 14796)]
[New Thread 0x7fffe64216c0 (LWP 14797)]
[Thread 0x7fffe5c206c0 (LWP 14796) exited]
[Thread 0x7fffe64216c0 (LWP 14797) exited]
[Thread 0x7ffff56986c0 (LWP 14787) exited]
[Thread 0x7fffe7fff6c0 (LWP 14789) exited]
[Thread 0x7ffff4e976c0 (LWP 14788) exited]
[Thread 0x7ffff5e996c0 (LWP 14786) exited]
[Thread 0x7ffff6449e40 (LWP 14782) exited]

Program terminated with signal SIGKILL, Killed.
The program no longer exists.
(gdb) bt full
No stack.
(gdb) 

[bdantas]

Hi LBCrion. I've been running sfwbar with strace. I was able to capture two crashes, both of which forced me to send SIGKILL to sfwbar and restart it.

Here is the last screenful of lines from first crash:

recvmsg(12, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=[{{len=20, type=NLMSG_DONE, flags=NLM_F_MULTI, seq=1699847431, pid=3279064968}, "\0\0\0\0"}, {{len=1, type=0x14 /* NLMSG_??? */, flags=NLM_F_REQUEST, seq=0, pid=0}}], iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 20
close(12)                               = 0
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 12
ioctl(12, SIOCGIWSTATS, 0x7ffd27273110) = 0
close(12)                               = 0
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 12
ioctl(12, SIOCGIWSTATS, 0x7ffd27273110) = 0
close(12)                               = 0
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 12
ioctl(12, SIOCGIWSTATS, 0x7ffd27273110) = 0
close(12)                               = 0
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 12
ioctl(12, SIOCGIWSTATS, 0x7ffd27273110) = 0
close(12)                               = 0
write(5, "\1\0\0\0\0\0\0\0", 8)         = 8
poll([{fd=3, events=POLLIN}, {fd=5, events=POLLIN}, {fd=10, events=POLLIN|POLLPRI}, {fd=11, events=POLLIN}], 4, 0) = 1 ([{fd=5, revents=POLLIN}])
read(5, "\2\0\0\0\0\0\0\0", 16)         = 8
--- SIGRT_16 {si_signo=SIGRT_16, si_code=SI_USER, si_pid=21692, si_uid=0} ---
futex(0x7fecb17f1c40, FUTEX_WAIT_PRIVATE, 2, NULL) = ?
+++ killed by SIGKILL +++

Full strace of sfwbar running until first crash is here.

Here is last screenful of second crash's strace:

newfstatat(AT_FDCWD, "/usr/local/etc/xkb", 0x7ffc9e83a3e8, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr/local/share/X11/xkb", {st_mode=S_IFDIR|0755, st_size=160, ...}, 0) = 0
newfstatat(AT_FDCWD, "/usr/local/share/X11/xkb", {st_mode=S_IFDIR|0755, st_size=160, ...}, 0) = 0
geteuid()                               = 1000
getegid()                               = 50
getuid()                                = 1000
getgid()                                = 50
access("/usr/local/share/X11/xkb", R_OK|X_OK) = 0
mmap(NULL, 53211, PROT_READ, MAP_SHARED, 15, 0) = 0x7f5a6803c000
munmap(0x7f5a6803c000, 53211)           = 0
close(15)                               = 0
newfstatat(AT_FDCWD, "/home/bruno/.config/xkb", 0x7ffc9e83a3e8, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/home/bruno/.xkb", 0x7ffc9e83a3e8, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr/local/etc/xkb", 0x7ffc9e83a3e8, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr/local/share/X11/xkb", {st_mode=S_IFDIR|0755, st_size=160, ...}, 0) = 0
newfstatat(AT_FDCWD, "/usr/local/share/X11/xkb", {st_mode=S_IFDIR|0755, st_size=160, ...}, 0) = 0
geteuid()                               = 1000
getegid()                               = 50
getuid()                                = 1000
getgid()                                = 50
access("/usr/local/share/X11/xkb", R_OK|X_OK) = 0
mmap(NULL, 53211, PROT_READ, MAP_SHARED, 16, 0) = 0x7f5a6803c000
--- SIGRT_8 {si_signo=SIGRT_8, si_code=SI_USER, si_pid=27133, si_uid=1000} ---
futex(0x7f5a6a7f1c40, FUTEX_WAIT_PRIVATE, 2, NULL) = ?
+++ killed by SIGKILL +++

Full strace of sfwbar running until this second crash is here.

Could this bug have something to do with RT signal handling and be totally unrelated to closing windows? I think bug only seems to be related to closing windows because it is only when I close a window--and notice the window still showing on sfwbar--that I become aware that sfwbar has crashed.

A few observations to help pin down the bug:

1. Last lines of both crashes (right before I need to send SIGKILL because sfwbar is unresponsive) are similar: Something about RT signal followed by something about futex.
2. None of my scripts send SIGRTMIN+8, so the appearance of that signal in the second strace is a bit of a mystery to me.
3. I use Tiny Core Linux, which uses Busybox (compiled with limited options) and not GNU coreutils or procps. The Busybox version that comes with TCL does not seem to support RT signals. My scripts send RT signals to sfwbar using a dedicated busybox binary that I compiled with all options enabled, which I call "busyboxFull". Scripts send RT signals to sfwbar as in this example: busyboxFull pkill -RTMIN+5 -x sfwbar
4. The crashes seem totally random. I cannot intentionally crash sfwbar by sending it multiple RT signals from a terminal emulator.

I'll be happy to provide whatever additional information you need to investigate this.

EDIT: Are there certain RT signals that are reserved (i.e., that user scripts should not use)? Right now various of my shell scripts use SIGRTMIN+3 through SIGRTMIN+6, and also SIGRTMIN+10 through SIGRTMIN+22.

[LBCrion]

This is interesting. Signal 8 is SIGFPE - which is a computational exception, and signal 16 is a SIGSTKFLT - an FPU fault signal. The later shouldn't be triggered on x86 hardware normally.

This beings me to a question, what hardware and libc are you using please?

On Mon, 13 Nov 2023, 04:48 GNUser, @.***> wrote:

Hi LBCrion. I've been running sfwbar with strace. I was able to capture two crashes, requiring sending SIGKILL and restarting sfwbar.

Here is the last screenful of lines from first crash:

recvmsg(12, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=[{{len=20, type=NLMSG_DONE, flags=NLM_FMULTI, seq=1699847431, pid=3279064968}, "\0\0\0\0"}, {{len=1, type=0x14 /* NLMSG??? */, flags=NLM_F_REQUEST, seq=0, pid=0}}], iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 20 close(12) = 0 socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 12 ioctl(12, SIOCGIWSTATS, 0x7ffd27273110) = 0 close(12) = 0 socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 12 ioctl(12, SIOCGIWSTATS, 0x7ffd27273110) = 0 close(12) = 0 socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 12 ioctl(12, SIOCGIWSTATS, 0x7ffd27273110) = 0 close(12) = 0 socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 12 ioctl(12, SIOCGIWSTATS, 0x7ffd27273110) = 0 close(12) = 0 write(5, "\1\0\0\0\0\0\0\0", 8) = 8 poll([{fd=3, events=POLLIN}, {fd=5, events=POLLIN}, {fd=10, events=POLLIN|POLLPRI}, {fd=11, events=POLLIN}], 4, 0) = 1 ([{fd=5, revents=POLLIN}]) read(5, "\2\0\0\0\0\0\0\0", 16) = 8 --- SIGRT_16 {si_signo=SIGRT_16, si_code=SI_USER, si_pid=21692, si_uid=0} --- futex(0x7fecb17f1c40, FUTEX_WAIT_PRIVATE, 2, NULL) = ? +++ killed by SIGKILL +++

Full strace of sfwbar running until first crash is here https://gnuser.ddns.net/public/log1.txt.

Here is last screenful of second crash's strace:

newfstatat(AT_FDCWD, "/usr/local/etc/xkb", 0x7ffc9e83a3e8, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/usr/local/share/X11/xkb", {st_mode=S_IFDIR|0755, st_size=160, ...}, 0) = 0 newfstatat(AT_FDCWD, "/usr/local/share/X11/xkb", {st_mode=S_IFDIR|0755, st_size=160, ...}, 0) = 0 geteuid() = 1000 getegid() = 50 getuid() = 1000 getgid() = 50 access("/usr/local/share/X11/xkb", R_OK|X_OK) = 0 mmap(NULL, 53211, PROT_READ, MAP_SHARED, 15, 0) = 0x7f5a6803c000 munmap(0x7f5a6803c000, 53211) = 0 close(15) = 0 newfstatat(AT_FDCWD, "/home/bruno/.config/xkb", 0x7ffc9e83a3e8, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/home/bruno/.xkb", 0x7ffc9e83a3e8, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/usr/local/etc/xkb", 0x7ffc9e83a3e8, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/usr/local/share/X11/xkb", {st_mode=S_IFDIR|0755, st_size=160, ...}, 0) = 0 newfstatat(AT_FDCWD, "/usr/local/share/X11/xkb", {st_mode=S_IFDIR|0755, st_size=160, ...}, 0) = 0 geteuid() = 1000 getegid() = 50 getuid() = 1000 getgid() = 50 access("/usr/local/share/X11/xkb", R_OK|X_OK) = 0 mmap(NULL, 53211, PROT_READ, MAP_SHARED, 16, 0) = 0x7f5a6803c000 --- SIGRT_8 {si_signo=SIGRT_8, si_code=SI_USER, si_pid=27133, si_uid=1000} --- futex(0x7f5a6a7f1c40, FUTEX_WAIT_PRIVATE, 2, NULL) = ? +++ killed by SIGKILL +++

Full strace of sfwbar running until this second crash is here https://gnuser.ddns.net/public/log2.txt.

Could this bug have something to do with RT signal handling and be totally unrelated to closing windows? I think bug only seems to be related to closing windows because it is only when I close a window--and notice the window still showing on sfwbar--that I become aware that sfwbar has crashed.

A few observations to help pin down the bug:

1. Last lines of both crashes (right before I need to send SIGKILL because sfwbar is unresponsive) are similar: Something about RT signal followed by something about futex.
2. None of my scripts send SIGRTMIN+8, so the appearance of that signal in the second strace is a bit of a mystery to me.
3. I use Tiny Core Linux, which uses Busybox (compiled with limited options) and not GNU coreutils or procps. The Busybox version that comes with TCL does not seem to support RT signals. My scripts send RT signals to sfwbar using a dedicated busybox binary that I compiled with all options enabled, which I call "busyboxFull". Scripts send RT signals to sfwbar like so: busyboxFull pkill -RTMIN+5 -x sfwbar
4. The crashes seem totally random. I cannot intentionally crash sfwbar by sending it multiple RT signals from a terminal emulator.

I'll be happy to provide whatever additional information you need to investigate this.

— Reply to this email directly, view it on GitHub https://github.com/LBCrion/sfwbar/issues/121#issuecomment-1807472446, or unsubscribe https://github.com/notifications/unsubscribe-auth/ASHPFFC2PLZTHS3NM5SMXXLYEGQ77AVCNFSM6AAAAAA7BG5HROVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBXGQ3TENBUGY . You are receiving this because you commented.Message ID: @.***>

[bdantas]

Hardware is a ThinkPad X230 laptop with Core i7-3520M processor. BIOS is Coreboot. OS is Tiny Core Linux 14 x86_64.

Libc is GNU libc 2.36.

P.S. On this same laptop in a different partition, everything works in X environment with Fluxbox. In partition with wayland environment and labwc, everything else (other than these random sfwbar crashes) is working great. In other words, I don't think there is something wrong with the hardware.

[bdantas]

There are some widgets in my config file that don't show up in the bar (namely, bluez.widget, mpd-intmod.widget, and sway-lang.widget). Pardon my ignorance, but is it conceivable that one of these widgets may be to blame for these crashes?

Since these widgets don't show up in the bar anyway (probably because my system is not providing them with the information they need), I will remove them from my config file and see if it helps.

[bdantas]

Hi, LBCrion. I disabled the widgets that don't show up in the panel and restarted sfwbar. I've been running sfwbar with strace for an hour with no crashes. Interestingly, I see both SIGRT_8 and SIGRT_16 in the strace so far (again, no crashes in past hour).

I have no further leads other than the hypothesis that perhaps widgets that are unable to get the information they need were crashing the bar. Is this plausible?

EDIT: In the strace of sfwbar running without crashes, futex appears but only in one of these 3 forms:

futex (foo bar baz) = -1 EAGAIN (Resource temporarily unavailable)
futex (foo bar baz) = 0
futex (foo bar baz) = 1

This form only appears in strace associated with a crash:

futex (foo bar baz) = ?

Does this suggest a root cause to you?

[bdantas]

Good news! I narrowed the problem down to bluez.widget. I suspected this widget because of combination of a) it has calls to g_mutex_* plus b) bluetooth is not enabled on my laptop.

I've been running two instances of sfwbar concurrently, with identical configuration except that one contains include("bluez.widget") in its config file and the other one doesn't. The instance that has include("bluez.widget") crashes while other instance happily chugs along:

[LBCrion]

That's very odd, the bluez module code shouldn't even reach any of the mutex locks unless dbus is enabled and the bluez daemon is running. That said, if sfwbar is stable without it on your system I'll need to dig into it and see what could be causing this. I may need to ask you to run some test since your system is able to trigger this behavior.

One quick question, when you run the git version of sfwbar, do you install it or just run from the build directory directly? I'm asking because in the later case, unless the build directory is your CWD, sfwbar will load the modules from the last install.

[LBCrion]

Also, can you try running sfwbar with bluez module enabled and appending the following to your normal command line -d -g bluez

[bdantas]

Hi LBCrion. When I run git version I create a custom package and install it.

Darn it, bluez.widget turns out to be another red herring. Just now I was just running two sfwbar instances: One with bluez in configuration file, one without. The one with bluez was running as sfwbar -d -g bluez >log.txt 2>&1 & as you requested, to hopefully capture something meaningful in the log file. The instance without bluez is the one that crashed!

I think I'm ready to throw in the towel. The crashes seem to be completely unpredictable to me and I feel like a blind man looking for a needle in a haystack. If you want me to do specific tests (or if you'd like remote access to take a look around), let me know. But it's too time consuming to troubleshoot this waiting for crashes given their unpredictable nature.

[Consolatis]

Just tested with an older sfwbar version, this works for me:

gdb --args build/sfwbar --config ~/.config/sfwbar/own_bottom.config -d -g bluez
r

once looping / unresponsive: kill -SIGSTOP $(pidof sfwbar) in the gdb shell: bt If you want to continue the execution: c multiple times, q otherwise

[bdantas]

It turns out the bluez widget does not have to be active for a crash to occur (please see my previous post). Crashes seem totally random.

There is something about my system that causes sfwbar to be unstable. At this point I'm not optimistic that we'll find the root cause because I've invested hours in troubleshooting and don't think I'm close to finding the problem.

[Consolatis]

Right now various of my shell scripts use SIGRTMIN+3 through SIGRTMIN+6, and also SIGRTMIN+10 through SIGRTMIN+22.

Every signal breaks the current flow of the application. Depending on the amount and interval of them they may randomly hit a code part that is not prepared for that. That could explain the hard-to-reproduce behavior. To rule those out completely you could try to remove all the signal stuff from your shell scripts / don't start them and just run like that for a while to see if the issue still shows up.

[bdantas]

Hi Consolatis. Trying to run sfwbar without the SIGRTMIN signals from my scripts is a good idea. I'll give that a shot, thanks for the suggestion. I will report back after several days.

P.S. I will do the above but it will be inconvenient because I rely on icons to inform me of various things (e.g., whether vpn is on or off, whether my file manager is displaying hidden files or not). Is there a different application (other than sfwbar) that my shell scripts could use to add/change/remove icons from somewhere on my display?

[Consolatis]

You could use the Exec or File scanners instead: https://github.com/LBCrion/sfwbar/blob/main/doc/sfwbar.rst#scanner

I am not currently sure if sfwbar also supports spawning a subprocess once, throwing its stdout fd in a poll() and only reading its stdout when there is new data available and then fire up a trigger. That should provide the same performance / overhead as the signal based solution but without its potential issues.

[bdantas]

Hi Consolatis. Thank you very much. I will tweak sfwbar and my relevant shell scripts to use scanners rather than RT signals to show me the status icons I rely on. Fingers crossed! :)

[LBCrion]

@Consolatis, you can use ExecClient scanner to poll stdout fd of a script.

I agree that testing if repeated RT signals may be causing it is worth it. And if it does, it may be worth checking which signals are doing this. In theory sfwbar should be handling anything between RTMIN and RTMAX.

Another thing to try may be disabling the network module. Given the severity of a crash (program needs to be killed at the kernel level), I suspect this must be caused by something deep in the OS or something hardware related. Network module uses ioctl to probe wifi hardware periodically, which makes it a potential culprit.

[bdantas]

I disabled the network module, rebooted, and after a while I experienced a crash. So I think network module is innocent.

[LBCrion]

I think signals, re-entrancy and tread safety is a bit of a can of worms, but I made a (naive) change in the latest git to handle signals across all threads. Would you mind testing the latest git version to see if it behaves any better?

[bdantas]

Every signal breaks the current flow of the application. Depending on the amount and interval of them they may randomly hit a code part that is not prepared for that. That could explain the hard-to-reproduce behavior. To rule those out completely you could try to remove all the signal stuff from your shell scripts / don't start them and just run like that for a while to see if the issue still shows up.

@Consolatis - I did as you recommended and removed all the RTMIN signals my scripts. I have not had a crash yet, so at this point this is the leading theory to explain the crashes.

I think signals, re-entrancy and tread safety is a bit of a can of worms, but I made a (naive) change in the latest git to handle signals across all threads. Would you mind testing the latest git version to see if it behaves any better?

@LBCrion - Let me test sfwbar without the RTMIN signals for another day or two so that I can hopefully establish a baseline of stability. After that, I'll be happy to test the latest git by hitting it with RTMIN signals from my scripts.

P.S. The File scanners provided all the functionality I needed to replace my RTMIN-sending shell scripts with custom widgets. I'm quite amazed at the power and flexibility of sfwbar. Many thanks to LBCrion for creating something so robust, and for generously making it free software!

[LBCrion]

Glad to hear the scanner functionality works. I re-wrote the signal handler to be re-entrant now. If you do confirm that sfwbar runs ok without signals, I would appreciate if you could test the setup with signals in the latest version to see if it's safe. Unless I'm mistaken the issue is caused by two signals arriving too close together, calling a signal handler while it's already running.

[bdantas]

I have compiled, packaged, and installed latest git. I will put it through the wringer with RTMIN signals. Will report back when I have results.

[bdantas]

I've been using latest git with my scripts that periodically send RTMIN signals to sfwbar. No crashes yet! I even tried stressing sfwbar by using my scripts and this in the background:

while true; do busyboxFull pkill -RTMIN+5 -x sfwbar; sleep 1; done 

I think your latest commit solved the problem. Amazing!

Let me keep using the laptop normally until tomorrow afternoon. If no crashes, I think we can be pretty confident that problem is solved and close the issue.

[bdantas]

sfwbar has been rock-stable since last commit--not a single crash despite normal use and abuse of RTMIN signals. Kudos!

Thanks again for sfwbar. It is the definitive solution in every regard: aesthetics, functionality, customizability. It leaves absolutely nothing to be desired.

[LBCrion]

Great to hear! If you do encounter the issue again please do re-open the bug. In retrospect, this does make sense. If sfwbar received a signal while still handling the previous signal it could hit one of glib's internal mutexes and lock up inside a signal call, which in turn would require a SIGKILL.

Thank you for all the help testing this!