Blank LED strip since changes to 'improvserial.h'

[kriztioan]

After syncing my fork with upstream, building and uploading the firmware to my esp32-s3-devkitc-1-n16r8v, the 107 LEDs on my strip stay blank. The development board is up-and-running, with Serial happily reporting all is well. The same for the web server, which allows me to change effects, but the LEDs remain blank.

Manually bisecting the issue traces it down to commit Blindly declare presence made in pull request #573 to improvserial.h. Unfortunately, the commit does not provide a rationale for the change. However, undoing c92e39ea39fd885dbee2686ad7b0ad383d4696cd fixes the issue.

[rbergen]

@kriztioan Do you have the exact codebase that now works for you (i.e. based on upstream:main minus only the commit you mention) pushed to a branch somewhere? If not, could you make one and provide a link to it? The reason I ask is that I don't understand how a few potential extra bytes being output to serial (which is the only thing that the commit you mention does, with a real chance that in reality nothing changes) would have that effect on the functioning of the LEDs.

[kriztioan]

@kriztioan Do you have the exact codebase that now works for you (i.e. based on upstream:main minus only the commit you mention) pushed to a branch somewhere? If not, could you make one and provide a link to it? The reason I ask is that I don't understand how a few potential extra bytes being output to serial (which is the only thing that the commit you mention does, with a real chance that in reality nothing changes) would have that effect on the functioning of the LEDs.

@rbergen Thank you for taking a look. The exact codebase (minus secrets.h) working for me is at i596. Note that this includes my boards-directory, custom_*-files, and a tiny adjustment to ledstripeffect.h that reverses the fill pattern.

[rbergen]

@kriztioan To be honest this one intrigues me, because I again can't see how a handful of extra bytes to serial could knock out 107 LEDs.

Of course, what I should have asked immediately is if you could also provide me (us) with captures of serial output from 1) a full boot cycle with the commit in, and 2) a full boot cycle with the commit out. Would that be possible?

[robertlipe]

I know these discussions are rarely advanced by some random nerd piping in with a "yeah, huh!" response but FWIW. If Rutger hadn't been the one to hop on this (and I'd been feeling better) it would have probably landed on my desk. Since Improve is "his", you got the better person first.

I haven't put in a debugger and tried to repro this either way, but when I saw the diff this was blamed on, I, too, was extremely skeptical. I'm more than a bit familiar with the code paths involved and I'm actually involved in a conversation right now about the weirdest computer problems one has ever had to solve, but the presence or absence of https://github.com/PlummersSoftwareLLC/NightDriverStrip/pull/573/commits/c92e39ea39fd885dbee2686ad7b0ad383d4696cd taking out the FastLED/render systems just seems whackadoodle.

I hate to ask the "have you tried turning if off and back on again" question, but are you really sure that building and testing that code works, and rolling back that one line makes it not work?

If anything, hanging out in STATE_PROVISIONED instead of STATE_CONNECTED might delay bringing up teh WiFi. ISTR you were running one of the CONFIG_COLORSERVER builds that accepts frames over the network. OK, if you don't get network, you're not going to get pixels that way but even so, within just a few seconds. If you're on a build like LEDSTRIP, that'd hold you in that state courtesy of WAIT_FOR_WIFI. I'm not really sure when TIME_BEFORE_LOCAL lights up and times out after 5 seconds (for that build) before it starts running something from include/effects/strip/.h that was configured in via effects. I just ran over the code and everything seems to have SOME local content so it wouldn't appear dead. Maybe you're really hanging in PROVISIONED.

On the third hand, since you made it a point to withold secrets.h, I'm assuming you have a WiFi AP/passphrase configured there and aren't doing fresh installs via the web downloader, so I'm not sure why you'd even be IN the Improv path.

It really doesn't make sense.

Can you please check the content of the serial monitor on a successful and unsuccessful run and diff them and let us know what's different in the two runs? Since you're on S3, grabbing that early serial data can be a little tricky. If you have a board with two ports, one will have a "real" UART that'll immediately pick up every byte out of a boot while the other has the USB CDC/ACM serial bridge running and it'll miss the first several dozen lines coming out of a boot. If you've conquered the JTAG beast, if you don't have a real UART, breakpoints in app_main() and/or use of something like RTT https://www.segger.com/products/debug-probes/j-link/technology/about-real-time-transfer/ (which we probably should integrate anyway - it's awesome) to catch all the startup chatter will help.

I think we should be blinking LEDs withing a few moments no matter how dead WiFi is, but if I understand your setup, I don't think this code should ever even be executing for you. We should be connectingn directly from the credentials in your Secrets.h.

A successful startup, from the beginning, will look something like this, with the first data coming from the bootrom and the subsequent code being ESP-IDF, FreeRTOS, and us as we start setting things up.:

ESP-ROM:esp32s3-20210327 Build:Mar 27 2021 rst:0x1 (POWERON),boot:0x28 (SPI_FAST_FLASH_BOOT) SPIWP:0xee mode:DIO, clock div:1 load:0x3fce3808,len:0x44c load:0x403c9700,len:0xbd8 load:0x403cc700,len:0x2a80 entry 0x403c98d0 E (256) esp_core_dump_flash: No core dump partition found! E (256) esp_core_dump_flash: No core dump partition found! Replacing Idle Tasks with TaskManager... (I) (PrintOutputHeader)(C1) NightDriverStrip (I) (I) (PrintOutputHeader)(C1)

(I) (PrintOutputHeader)(C1) M5STICKC: 0, USE_M5DISPLAY: 0, USE_OLED: 0, USE_TFTSPI: 0, USE_LCD: 0, USE_AUDIO: 0, ENABLE_REMOTE: 0 (I) (PrintOutputHeader)(C1) ESP32 Free Memory: 306272 (I) (PrintOutputHeader)(C1) ESP32 PSRAM not configured. (I) (PrintOutputHeader)(C1) Version 40: Wifi SSID: "ElderOfTheInternet" (I) (PrintOutputHeader)(C1) ESP32 Clock Freq : 2 cores @ 240 MHz (I) (setup)(C1) Startup! (I) (setup)(C1) Starting DebugLoopTaskEntry

On Tue, Jan 23, 2024 at 2:12 AM Rutger van Bergen @.***> wrote:

@kriztioan https://github.com/kriztioan To be honest this one intrigues me, because I again can't see how a handful of extra bytes to serial could knock out 107 LEDs.

Of course, what I should have asked immediately is if you could also provide me (us) with captures of serial output from 1) a full boot cycle with the commit in, and 2) a full boot cycle with the commit out. Would that be possible?

— Reply to this email directly, view it on GitHub https://github.com/PlummersSoftwareLLC/NightDriverStrip/issues/596#issuecomment-1905499216, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCSD35NHFKO2XAODG4TMD3YP5WIDAVCNFSM6AAAAABCELWUYWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMBVGQ4TSMRRGY . You are receiving this because you are subscribed to this thread.Message ID: @.*** com>

[kriztioan]

@rbergen The intrigue deepens. When hitting Monitor in pio with the 'broken' build uploaded to the device the LEDs spring to life. I'm able to consistently reproduce this behavior. That is, 1. turn the device ON via the power supply--LEDS are off; 2. Plug in the USB cable--LEDs are still off; and 3. hit Monitor in pio--LEDs spring to life. Thus far, I've been unable to catch the earlier serial data. My device has two USB-C ports and I will try the other. At the end of this comment is what I was able to catch thus far for the 'working' build.

@robertlipe My understanding of the inner workings of the code base, its knobs, and how and what each of the different libraries exactly does is not sufficient enough to assess whether "It really doesn't make sense". I'm simply providing a data point to consider to help out this amazing project. Perhaps when others encounter the same the signal-to-noise ratio will increase enough to get a clearer picture. I do really enjoy the Aha! moments when these puzzles get solved, though.

Serial output 'working' build:

[  1422][E][vfs_api.cpp:182] remove(): /improv.log does not exists or is directory
(I) (setup)(C1) Sending Improv packet to declare we're up. Ignore any IMPROV lines that follow this one.
>> Launching JSON Writer Thread.  Mem: 295808, LargestBlk: 278516, PSRAM Free: 8385939/8386231, (I) (LoadJSONFile)(C1) Attempting to read JSON file /device.cfg
(I) (DeviceConfig)(C1) Loading DeviceConfig from JSON
(W) (InitializeHardware)(C1) Allocating LEDStripGFX for channel 0
(I) (AddLEDsToFastLED)(C1) Adding LEDs to FastLED...
(I) (AddLEDsToFastLED)(C1) Adding 107 LEDs to pin 5 from channel 0 on FastLED.
(I) (SetupBufferManagers)(C1) Could allocate 20922 buffers but limiting it to 500
(I) 
(W) (SetupBufferManagers)(C1) Reserving 500 LED buffers for a total of 176500 bytes...
(W) (InitEffectsManager)(C1) InitEffectsManager...
(I) (LoadJSONFile)(C1) Attempting to read JSON file /effects.cfg
(W) (LoadJSONFile)(C1) Out of memory reading JSON from file /effects.cfg - increasing buffer to 7228 bytes
(W) (LoadJSONFile)(C1) Out of memory reading JSON from file /effects.cfg - increasing buffer to 9276 bytes
(W) (LoadJSONFile)(C1) Out of memory reading JSON from file /effects.cfg - increasing buffer to 11324 bytes
(I) (InitEffectsManager)(C1) Creating EffectManager from JSON config
(I) (ReadCurrentEffectIndex)(C1) Attempting to read file /current.cfg
>> Launching Drawing Thread.  Mem: 282136, LargestBlk: 270324, PSRAM Free: 8175899/8369495, (W) (DrawLoopTaskEntry)(C1) >> DrawLoopTaskEntry
(W) 
(W) (DrawLoopTaskEntry)(C1) Entering main draw loop!
(I) (setup)(C1) Making initial attempt to connect to WiFi.
(I) (ConnectToWiFi)(C1) WiFi credentials passed for SSID "C&C@centralpark"
(I) (ConnectToWiFi)(C1) Setting host name to NightDriverStrip...
(W) (ConnectToWiFi)(C1) Connecting to Wifi SSID: "C&C@centralpark" - ESP32 Free Memory: 239544, PSRAM:8369351, PSRAM Free: 8164939
(W) 
(W) (ConnectToWiFi)(C1) Not yet connected to WiFi, waiting...
>> Launching Network Thread.  Mem: 239496, LargestBlk: 229364, PSRAM Free: 8164939/8369351, >> Launching ColorData Thread.  Mem: 225336, LargestBlk: 217076, PSRAM Free: 8164939/8369351, (W) (NotifyJSONWriterThread)(C1) >> Notifying JSON Writer Thread
(W) (ConnectToWiFi)(C1) Not yet connected to WiFi, waiting...
(W) (ConnectToWiFi)(C1) Not yet connected to WiFi, waiting...
(W) (ConnectToWiFi)(C1) Not yet connected to WiFi, waiting...
(I) (SaveToJSONFile)(C0) Number of bytes written to JSON file /effects.cfg: 5180
(I) (ConnectToWiFi)(C1) Setting host name to NightDriverStrip...
(W) (ConnectToWiFi)(C1) Connecting to Wifi SSID: "C&C@centralpark" - ESP32 Free Memory: 222384, PSRAM:8369351, PSRAM Free: 8164939
(W) 
(W) (ConnectToWiFi)(C1) Not yet connected to WiFi, waiting...
(I) (loop)(C1) WiFi: WL_NO_SSID_AVAIL, IP: 0.0.0.0, Mem: 222128, LargestBlk: 212980, PSRAM Free: 8164939/8369351, LED FPS: 28 LED Bright:  24%, LED Watts: 3, CPU: 040%, 003%, FreeDraw: 0.030
(W) (ConnectToWiFi)(C1) Not yet connected to WiFi, waiting...
(W) (ConnectToWiFi)(C1) Not yet connected to WiFi, waiting...
(W) (ConnectToWiFi)(C1) Not yet connected to WiFi, waiting...
(W) (ConnectToWiFi)(C1) Not yet connected to WiFi, waiting...
(I) (ConnectToWiFi)(C1) Setting host name to NightDriverStrip...
(W) (ConnectToWiFi)(C1) Connecting to Wifi SSID: "C&C@centralpark" - ESP32 Free Memory: 221952, PSRAM:8369351, PSRAM Free: 8164939
(W) 
(W) (ConnectToWiFi)(C1) Not yet connected to WiFi, waiting...
(I) (loop)(C1) WiFi: WL_NO_SSID_AVAIL, IP: 0.0.0.0, Mem: 221840, LargestBlk: 212980, PSRAM Free: 8164939/8369351, LED FPS: 30 LED Bright:  24%, LED Watts: 3, CPU: 000%, 000%, FreeDraw: 0.030
(W) (ConnectToWiFi)(C1) Not yet connected to WiFi, waiting...
(W) (ConnectToWiFi)(C1) Not yet connected to WiFi, waiting...
(I) (loop)(C1) Sending Improv packets to indicate WiFi is connected. Ignore any IMPROV lines that follow this one.
IMPROV�
IMPROVhttp://192.168.20.17
(W) (ConnectToWiFi)(C1) Connected to AP with BSSID: "A0:04:60:31:7D:E2", received IP: 192.168.20.17
(I) (ConnectToWiFi)(C1) Setting Clock...
[ 14961][E][WiFiUdp.cpp:221] parsePacket(): could not receive data: 9
(W) (ColorDataTaskEntry)(C1) Started color data server!
(W) (UpdateClockFromWeb)(C1) NTP clock: Raw values sec=828423580, usec=192882
(I) (UpdateClockFromWeb)(C1) Old Time: 1706024124.157608, New Time: 1706024125.192882, Delta: 1.035274
(I) (UpdateClockFromWeb)(C1) Adjusting time by 1.035274 to 1706024125.192882
(I) (UpdateClockFromWeb)(C1) NTP clock: response received, updated time to: 1706024125.192882, DELTA: 1.035274
(I) 
(I) (ConnectToWiFi)(C1) Starting Web Server...
(I) (begin)(C1) Embedded html file size: 502
(I) (begin)(C1) Embedded jsx file size: 15259
(I) (begin)(C1) Embedded ico file size: 6434
(I) (begin)(C1) Embedded timezones file size: 17834
(I) (begin)(C1) Connecting Web Endpoints
(I) (begin)(C1) HTTP server started
(I) (ConnectToWiFi)(C1) Web Server begin called!
(I) (loop)(C1) WiFi: WL_CONNECTED, IP: 192.168.20.17, Mem: 193656, LargestBlk: 184308, PSRAM Free: 8163691/8369143, LED FPS: 30 LED Bright:  24%, LED Watts: 3, CPU: 000%, 000%, FreeDraw: 0.030
(I) (loop)(C1) WiFi: WL_CONNECTED, IP: 192.168.20.17, Mem: 193828, LargestBlk: 184308, PSRAM Free: 8163691/8369143, LED FPS: 30 LED Bright:  24%, LED Watts: 3, CPU: 000%, 000%, FreeDraw: 0.030
(I) (loop)(C1) WiFi: WL_CONNECTED, IP: 192.168.20.17, Mem: 193828, LargestBlk: 184308, PSRAM Free: 8163691/8369143, LED FPS: 30 LED Bright:  24%, LED Watts: 3, CPU: 000%, 000%, FreeDraw: 0.030
(I) (loop)(C1) WiFi: WL_CONNECTED, IP: 192.168.20.17, Mem: 193828, LargestBlk: 184308, PSRAM Free: 8163691/8369143, LED FPS: 30 LED Bright:  24%, LED Watts: 3, CPU: 000%, 000%, FreeDraw: 0.030
(I) (loop)(C1) WiFi: WL_CONNECTED, IP: 192.168.20.17, Mem: 193828, LargestBlk: 184308, PSRAM Free: 8163691/8369143, LED FPS: 30 LED Bright:  24%, LED Watts: 3, CPU: 000%, 000%, FreeDraw: 0.030

[rbergen]

@kriztioan That is interesting. In theory, this could be caused by the IMPROV package write to serial blocking when nobody's listening. The write in the commit you reversed in your fork happens during setup, so before the effects are activated. I'd still call this behavior weird, but I guess it's possible.

I'll think of a way to test this by moving the "offending" write from ImprovSerial::setup() to ImprovSerial::loop(). Can I open a PR on your fork to apply that change so you can test it? The problem is still not reproduceable for me. If yes, what branch shall I open it on?

[kriztioan]

@rbergen I have no luck yet with catching the earlier serial output. Please use the i596 branch for the PR--I'll pull it in and give it a test.

[rbergen]

@kriztioan Thanks. What's the config I should build to check things at least compile?

[kriztioan]

@kriztioan Thanks. What's the config I should build to check things at least compile?

kriztioan ;-)

[rbergen]

Well, that's another lesson learned. I had checked out your i596 branch and the PlatformIO plugin in VSCode did update, but did not pick up that added config. I had to restart VSCode for that to happen.
I'll remember that one as well.

Anyway, the kriztioan config builds on my machine. I'll open the PR on your fork in the next few minutes.

[rbergen]

@kriztioan I opened kriztioan#1 with an alternative approach to declaring device presence via Improv. Please let me know if this makes a functional difference for you.

[kriztioan]

@rbergen That fixed it! Thank you for looking into this! I'll let you deal with closing the issue etc.

[rbergen]

Glad to hear that! I'll open a PR on main to propagate this "hello!" approach to the main tree, and close this issue with the merge of that. Thank you for raising this, and finding the exact steps to unblock the start-up; in the end that's what led us to the resolution of this.

[robertlipe]

Should we think about doing something like starting a timer in setup and canceling it in main or loop or whatever? Or are timers also inoperational in the setup phase?

I'm willing to swing that axe if this is worth thinking about and we have something like a repro case. I'm willing to try a for(;;){} as a way of blocking it. OTOH, the watchdog should have also caught it if we we really were blocking on a tx fifo to drain or something, no?

i'm more interested in investigating the general class of failures than the exact "why did Improv do a silly thing (again)" investigation.

RJL

On Thu, Jan 25, 2024 at 12:28 AM Rutger van Bergen @.***> wrote:

Glad to hear that! I'll open a PR on main to propagate this "hello!" approach to the main tree, and close this issue with the merge of that. Thank you for raising this, and finding the exact steps to unblock the start-up; in the end that's what led us to the resolution of this.

— Reply to this email directly, view it on GitHub https://github.com/PlummersSoftwareLLC/NightDriverStrip/issues/596#issuecomment-1909435248, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCSD34SGR2FVE6WISZWOO3YQH3SXAVCNFSM6AAAAABCELWUYWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMBZGQZTKMRUHA . You are receiving this because you were mentioned.Message ID: @.***>

[rbergen]

Chances are good that I'm missing something, but I don't understand what a timer on its own would solve? If you start a timer and then make a blocking call, you're still stuck. For that to work you'd have to do the offending call in a separate thread, and I really, really don't want to kick off another one of those if we don't have to - if only because of the few KB of heap-allocated stack each of them chews up.

I believe the watchdog effectively only kicks in if one of the idle threads (one per core) doesn't have enough CPU breathing room to reset the watchdog timer. If the block happens in a different thread in a way that doesn't involve a busy wait, then the watchdog will happily conclude that the CPU is in its happy place - which it is.

I don't think Improv is the thing to blame as such in this case. All my devices happily thunder through the ImprovSerial::setup() code with or without USB buffers being drained, and I'm sure we would have already heard it if this was a common thing. I think there's something about @kriztioan's specific hardware configuration that triggers this. Undoubtedly whatever it is is not unique, but unusual.
...which also means I wouldn't currently know how to properly generalize this to a class of failures, let alone in a reproducible way.

[robertlipe]

On Fri, Jan 26, 2024 at 3:14 AM Rutger van Bergen @.***> wrote:

Chances are good that I'm missing something, but I don't understand what a timer on its own would solve? If you start a timer and then make a blocking call, you're still stuck.

Uhm, no. The timer interrupt fires, preempts the current (presumably spin-waiting) task, and then has a chance to recover. If it's cancelled by loop, we carry on. If the timer actually fires, we know we never got to loop. Timers run asynchronously to the scheduler.

For that to work you'd have to do the offending call in a separate thread, and I really, really don't want to kick off another one of those if we don't have to - if only because of the few KB of heap-allocated stack each of them chews up.

I'd agree, but it wouldn't need to be in another thread. In POSIX parlance, a firing timer would be like a signal.

Reviewing stack size remains on my TOOD list, but that's another thread, so to speak.

I believe the watchdog effectively only kicks in if one of the idle threads (one per core) doesn't have enough CPU breathing room to reset the watchdog timer. If the block happens in a different thread in a way that doesn't involve a busy wait, then the watchdog will happily conclude that the CPU is in its happy place - which it is.

That's a good sign that we're misusing the watchdog, actually. If we've effectively lost control of the CPU in a way we're not progressing toward blinkage, we shouldn't be patting the doggy on the head. (This is actually a depressingly common misuse of watchdogs in embeddedsville. Too much code just acks the watchdog in a timer interrupt without checking that the code is actually moving forward toward tis goal.)

I don't think Improv is the thing to blame as such in this case. All my devices happily thunder through the ImprovSerial::setup() code with or without USB buffers being drained, and I'm sure we would have already heard it if this was a common thing. I think there's something about @kriztioan https://github.com/kriztioan's specific hardware configuration that triggers this. Undoubtedly whatever it is is not unique, but unusual. ...which also means I wouldn't currently know how to properly generalize this to a class of failures, let alone in a reproducible way.

So you were never able to actually repro this and you just got lucky (that sounds insulting, but you hopefully know what I mean) with a brute force fix?

If we really don't know what was hanging or why, you're probably right that it's a bad idea to rip up walls and try to plumb around this. I was hoping that we had a repro case and/or stack trace with SOME kind of a viable explanation.

If you want me to take a look at replacing it, I will, but it sounds like that may be just overkill to kill a mystery bug that may affect only one exact setup that's not even really in captivity at this poitn.

Maybe we should leave Pandora's Box closed on this one after all...but this is the kind of thing that makes me crazy. I don't like unexplained mystery crashes.

RJL

Message ID: <PlummersSoftwareLLC/NightDriverStrip/issues/596/1911718226@ github.com>

[rbergen]

Uhm, no. The timer interrupt fires, preempts the current (presumably spin-waiting) task, and then has a chance to recover. If it's cancelled by loop, we carry on. If the timer actually fires, we know we never got to loop. Timers run asynchronously to the scheduler.

Right, so I did miss something. :) I missed the part where the timer kicks off some code that could act.

So you were never able to actually repro this and you just got lucky (that sounds insulting, but you hopefully know what I mean) with a brute force fix?

I know what you mean, and I think me getting lucky is a fair way to put it. Because no, I was never able to reproduce this.

Maybe we should leave Pandora's Box closed on this one after all...but this is the kind of thing that makes me crazy. I don't like unexplained mystery crashes.

Same here, but sometimes I choose to count my blessings and move on. Who knows, it may come back to haunt us in a more persistent and reproducible way at a later time. At which time you're totally entitled to pull the "I told you so" card. :)

[robertlipe]

I have fewer of those cards to play these days. Let's chalk it up to the mystery ghosts and move on.

Let's still add it to the running tally of Improve "WTH?!!"!" list. This is pretty whackadoodle, but since we don't have a repro case in hand and the symptoms so totally violate the Good Taste rules of how computers should work, let's bury our heads and move on, I suppose.

Something here is pretty fundamentally very wrong, but without repro, we could chase these ghosts across attics for months and that doesn't make sense for one observed failure. I guess I'll close this book.

Still, this kind of thing bugs me...

On Fri, Jan 26, 2024 at 6:35 AM Rutger van Bergen @.***> wrote:

Uhm, no. The timer interrupt fires, preempts the current (presumably spin-waiting) task, and then has a chance to recover. If it's cancelled by loop, we carry on. If the timer actually fires, we know we never got to loop. Timers run asynchronously to the scheduler.

Right, so I did miss something. :) I missed the part where the timer kicks off some code that could act.

So you were never able to actually repro this and you just got lucky (that sounds insulting, but you hopefully know what I mean) with a brute force fix?

I know what you mean, and I think me getting lucky is a fair way to put it. Because no, I was never able to reproduce this.

Maybe we should leave Pandora's Box closed on this one after all...but this is the kind of thing that makes me crazy. I don't like unexplained mystery crashes.

Same here, but sometimes I choose to count my blessings and move on. Who knows, it may come back to haunt us in a more persistent and reproducible way at a later time. At which time you're totally entitled to pull the "I told you so" card. :)

— Reply to this email directly, view it on GitHub https://github.com/PlummersSoftwareLLC/NightDriverStrip/issues/596#issuecomment-1912003435, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCSD3YBKR2R6SQRNDKISZDYQOPIXAVCNFSM6AAAAABCELWUYWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJSGAYDGNBTGU . You are receiving this because you were mentioned.Message ID: @.***>

[rbergen]

Still, this kind of thing bugs me...

Really? Man, you really know how to hide that well! 😂