Document how to use the new sandboxing settings

[RecursiveGreen]

Is your feature request related to a problem? Please describe. With sandboxing now enabled by default, any use of external script calls fail. It would be nice to know how to properly configure for these types of setups.

I've attempted to make a very simple configuration which I thought would work (rw = "/" and network = true), but my external calls to python seem to always return an exit code 1 or blank.

I suspect a lot of this is due to my lack of understanding, and further lack of helpful beginner documentation to bubblewrap outside of liquidsoap's usage.

Describe the solution you'd like Although the rest of the configuration seems straight-forward, it's be nice to have something to explain how the "ro" and "rw" folders work. Are folders listed assumed to be recursive? How does this relate to running scripts?

Describe alternatives you've considered Disabling sandboxing until I know what I'm doing. 😆

[toots]

Hi,

We're hoping to get sandboxing to work for almost any user with the default configuration. Could you share more about your configuration/script so we can try to see what's going wrong?

[RecursiveGreen]

My exact radio configuration is listed here at issue #865.

When I ran it with the default sandboxing on, using the s3:// protocol (which calls aws) always returned an exit code of 1 without any other error logging. The files were not downloaded from S3, but the empty files were left untouched in the /tmp directory (which I had to clean up manually).

Also, my python scripts would either output blank or seemingly do nothing. They both make RESTful calls to a webservice to either get the next song or report when a song is played. In the case of the next song script, it is supposed to output an annotate protocol line for liquidsoap to parse. The played song script doesn't output anything, but I can confirm that the webservice never receives the information.

So far the common thread among all three is that they are all python-based and all make calls to the Internet.

[toots]

Any chance you'd be able to test the cleanup-sandboxing branch? You can pin via opam:

git clone https://github.com/savonet/liquidsoap.git
cd liquidsoap && git checkout cleanup-sandboxing
opam pin add .

Or else, you can also use the debian packages at: https://www.liquidsoap.info/doc-dev/install.html#debianubuntu

[RecursiveGreen]

I checked out the branch and recompiled.

For the s3:// test, it is still not working and reporting an error code 1. However, I also tested with a small .pls file of http:// calls and it works. So not all external calls are erroring out.

[toots]

Okay. Would you send me the full command output? I think you can have it by setting set("log.level",5). Also, what distribution are you using?

[RecursiveGreen]

I've tested with both Ubuntu 18.04.2 and Ubuntu 19.04

EDIT: I should also mention that this was recompiled from your latest commit when you asked for the log.

Here are the logs with redactions:

2019/07/11 20:36:25 >>> LOG START
2019/07/11 20:36:25 [main:3] Liquidsoap 1.4.0+scm (file:///home/josh/Development/liquidsoap-dev@c20ed8c16b4cf3b2ff6dde6b5681c6147027b7de:20190711:193118)
2019/07/11 20:36:25 [main:3] Using: bytes=[distributed with OCaml 4.02 or above] pcre=7.4.1 sedlex=2.1 menhirLib=20190626 dtools=0.4.1 duppy=0.8.0 cry=0.6.2 mm=0.5.0 xmlplaylist=0.1.4 ogg=
0.5.2 vorbis=0.7.1 opus=0.1.2 speex=0.2.1 mad=0.4.5 flac=0.1.5 flac.ogg=0.1.5 dynlink=[distributed with Ocaml] lame=0.3.3 shine=0.2.1 fdkaac=0.3.0 theora=0.3.1 ffmpeg=0.2.1 bjack=0.1.5 als
a=0.2.3 ao=0.2.1 samplerate=0.1.4 taglib=0.3.3 ssl=[unspecified] magic=0.7.3 camomile=1.0.1 inotify=2.3 yojson=1.7.0 faad=0.4.0 soundtouch=0.1.8 portaudio=0.2.1 pulseaudio=0.1.3 ladspa=0.1
.5 dssi=0.1.2 lo=0.1.2
2019/07/11 20:36:25 [main:2] 
2019/07/11 20:36:25 [main:2] DISCLAIMER: This version of Liquidsoap has been
2019/07/11 20:36:25 [main:2] compiled from a snapshot of the development code.
2019/07/11 20:36:25 [main:2] As such, it should not be used in production
2019/07/11 20:36:25 [main:2] unless you know what you are doing!
2019/07/11 20:36:25 [main:2] 
2019/07/11 20:36:25 [main:2] We are, however, very interested in any feedback
2019/07/11 20:36:25 [main:2] about our development code and committed to fix
2019/07/11 20:36:25 [main:2] issues as soon as possible.
2019/07/11 20:36:25 [main:2] 
2019/07/11 20:36:25 [main:2] If you are interested in collaborating to
2019/07/11 20:36:25 [main:2] the development of Liquidsoap, feel free to
2019/07/11 20:36:25 [main:2] drop us a mail at <savonet-devl@lists.sf.net>
2019/07/11 20:36:25 [main:2] or to join the #savonet IRC channel on Freenode.
2019/07/11 20:36:25 [main:2] 
2019/07/11 20:36:25 [main:2] Please send any bug report or feature request
2019/07/11 20:36:25 [main:2] at <https://github.com/savonet/liquidsoap/issues>.
2019/07/11 20:36:25 [main:2] 
2019/07/11 20:36:25 [main:2] We hope you enjoy this snapshot build of Liquidsoap!
2019/07/11 20:36:25 [main:2] 
2019/07/11 20:36:25 [frame:3] Using 44100Hz audio, 25Hz video, 44100Hz master.
2019/07/11 20:36:25 [frame:3] Frame size must be a multiple of 1764 ticks = 1764 audio samples = 1 video samples.
2019/07/11 20:36:25 [frame:3] Targetting 'frame.duration': 0.04s = 1764 audio samples = 1764 ticks.
2019/07/11 20:36:25 [frame:3] Frames last 0.04s = 1764 audio samples = 1 video samples = 1764 ticks.
2019/07/11 20:36:25 [sandbox:3] Sandboxing using bubblewrap at /usr/bin/bwrap
2019/07/11 20:36:25 [sandbox:3] Temporary directory: /tmp
2019/07/11 20:36:25 [sandbox:3] Read/write directories: 
2019/07/11 20:36:25 [sandbox:3] Read-only directories: /
2019/07/11 20:36:25 [sandbox:3] Network allowed: true
2019/07/11 20:36:25 [video.converter:3] Couldn't find preferred video converter: gavl.
2019/07/11 20:36:25 [audio.converter:3] Using preferred samplerate converter: libsamplerate.
2019/07/11 20:36:25 [threads:4] Created thread "generic queue #1".
2019/07/11 20:36:25 [threads:4] Created thread "generic queue #2".
2019/07/11 20:36:25 [threads:4] Created thread "non-blocking queue #1".
2019/07/11 20:36:25 [threads:4] Created thread "non-blocking queue #2".
2019/07/11 20:36:25 [clock:4] Currently 1 clocks allocated.
2019/07/11 20:36:25 [clock.wallclock_pulse:4] Starting 1 sources...
2019/07/11 20:36:25 [source:4] Source output.pulseaudio_8180 gets up.
2019/07/11 20:36:25 [source:4] Source mksafe gets up.
2019/07/11 20:36:25 [source:4] Source playlist_8177 gets up.
2019/07/11 20:36:25 [s3(dot)pls:3] Loading playlist...
2019/07/11 20:36:25 [s3(dot)pls:3] No mime type specified, trying autodetection.
2019/07/11 20:36:25 [playlist parser:4] Trying application/rss+xml parser
2019/07/11 20:36:25 [playlist.xml:5] Parsing failed: xml error: expected root element
2019/07/11 20:36:25 [playlist parser:4] Trying application/xspf+xml parser
2019/07/11 20:36:25 [playlist.xml:5] Parsing failed: xml error: expected root element
2019/07/11 20:36:25 [playlist parser:4] Trying application/smil+xml parser
2019/07/11 20:36:25 [playlist.xml:5] Parsing failed: xml error: expected root element
2019/07/11 20:36:25 [playlist parser:4] Trying application/smil parser
2019/07/11 20:36:25 [playlist.xml:5] Parsing failed: xml error: expected root element
2019/07/11 20:36:25 [playlist parser:4] Trying application/xml parser
2019/07/11 20:36:25 [playlist.xml:5] Parsing failed: xml error: expected root element
2019/07/11 20:36:25 [playlist parser:4] Trying text/xml parser
2019/07/11 20:36:25 [playlist.xml:5] Parsing failed: xml error: expected root element
2019/07/11 20:36:25 [playlist parser:4] Trying audio/x-ms-asx parser
2019/07/11 20:36:25 [playlist.xml:5] Parsing failed: xml error: expected root element
2019/07/11 20:36:25 [playlist parser:4] Trying video/x-ms-asf parser
2019/07/11 20:36:25 [playlist.xml:5] Parsing failed: xml error: expected root element
2019/07/11 20:36:25 [playlist parser:4] Trying youtube-dl parser
2019/07/11 20:36:25 [playlist parser:4] Trying application/x-cue parser
2019/07/11 20:36:25 [playlist parser:4] Trying audio/x-scpls parser
2019/07/11 20:36:25 [playlist parser:4] Trying application/x-mpegURL parser
2019/07/11 20:36:25 [s3(dot)pls:3] Playlist treated as format application/x-mpegURL
2019/07/11 20:36:25 [s3(dot)pls:3] Successfully loaded a playlist of 10 tracks.
2019/07/11 20:36:25 [s3(dot)pls:4] Content kind is {audio=2;video=0;midi=0}.
2019/07/11 20:36:25 [s3(dot)pls:4] Activations changed: static=[], dynamic=[mksafe:pulse_out(liquidsoap:):pulse_out(liquidsoap:)].
2019/07/11 20:36:25 [source:4] Source safe_blank gets up.
2019/07/11 20:36:25 [safe_blank:4] Content kind is {audio=2;video=0;midi=0}.
2019/07/11 20:36:25 [safe_blank:4] Activations changed: static=[], dynamic=[mksafe:pulse_out(liquidsoap:):pulse_out(liquidsoap:)].
2019/07/11 20:36:25 [mksafe:4] Activations changed: static=[pulse_out(liquidsoap:):pulse_out(liquidsoap:)], dynamic=[].
2019/07/11 20:36:25 [s3(dot)pls:5] Queue is empty!
2019/07/11 20:36:25 [s3(dot)pls:5] Failed to prepare track: no file.
2019/07/11 20:36:25 [pulse_out(liquidsoap:):4] Activations changed: static=[pulse_out(liquidsoap:)], dynamic=[].
2019/07/11 20:36:25 [pulse_out(liquidsoap:):4] Enabling caching mode: active source.
2019/07/11 20:36:25 [clock.wallclock_pulse:4] Delegating clock to active sources.
2019/07/11 20:36:25 [threads:4] Created thread "wallclock_pulse" (1 total).
2019/07/11 20:36:25 [clock:4] Main phase starts.
2019/07/11 20:36:25 [protocol.process:4] Processing .flac,aws --endpoint-url "https$(colon)//*****" s3 cp s3$(colon)//*****.flac $(output)
2019/07/11 20:36:25 [protocol.process:4] Executing aws --endpoint-url "https://*****" s3 cp s3://*****.flac "/tmp/liq-processf7f287..flac"
2019/07/11 20:36:25 [sandbox:5] Command: /usr/bin/bwrap --new-session --setenv TMPDIR "/tmp" --setenv TMP "/tmp" --setenv TEMPDIR "/tmp" --setenv TEMP "/tmp"  --ro-bind "/" "/" --bind "/tmp" "/tmp" --proc /proc --dev /dev aws --endpoint-url "https://*****" s3 cp s3://*****.flac "/tmp/liq-processf7f287..flac"
2019/07/11 20:36:25 [lang.run_process:4] Starting process
2019/07/11 20:36:25 [lang.run_process:4] Closing process's stdin
2019/07/11 20:36:25 [clock.wallclock_pulse:3] Streaming loop starts, synchronized by active sources.
2019/07/11 20:36:25 [mksafe:3] Switch to safe_blank.
2019/07/11 20:36:25 [safe_blank:4] Activations changed: static=[mksafe:pulse_out(liquidsoap:):pulse_out(liquidsoap:)], dynamic=[mksafe:pulse_out(liquidsoap:):pulse_out(liquidsoap:)].
2019/07/11 20:36:25 [s3(dot)pls:5] Queue is empty!
2019/07/11 20:36:25 [s3(dot)pls:5] Failed to prepare track: no file.
2019/07/11 20:36:25 [s3(dot)pls:5] Queue is empty!
2019/07/11 20:36:25 [s3(dot)pls:5] Failed to prepare track: no file.
2019/07/11 20:36:25 [s3(dot)pls:5] Queue is empty!
2019/07/11 20:36:25 [s3(dot)pls:5] Failed to prepare track: no file.
2019/07/11 20:36:25 [s3(dot)pls:5] Queue is empty!
2019/07/11 20:36:25 [s3(dot)pls:5] Failed to prepare track: no file.
2019/07/11 20:36:25 [s3(dot)pls:5] Queue is empty!
2019/07/11 20:36:25 [s3(dot)pls:5] Failed to prepare track: no file.
2019/07/11 20:36:25 [s3(dot)pls:5] Queue is empty!
2019/07/11 20:36:25 [s3(dot)pls:5] Failed to prepare track: no file.
2019/07/11 20:36:25 [lang.run_process:4] Process exited with code 1
2019/07/11 20:36:25 [lang.run_process:4] Cleaning up process
2019/07/11 20:36:25 [protocol.process:3] Failed to execute aws --endpoint-url "https://*****" s3 cp s3://*****.flac "/tmp/liq-processf7f287..flac": ("exit", "1")
2019/07/11 20:36:25 [request:4] Failed to resolve "process:.flac,aws --endpoint-url \"https$(colon)//*****\" s3 cp s3$(colon)//*****.flac $(output)"! For more info, see server command 'trace 1'.
2019/07/11 20:36:25 [protocol.process:4] Processing .flac,aws --endpoint-url "https$(colon)//*****" s3 cp s3$(colon)//*****.flac $(output)
2019/07/11 20:36:25 [protocol.process:4] Executing aws --endpoint-url "https://*****" s3 cp s3://*****.flac "/tmp/liq-processaafda8..flac"
2019/07/11 20:36:25 [sandbox:5] Command: /usr/bin/bwrap --new-session --setenv TMPDIR "/tmp" --setenv TMP "/tmp" --setenv TEMPDIR "/tmp" --setenv TEMP "/tmp"  --ro-bind "/" "/" --bind "/tmp" "/tmp" --proc /proc --dev /dev aws --endpoint-url "https://*****" s3 cp s3://*****.flac "/tmp/liq-processaafda8..flac"
2019/07/11 20:36:25 [lang.run_process:4] Starting process
2019/07/11 20:36:25 [lang.run_process:4] Closing process's stdin
2019/07/11 20:36:25 [s3(dot)pls:5] Queue is empty!
2019/07/11 20:36:25 [s3(dot)pls:5] Failed to prepare track: no file.
2019/07/11 20:36:25 [lang.run_process:4] Process exited with code 1
2019/07/11 20:36:25 [s3(dot)pls:5] Queue is empty!
2019/07/11 20:36:25 [s3(dot)pls:5] Failed to prepare track: no file.
2019/07/11 20:36:25 [protocol.process:3] Failed to execute aws --endpoint-url "https://*****" s3 cp s3://*****.flac "/tmp/liq-processaafda8..flac": ("exit", "1")
2019/07/11 20:36:25 [request:4] Failed to resolve "process:.flac,aws --endpoint-url \"https$(colon)//*****\" s3 cp s3$(colon)//*****.flac $(output)"! For more info, see server command 'trace 2'.
2019/07/11 20:36:25 [lang.run_process:4] Cleaning up process
2019/07/11 20:36:25 [protocol.process:4] Processing .flac,aws --endpoint-url "https$(colon)//*****" s3 cp s3$(colon)//*****.flac $(output)
2019/07/11 20:36:25 [protocol.process:4] Executing aws --endpoint-url "https://*****" s3 cp s3://*****.flac "/tmp/liq-processb130aa..flac"
2019/07/11 20:36:25 [sandbox:5] Command: /usr/bin/bwrap --new-session --setenv TMPDIR "/tmp" --setenv TMP "/tmp" --setenv TEMPDIR "/tmp" --setenv TEMP "/tmp"  --ro-bind "/" "/" --bind "/tmp" "/tmp" --proc /proc --dev /dev aws --endpoint-url "https://*****" s3 cp s3://*****.flac "/tmp/liq-processb130aa..flac"
2019/07/11 20:36:25 [lang.run_process:4] Starting process
2019/07/11 20:36:25 [lang.run_process:4] Closing process's stdin
2019/07/11 20:36:25 [lang.run_process:4] Process exited with code 1
2019/07/11 20:36:25 [lang.run_process:4] Cleaning up process
2019/07/11 20:36:25 [protocol.process:3] Failed to execute aws --endpoint-url "https://*****" s3 cp s3://*****.flac "/tmp/liq-processb130aa..flac": ("exit", "1")
2019/07/11 20:36:25 [request:4] Failed to resolve "process:.flac,aws --endpoint-url \"https$(colon)//*****\" s3 cp s3$(colon)//*****.flac $(output)"! For more info, see server command 'trace 3'.
2019/07/11 20:36:25 [protocol.process:4] Processing .flac,aws --endpoint-url "https$(colon)//*****" s3 cp s3$(colon)//*****.flac $(output)
2019/07/11 20:36:25 [protocol.process:4] Executing aws --endpoint-url "https://*****" s3 cp s3://*****.flac "/tmp/liq-processbc3e8d..flac"
2019/07/11 20:36:25 [sandbox:5] Command: /usr/bin/bwrap --new-session --setenv TMPDIR "/tmp" --setenv TMP "/tmp" --setenv TEMPDIR "/tmp" --setenv TEMP "/tmp"  --ro-bind "/" "/" --bind "/tmp" "/tmp" --proc /proc --dev /dev aws --endpoint-url "https://*****" s3 cp s3://*****.flac "/tmp/liq-processbc3e8d..flac"
2019/07/11 20:36:25 [lang.run_process:4] Starting process
2019/07/11 20:36:25 [lang.run_process:4] Closing process's stdin
2019/07/11 20:36:25 [s3(dot)pls:5] Queue is empty!
2019/07/11 20:36:25 [s3(dot)pls:5] Failed to prepare track: no file.
2019/07/11 20:36:25 [lang.run_process:4] Process exited with code 1
2019/07/11 20:36:25 [lang.run_process:4] Cleaning up process
2019/07/11 20:36:25 [protocol.process:3] Failed to execute aws --endpoint-url "https://*****" s3 cp s3://*****.flac "/tmp/liq-processbc3e8d..flac": ("exit", "1")
2019/07/11 20:36:25 [request:4] Failed to resolve "process:.flac,aws --endpoint-url \"https$(colon)//*****\" s3 cp s3$(colon)//*****.flac $(output)"! For more info, see server command 'trace 4'.
2019/07/11 20:36:25 [s3(dot)pls:5] Queue is empty!
2019/07/11 20:36:25 [s3(dot)pls:5] Failed to prepare track: no file.
2019/07/11 20:36:25 [s3(dot)pls:5] Queue is empty!
2019/07/11 20:36:25 [s3(dot)pls:5] Failed to prepare track: no file.
2019/07/11 20:36:25 [s3(dot)pls:5] Queue is empty!

[...this repeats for a while so I killed the radio...]

2019/07/11 20:36:30 [main:3] Shutdown started!
2019/07/11 20:36:30 [main:3] Waiting for threads to terminate...
2019/07/11 20:36:30 [threads:4] Shuting down thread wallclock_pulse
2019/07/11 20:36:30 [pulse_out(liquidsoap:):4] Activations changed: static=[], dynamic=[].
2019/07/11 20:36:30 [source:4] Source pulse_out(liquidsoap:) gets down.
2019/07/11 20:36:30 [clock.wallclock_pulse:4] All active sources stopped, synching with wallclock.
2019/07/11 20:36:30 [mksafe:4] Activations changed: static=[], dynamic=[].
2019/07/11 20:36:30 [source:4] Source mksafe gets down.
2019/07/11 20:36:30 [s3(dot)pls:4] Activations changed: static=[], dynamic=[].
2019/07/11 20:36:30 [source:4] Source s3(dot)pls gets down.
2019/07/11 20:36:30 [s3(dot)pls:4] Waiting for feeding task to stop...
2019/07/11 20:36:30 [s3(dot)pls:4] Cleaning up request queue...
2019/07/11 20:36:30 [safe_blank:4] Activations changed: static=[mksafe:pulse_out(liquidsoap:):pulse_out(liquidsoap:)], dynamic=[].
2019/07/11 20:36:30 [safe_blank:4] Activations changed: static=[], dynamic=[].
2019/07/11 20:36:30 [source:4] Source safe_blank gets down.
2019/07/11 20:36:30 [clock.wallclock_pulse:3] Streaming loop stopped.
2019/07/11 20:36:30 [threads:4] Thread "wallclock_pulse" terminated (0 remaining).
2019/07/11 20:36:30 [main:3] Threads terminated.
2019/07/11 20:36:30 [threads:3] Shutting down scheduler...
2019/07/11 20:36:30 [threads:3] Scheduler shut down.
2019/07/11 20:36:30 [threads:3] Shutting down queues...
2019/07/11 20:36:30 [threads:4] Thread "generic queue #2" terminated.
2019/07/11 20:36:30 [threads:4] Thread "generic queue #1" terminated.
2019/07/11 20:36:30 [threads:4] Thread "non-blocking queue #2" terminated.
2019/07/11 20:36:30 [threads:4] Thread "non-blocking queue #1" terminated.
2019/07/11 20:36:30 [threads:3] Queues shut down
2019/07/11 20:36:30 [main:3] Cleaning downloaded files...
2019/07/11 20:36:30 [main:3] Freeing memory...
2019/07/11 20:36:30 [source:4] Garbage collected pulse_out(liquidsoap:).
2019/07/11 20:36:30 [source:4] Garbage collected mksafe.
2019/07/11 20:36:30 [source:4] Garbage collected safe_blank.
2019/07/11 20:36:30 [source:4] Garbage collected s3(dot)pls.
2019/07/11 20:36:30 >>> LOG END

[toots]

Thanks for these. I tried the command manually on a ubuntu and it worked. Any chance you could try to do the same? Trying to figure out if there's anything different from when it's run within liq and when it's run in a normal shell environment..

[RecursiveGreen]

I tried one of the commands manually:

josh@picard:~$ /usr/bin/bwrap --new-session --setenv TMPDIR "/tmp" --setenv TMP "/tmp" \
--setenv TEMPDIR "/tmp" --setenv TEMP "/tmp"  --ro-bind "/" "/" --bind "/tmp" "/tmp" --proc /proc \
--dev /dev aws --endpoint-url "https://*****" s3 cp s3://*****.flac "/tmp/liq-processaafda8..flac"
need to run as root or suid
josh@picard:~$

a-HA! Looks like a permissions issue. Is there a way to set that as part of the command line?

[RecursiveGreen]

This might be unrelated, but which version of aws did you use? There's the apt install and the snapd install.

I installed the snapd version as it's more current, but doesn't snapd use some form of sandboxing as well? Might there be a conflict here?

[toots]

I used the packaged version. I'm not sure about snapd but I'd be tempted to look at bwrap first. Does the user that run liquidsoap have sudo power on the system? One thing you could try:

sudo chmod u+s /usr/bin/bwrap

This would give SUID to the binary, allowing it to be run as root by any user, not just users with sudo power..

[RecursiveGreen]

I tried what you suggested and nothing changed. I was getting the same error.

On a hunch, I started researching issues with snapd and bubblewrap. It appears that there are some oddities between snap packages and bubblewrap in general--usually with having to make sure the package is configured to allow things like SUID.

I decided to uninstall it and instead install the Python package (which is the same version as the snap package). This version works with no issues. 😅

Although this fixes the aws issue, I now want to do one last sanity check on my Python scripts to see if they work with your updates from yesterday.

[RecursiveGreen]

I finally found a use for the "rw" setting. My python scripts do write to a separate log file and they were erroring out until I added the log directory to the "rw" list.

However, I ran into another problem (which was easily fixed). Due to a python UTF-8 quirk in Ubuntu 18.04, I have to set an environment variable in the commandline to force UTF-8 IO. The bwrap command did not allow the usual declaration in the beginning of the command, so I had to change it to --setenv PYTHONIOENCODING \"utf-8\". I ran the script again and everything worked again.

With that in mind, would it be possible to add one more config setting for any extra environment variables people want to pass to bwrap so that it will set them automatically? I'm thinking something like this:

set("sandbox.env",[("PYTHONIOENCODING","utf-8")])

Thanks again for the great support!

[toots]

Thanks for the feedback! I've just switch the sandboxing environment to run through shell /bin/sh by defaut.. Would you mind trying that? That should make FOO=bla /path/to/script calls work by default.

[RecursiveGreen]

I recompiled, set the variable back, and everything is working as intended again.

Unless you have anything further to clean up, I believe this can be closed and the branch merged.

Thanks again!