deliciouslytyped
commented
4 years ago An attempt at debugging that I forogt to mention, also the most fruitful so far, was trying to use strace. A short excerpt is attached here:
....
read(3, ",placeholder"..., 498) = 498
sendto(6, "placeholder"..., 545, 0, {sa_family=AF_INET, sin_port=htons(1676), sin_addr=inet_addr("[redacted]")}, 16) = -1 EMSGSIZE (Message too long)
read(3, 0x7ffdbe246372, 66033) = -1 EFAULT (Bad address)
read(3, 0x7ffdbe246371, 66034) = -1 EFAULT (Bad address)
read(3, 0x7ffdbe246370, 66035) = -1 EFAULT (Bad address)
read(3, 0x7ffdbe24636f, 66036) = -1 EFAULT (Bad address)
read(3, 0x7ffdbe24636e, 66037) = -1 EFAULT (Bad address)
read(3, 0x7ffdbe24636d, 66038) = -1 EFAULT (Bad address)
read(3, 0x7ffdbe24636c, 66039) = -1 EFAULT (Bad address)
read(3, 0x7ffdbe24636b, 66040) = -1 EFAULT (Bad address)
....There's a bunch of sendto() and reads() with the return value (?) decreasing one by one, and then a bunch of those faults. I'm assuming this is what trashes the stack.
TODO: get a better strace and don't axe so much info
Nable80
fangfufu
splitice
Moving from a specific physical wifi network to another (and removing my laptop from the dock (TODO: rule this out)) causes the tincd I have running in a container (TODO: rule this out) to segfault reproducibly, but I haven't been able to isolate the bug. It consistently gets up to the
Sending 70 bytes of metadata to [redacted] ([redacted] port 20)portion of the attached log, and then (obviously at some point after this) segfaults.Starting tincd with gdb results in the stack being incomprehensible (to me). Starting it with
rr recordresults in a stack smashing notification, which probably explains the previous point. However due to anrr replayissue I haven't been able to yield any good info from that either. Maybe valgrind is worth a shot?Here is a log:
One may note (what I assume is) a race condition at
Cannot open config file /etc/tinc/gate/hosts/[redacted]: No such file or directory(probably caused by--chaos), which succeeds a few lines later.Notes on debugging: (mainly as a note to myself), systemd: this is running in a container on nixos, and the main service configs are on a read-only file system. This makes changing the service file to run a tincd with debug symbols, or otherwise, impossible. It turns out to be a bit obscure but
systemd-analyze unit-pathslist the list of paths systemd searches for service files. However I'm told this "isn't accurate" on nixos. Before that I found a list at https://www.freedesktop.org/software/systemd/man/systemd.unit.html (via https://askubuntu.com/questions/876733/where-are-the-systemd-units-services-located-in-ubuntu) , however I don't know how the list is derived, nor do I know how to change or add entries. Furthermore some of the directories listed seem to be special purpose(randomly putting the .service in some of them didn't work and caused systemd to not see the service or complain). (TODO: look into this)This was need for two debugging variants: gdb:
However, in the end it wasn't actually necessary to start tincd via the service files, the issues were reproducible just fine by calling the executable on the command line. I was just trying to keep reproduction in as similar an environment as the original. Part of my confusion was caused by tincd behaving different if started by gdb, than if gdb was just attached to it. This ended up being an issue with aslr, and was fixed with
set disable-randomisation off. I don't know what could cause this, but the symptoms were that tinc would seemingly hang instead of crashing.gdb variant: I ended up copying the containers existing tinc network service file to /run/systemd/system and editing the service files there. I used "nix-store --add" to add the Exec script to the nix store to work around an issue that I didn't look into further, involving when I tried to put the script in the same directory as the service file, and referring to it by absolute path in the service definition ("/run/systemd/system/gate-init" or such). That may have been a problem of me mixing up the outer system and the container, but I'm not sure. It might have been systemd sandboxing related. The gdb command ended up being: lost it in scroll, but its similar to the later rr command, something like
gdb path/to/tincd -ex "run -tinc -parameters -here". The tincd path used was the nix store path corresponding to the output ofnix-build -I nixpkgs=channel:nixos-unstable -E "with import <nixpkgs> {}; tinc_pre.overrideAttrs (old: { dontStrip = true; })" -v --no-out-link.rr variant: The first issue while trying to run rr in the container was
[FATAL /build/source/src/PerfCounters.cc:317:start_counter() errno: EPERM] Failed to initialize counterand some backtrace that I'm missing. I'd recently heard that rr does stuff with cpu performance counter registers, so I figured this was a container permissions issue (eventually discovering https://github.com/mozilla/rr/wiki/Docker ). So it turns out in the wonderful feat of engineering that systemd is,systemd-nspawndoesn't currently allow disabling it's seccomp protections per: https://lists.freedesktop.org/archives/systemd-devel/2020-June/044756.html . I ended up getting a list of syscalls from https://man7.org/linux/man-pages/man2/syscalls.2.html , cleaning it up a bit, and runningcat syscalls | cut -d " " -f 8 | cut -d "(" -f 1 | xargs -I{} echo --system-call-filter="{}" | xargs echoto generate the syscall whitelist. Using the systemd service methods listed above I constructed anixos-containerderived configuration manually (I really should have just generated a new container with the parameters set...). For anyone needing to do such a hack sometime, you can copy it from appendix 2. I also added--capability="CAP_SYS_PTRACE" \and--capability="CAP_SYS_ADMIN" \in the hopes that it would be enough, before gebnerating the whitelist, but I don't know if these did anything. Which is to say, I don't know which of all this stuff I did at the same time did anything. The rr command ended up being:/nix/store/yairbna66xfrib2snjdq9z5gxqxlk4bw-rr-5.3.0/bin/rr record /nix/store/3218jrqqrh8cbbnhijhh2vxw2m9vm85m-tinc-1.1pre17/bin/tincd -D -U gate -n gate --pidfile /run/gate.pid -d 4There was an issue at some point where trying to run the tincd in rr complained about some sort of dynamic linking error:I don't know the cause of this but I probably used the wrong tincd path (nixos lets you link different versions of stuff against eachother while on the same system), I just got the one from the above
gdb variantsection and it worked. After that therr recordcommand worked, but now we are at the stage whererr replayis broken with the appendix 1 message.Appendix 1: The assertion
Assertion!syscall_bp_vm' failed to hold.` is triggered at https://github.com/mozilla/rr/blob/153d46c1717e31bf3357abaaf0d42cbe35f6f672/src/ReplaySession.cc#L594Appendix 2: (TODO) Syscall whitelist for systemd-nspawn derived from https://man7.org/linux/man-pages/man2 . https://github.com/mozilla/rr/wiki/Docker lists some information about what would probably be needed but it was easier to just script it (
cat syscalls | cut -d " " -f 8 | cut -d "(" -f 1 | xargs -I{} echo --system-call-filter="{}" | xargs echo, where syscalls is a hand edited text file) and do everything. Note this only contains IA64 syscalls - ARM and other such syscalls have (perhaps unnecessarily - maybe it doesn't complain about missing ones) removed. If you look at the--system-call-filterentry on thesystemd-nspawnman page, it says something about wildcards or something. Maybe it's sufficient to pass "--system-call-filter=~*", but I didn't read carefully enough to tell if it would blacklist or whitelist everything, and I didn`t test this after the man page section was brought to my attention. Also you can pass a list to the argument, so that would also make it a lot nicer.Self TODO: cleaner systemd interactions and figure out what's going on with that stuff, get rr working, clean up the system call filter stuff