segfault during recoding

[ChipMonto]

Hi, Thanks for sharing your tool. I'm trying to run it on an embedded system that's based on Centos 8. EnvFuzz is built fine, however recording doesn't work, even for simple commands like ./env-fuzz record -- ls -la / It returns 'EXIT 0' but the RECORD.pcap is empty. How can I investigate what could be the problem?

I've also tried the same in a Centos8 docker, and an attempt to record the same command results in a segfault. Attached is the strace output of that run. If an old libc or other dependencies are a problem, would it be possible to build EnvFuzz statically to make it usable on older systems?

Any help would be appreciated. Thanks, CM EnvFuzz.strace.gz

[ChipMonto]

I've just tried copying RECORD.pcap.gz for the same command from another (modern Ubuntu) host where EnvFuzz is working to see if it get things going on the target system. However, still no luck, so there's something else going on here.

[GJDuck]

How can I investigate what could be the problem?

Likely the LD_LIBRARY_PATH variable (set by the env-fuzz wrapper) is being ignored. This could be caused by:

1. The binary is statically linked
2. The binary runs in AT_SECURE mode

Since this is for an embedded system, maybe ls is statically linked? Try file /usr/bin/ls to check.

Currently, statically linked binaries are not "officially" supported. However, it might be possible to applying the instrumentation to the executable itself, rather than libc, but this has never really been tested.

the same command results in a segfault.

This could be an ordinary bug, in which case I'd need more information.

I've just tried copying RECORD.pcap.gz for the same command from another (modern Ubuntu) host

The record&replay infrastructure was not desgined for this use case, so I think it is unlikely to work.

[ChipMonto]

Ok, so I missed the fact that EnvFuzz will work only with 64-bit binaries, so we can cross off the 0-sized RECORD.pcap. However, the segfault remains an issue. My target system is a multilib system and I can reproduce it also there as well as inside my Centos8 docker. How can I help you with more information? Below is the gdb session but I doubt it's very helpful:

(gdb) set follow-exec-mode new
(gdb) b openPCAPFile
Breakpoint 1 at 0x404740: file env-fuzz.cpp, line 158.
(gdb) r
Starting program: /root/EnvFuzz/env-fuzz record -- /usr/bin/coreutils --coreutils-prog-shebang=ls /

Breakpoint 1, openPCAPFile (filename=0x61af00 "/root/EnvFuzz/out/RECORD.pcap.gz", mode=mode@entry=119 'w') at env-fuzz.cpp:158
158         switch (mode)
(gdb) list
153                 oldfd, newfd, strerror(errno));
154     }
155     static int openPCAPFile(const char *filename, char mode)
156     {
157         int fd = -1;
158         switch (mode)
159         {
160             case 'r':
161                 fd = open(filename, O_RDONLY);
162                 break;
(gdb) n
164                 fd = open(filename, O_WRONLY | O_CREAT, 0644);
(gdb) n
169         if (fd < 0)
(gdb) n
174         if (pipe(fds) < 0)
(gdb)
176         pid_t child = fork();
(gdb)
[Attaching after process 1451 fork to child process 1455]
[New inferior 2 (process 1455)]
[Detaching after fork from parent process 1451]
[Inferior 1 (process 1451) detached]
[Switching to process 1455]
openPCAPFile (filename=0x61af00 "/root/EnvFuzz/out/RECORD.pcap.gz", mode=mode@entry=119 'w') at env-fuzz.cpp:177
177         if (child < 0)
(gdb) Segmentation fault

The pcap file is partially created (attached). Please let me know what additional information you need. RECORD.pcap.gz

[GJDuck]

I think the segault is real but I am unable to reproduce it (your RECORD.pcap.gz does not cause a segfault for me). Your gdb session is not clear, it just prints "Segmentation fault" on the last line without GDB stopping or showing the location?

[ChipMonto]

After some digging in, it looks as if the thread running the recorded program crashes. I've tried recording different commands, including simple hello world program and they all crash at the same place in the dynamic linker inside do_lookup_x() function. As I understand it's trying to resolve 'clock_gettime' symbol. Please see the attached gdb session output. Again, this is only on my Centos8 system. gdb_session.txt

[GJDuck]

It is not clear what the underlying problem is. Although EnvFuzz modifies the VDSO and clock_gettime(), I do not immediately see a connection with do_lookup_x() which is part of symbol resolution I think. Does the problem only occur on Centos8?

[ChipMonto]

Well, the only other system I tried is Ubuntu 24.04, which works fine. The only reason I'm using Centos8 is because that's what my target is based on, which means it's easy for me to build EnvFuzz for it.

[GJDuck]

The current system was developed on Ubuntu, so there is a likely a bug that is only triggered on different systems. I will have to try it on a Red Hat installation, and will report back.

[ChipMonto]

That'd be great, thanks in advance!

[GJDuck]

My colleage (@Marti2203) tried env-fuzz on SuSE and other RPM-based Linux. The recording worked (no crash), but the replay did not (replay not in sync with recording). Probably env-fuzz is limited to Ubuntu-like systems for now until the tool becomes more mature.

[ChipMonto]

Understood, thank you for the update.