search

VIDA-NYU / reprozip

ReproZip is a tool that simplifies the process of creating reproducible experiments from command-line executions, a frequently-used common denominator in computational science.
https://www.reprozip.org/
BSD 3-Clause "New" or "Revised" License
302 stars 33 forks source link

tests: Old bundle no longer working on CI with 'directory' unpacker #373

Closed remram44 closed 3 years ago

remram44 commented 3 years ago

Those old bundles are mostly there to test for old ReproZip version, not the old software in them. However simple-0.4.0.rpz was not only recorded with ReproZip 0.4, but Debian 8.1.

This bundle runs fine with other isolation methods, including chroot (so using the same libs unpacked for directory). Interestingly the binary in there also runs fine if I don't use any of the bundle libraries, just the system's libc (Ubuntu 20.04).

This is difficult to debug because the crash happens at load time. Attaching gdb for example doesn't work ("During startup program terminated with signal SIGSEGV).ldd` and similar tools can't be used because they try to load the binary.

https://github.com/VIDA-NYU/reprozip/actions/runs/947409433 > `[REPROUNZIP] 18:23:31.015 WARNING: Rewrote command-line as: ./simple /tmp/reprozip_tests_ttgco5ee/simpledir/root/home/remram/Documents/programming/reprozip/reprozip/tests/simple_input.txt simple_output.txt [REPROUNZIP] 18:23:31.016 DEBUG: Running: export LD_LIBRARY_PATH=/tmp/reprozip_tests_ttgco5ee/simpledir/root/usr/lib/x86_64-linux-gnu/libfakeroot:/tmp/reprozip_tests_ttgco5ee/simpledir/root/usr/local/lib:/tmp/reprozip_tests_ttgco5ee/simpledir/root/lib/x86_64-linux-gnu:/tmp/reprozip_tests_ttgco5ee/simpledir/root/lib32:/tmp/reprozip_tests_ttgco5ee/simpledir/root/libx32:/tmp/reprozip_tests_ttgco5ee/simpledir/root/lib && cd /tmp/reprozip_tests_ttgco5ee/simpledir/root/tmp/reprozip_tests_q1mjxjhf && /usr/bin/env -i COLORFGBG="15;0" DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-wAyZwD1vI5,guid=45d4ddf6cf2e41558d59447155d3388a DESKTOP_SESSION=default DISPLAY=:0 DM_CONTROL=/var/run/xdmctl GPG_AGENT_INFO=/tmp/gpg-Icnp9P/S.gpg-agent:1180:1 GS_LIB=/home/remram/.fonts HOME=/home/remram KDE_FULL_SESSION=true KDE_MULTIHEAD=false KDE_SESSION_UID=1000 KDE_SESSION_VERSION=4 KONSOLE_DBUS_SERVICE=:1.96 KONSOLE_DBUS_SESSION=/Sessions/13 KONSOLE_DBUS_WINDOW=/Windows/1 KONSOLE_PROFILE_NAME=Shell LANG=en_US.UTF-8 LANGUAGE="" LOGNAME=remram LS_COLORS="rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:" OLDPWD=/home/remram/Documents/programming/reprozip/graph PAM_KWALLET_LOGIN=/tmp//remram.socket PROFILEHOME="" PS1="(reprozip34)\\[\\e]0;\\u@\\h: \\w\\a\\]\${debian_chroot:+(\$debian_chroot)}\\u@\\h:\\w\\\$ " *** QT_PLUGIN_PATH=/home/remram/.kde/lib/kde4/plugins/:/usr/lib/kde4/plugins/ SESSION_MANAGER="local/missy:@/tmp/.ICE-unix/1233,unix/missy:/tmp/.ICE-unix/1233" SHELL=/bin/bash SHELL_SESSION_ID=11a51ea391f74000a0fa60beb2bea4d3 SHLVL=1 SSH_AGENT_PID=1179 SSH_AUTH_SOCK=/tmp/ssh-7fXBqrM6mBVu/agent.1144 TERM=xterm USER=remram VIRTUAL_ENV=/home/remram/Documents/programming/_venv/reprozip34 WINDOWID=33554458 WINDOWPATH=7 XCURSOR_THEME=Oxygen_White XDG_CURRENT_DESKTOP=KDE XDG_DATA_DIRS=/usr/share:/usr/share:/usr/local/share XDG_RUNTIME_DIR=/run/user/1000 XDG_SEAT=seat0 XDG_SESSION_ID=1 XDG_VTNR=7 XDM_MANAGED=method=classic _=/home/remram/Documents/programming/_venv/reprozip34/bin/python PATH=/tmp/reprozip_tests_ttgco5ee/simpledir/root/home/remram/Documents/programming/_venv/reprozip34/bin:/tmp/reprozip_tests_ttgco5ee/simpledir/root/home/remram/pkgs/bin:/tmp/reprozip_tests_ttgco5ee/simpledir/root/usr/local/bin:/tmp/reprozip_tests_ttgco5ee/simpledir/root/usr/bin:/tmp/reprozip_tests_ttgco5ee/simpledir/root/bin:/tmp/reprozip_tests_ttgco5ee/simpledir/root/usr/local/games:/tmp/reprozip_tests_ttgco5ee/simpledir/root/usr/games:/home/remram/Documents/programming/_venv/reprozip34/bin:/home/remram/pkgs/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games ./simple /tmp/reprozip_tests_ttgco5ee/simpledir/root/home/remram/Documents/programming/reprozip/reprozip/tests/simple_input.txt simple_output.txt` > `Segmentation fault (core dumped)` > > `*** Command finished, status: 139`
strace env -i LD_LIBRARY_PATH=... ./simple ... ``` execve("/usr/bin/env", ["/usr/bin/env", "-i", "LD_LIBRARY_PATH=/tmp/simpledir/r"..., "./simple", "/tmp/simpledir/root/home/remram/"..., "simple_output.txt"], 0x7ffd29060e60 /* 64 vars */) = 0 ... execve("./simple", ["./simple", "/tmp/simpledir/root/home/remram/"..., "simple_output.txt"], 0x561687729440 /* 1 var */) = 0 brk(NULL) = 0x21b0000 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f91c0fb2000 access("/etc/ld.so.preload", R_OK) = 0 openat(AT_FDCWD, "/etc/ld.so.preload", O_RDONLY|O_CLOEXEC) = 3 fstat(3, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 close(3) = 0 openat(AT_FDCWD, "/tmp/simpledir/root/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\34\2\0\0\0\0\0"..., 832) = 832 fstat(3, {st_mode=S_IFREG|0755, st_size=1729984, ...}) = 0 mmap(NULL, 3836448, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f91c0c09000 mprotect(0x7f91c0da8000, 2097152, PROT_NONE) = 0 mmap(0x7f91c0fa8000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x19f000) = 0x7f91c0fa8000 mmap(0x7f91c0fae000, 14880, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f91c0fae000 close(3) = 0 mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f91c0c06000 arch_prctl(ARCH_SET_FS, 0x7f91c0c06700) = 0 mprotect(0x7f91c0fa8000, 16384, PROT_READ) = 0 mprotect(0x7f91c0fe1000, 4096, PROT_READ) = 0 --- SIGSEGV {si_signo=SIGSEGV, si_code=SI_KERNEL, si_addr=NULL} --- +++ killed by SIGSEGV (core dumped) +++ ```
strace env -i ./simple ... ``` execve("/usr/bin/env", ["/usr/bin/env", "-i", "./simple", "/tmp/simpledir/root/home/remram/"..., "simple_output.txt"], 0x7ffc4c2beb28 /* 64 vars */) = 0 ... execve("./simple", ["./simple", "/tmp/simpledir/root/home/remram/"..., "simple_output.txt"], 0x55f901f4e0d0 /* 0 vars */) = 0 brk(NULL) = 0x10d0000 arch_prctl(0x3001 /* ARCH_??? */, 0x7ffe40a81a20) = -1 EINVAL (Invalid argument) access("/etc/ld.so.preload", R_OK) = 0 openat(AT_FDCWD, "/etc/ld.so.preload", O_RDONLY|O_CLOEXEC) = 3 fstat(3, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 close(3) = 0 openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 fstat(3, {st_mode=S_IFREG|0644, st_size=237369, ...}) = 0 mmap(NULL, 237369, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f1d4f4d4000 close(3) = 0 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360q\2\0\0\0\0\0"..., 832) = 832 pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784 pread64(3, "\4\0\0\0\20\0\0\0\5\0\0\0GNU\0\2\0\0\300\4\0\0\0\3\0\0\0\0\0\0\0", 32, 848) = 32 pread64(3, "\4\0\0\0\24\0\0\0\3\0\0\0GNU\0\316x.\316\10\320\210\347~\352\334\10o\204\324\210"..., 68, 880) = 68 fstat(3, {st_mode=S_IFREG|0755, st_size=2029224, ...}) = 0 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1d4f4d2000 pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784 pread64(3, "\4\0\0\0\20\0\0\0\5\0\0\0GNU\0\2\0\0\300\4\0\0\0\3\0\0\0\0\0\0\0", 32, 848) = 32 pread64(3, "\4\0\0\0\24\0\0\0\3\0\0\0GNU\0\316x.\316\10\320\210\347~\352\334\10o\204\324\210"..., 68, 880) = 68 mmap(NULL, 2036952, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f1d4f2e0000 mprotect(0x7f1d4f305000, 1847296, PROT_NONE) = 0 mmap(0x7f1d4f305000, 1540096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x25000) = 0x7f1d4f305000 mmap(0x7f1d4f47d000, 303104, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x19d000) = 0x7f1d4f47d000 mmap(0x7f1d4f4c8000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1e7000) = 0x7f1d4f4c8000 mmap(0x7f1d4f4ce000, 13528, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f1d4f4ce000 close(3) = 0 arch_prctl(ARCH_SET_FS, 0x7f1d4f4d3540) = 0 mprotect(0x7f1d4f4c8000, 12288, PROT_READ) = 0 mprotect(0x7f1d4f53b000, 4096, PROT_READ) = 0 munmap(0x7f1d4f4d4000, 237369) = 0 brk(NULL) = 0x10d0000 brk(0x10f1000) = 0x10f1000 openat(AT_FDCWD, "/tmp/simpledir/root/home/remram/Documents/programming/reprozip/reprozip/tests/simple_input.txt", O_RDONLY) = 3 ... ```
remram44 commented 3 years ago

This is worked around by 0d6601c3: just use chroot. directory is not really supposed to work on different OS versions anyway, but I would like to know why this doesn't work.

remram44 commented 3 years ago

Interestingly this works if I call the linker directly. Maybe there's an incompatibility between the linker that gets called by env (host's) and the libc (bundle's)?

strace -f ../../lib64/ld-linux-x86-64.so.2 --library-path ... ./simple ... ``` execve("../../lib64/ld-linux-x86-64.so.2", ["../../lib64/ld-linux-x86-64.so.2", "--library-path", "/tmp/simpledir/root/lib64:/tmp/s"..., "./simple", "../../home/remram/Documents/prog"..., "simple_output.txt"], 0x7ffce8628d90 /* 64 vars */) = 0 brk(NULL) = 0x555556cdb000 open("./simple", O_RDONLY|O_CLOEXEC) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\2\0>\0\1\0\0\0\340\5@\0\0\0\0\0"..., 832) = 832 fstat(3, {st_mode=S_IFREG|0755, st_size=7976, ...}) = 0 getcwd("/tmp/simpledir/root/tmp/reprozip_tests_q1mjxjhf", 128) = 48 mmap(0x400000, 4096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0) = 0x400000 mmap(0x600000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0) = 0x600000 close(3) = 0 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe1bd2ba000 access("/etc/ld.so.preload", R_OK) = 0 open("/etc/ld.so.preload", O_RDONLY|O_CLOEXEC) = 3 fstat(3, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 close(3) = 0 open("/tmp/simpledir/root/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\34\2\0\0\0\0\0"..., 832) = 832 fstat(3, {st_mode=S_IFREG|0755, st_size=1729984, ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe1bd2b9000 mmap(NULL, 3836448, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fe1bccf3000 mprotect(0x7fe1bce92000, 2097152, PROT_NONE) = 0 mmap(0x7fe1bd092000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x19f000) = 0x7fe1bd092000 mmap(0x7fe1bd098000, 14880, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fe1bd098000 close(3) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe1bd2b8000 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe1bd2b7000 arch_prctl(ARCH_SET_FS, 0x7fe1bd2b8700) = 0 mprotect(0x7fe1bd092000, 16384, PROT_READ) = 0 mprotect(0x7fe1bd2bc000, 4096, PROT_READ) = 0 brk(NULL) = 0x555556cdb000 brk(0x555556cfc000) = 0x555556cfc000 open("../../home/remram/Documents/programming/reprozip/reprozip/tests/simple_input.txt", O_RDONLY) = 3 ... ```
remram44 commented 3 years ago

I can check that running patchelf --set-interpreter /tmp/simpledir/root/lib64/ld-linux-x86-64.so.2 /tmp/simpledir/root/tmp/reprozip_tests_q1mjxjhf/simple allows reprounzip directory run to work. I guess the directory unpacker needs to be updated to call ld-linux.so directly?

remram44 commented 3 years ago

Fixed by 0768cf99