openSUSE / obs-build

OBS build script, can be used with OBS or stand alone
GNU General Public License v2.0
132 stars 183 forks source link

SIGINT-ing osc build leads to infinite loop in obs-build #898

Closed jengelh closed 1 year ago

jengelh commented 1 year ago

When osc build is aborted, it leaves behind a process fragment that is running in an infinite loop. When ^C is first hit, ps xaf is like (so far so good):

27535 pts/1    S      0:00 /bin/bash /usr/bin/build --root=/run/ob/openSUSE_Tumbleweed-i586 --rpmlist=/tmp/rpmlist.4q48f6yj --dist=..
29030 pts/1    S      0:00  \_ su -c /.build.command - abuild
29031 ?        Zs     0:00      \_ [rpmbuild] <defunct>
29923 ?        S      0:00 /usr/bin/c++ -DDYN_GTK=1 -DHAVE_MMX=1 -DUSE_LIBVPX=1 -D__forceinline=inline -Dstricmp=strcasecmp -Dstrnicmp=strncasecmp
30509 ?        R      0:01  \_ /usr/lib/gcc/i586-suse-linux/12/../../../../i586-suse-linux/bin/as -I /usr/include/glslang/Public -I /usr/include/SP
30301 ?        S      0:00 /usr/bin/c++ -DDYN_GTK=1 -DHAVE_MMX=1 -DUSE_LIBVPX=1 -D__forceinline=inline -Dstricmp=strcasecmp -Dstrnicmp=strncasecmp
30302 ?        R      0:10  \_ /usr/lib/gcc/i586-suse-linux/12/cc1plus -quiet -I /usr/include/glslang/Public -I /usr/include/SPIRV -I /usr/include/
30454 ?        S      0:00 /usr/bin/c++ -DDYN_GTK=1 -DHAVE_MMX=1 -DUSE_LIBVPX=1 -D__forceinline=inline -Dstricmp=strcasecmp -Dstrnicmp=strncasecmp
30455 ?        R      0:03  \_ /usr/lib/gcc/i586-suse-linux/12/cc1plus -quiet -I /usr/include/glslang/Public -I /usr/include/SPIRV -I /usr/include/
[...]

then after a second or two,

27535 pts/1    R      0:04 /bin/bash /usr/bin/build --root=/run/ob/openSUSE_Tumbleweed-i586 --rpmlist=/tmp/rpmlist.4q48f6yj --dist=/home/j/gzdoom/.osc/_buildconfig-openSUSE_Tumbleweed-i586 --arch=i586 --statistics --norootforbuild --changelog --jobs=16 --debug /home/j/gzdoom/gzdoom.spec
strace: Process 27535 attached
newfstatat(AT_FDCWD, "/run/ob/openSUSE_Tumbleweed-i586//proc/sys/fs/binfmt_misc", {st_mode=S_IFDIR|0555, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(AT_FDCWD, "/run/ob/openSUSE_Tumbleweed-i586//proc/sys/fs/binfmt_misc", {st_mode=S_IFDIR|0555, st_size=0, ...}, 0) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [CHLD], 8) = 0
rt_sigprocmask(SIG_SETMASK, [CHLD], NULL, 8) = 0
newfstatat(AT_FDCWD, "/run/ob/openSUSE_Tumbleweed-i586//proc/sys/fs/binfmt_misc", {st_mode=S_IFDIR|0555, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(AT_FDCWD, "/run/ob/openSUSE_Tumbleweed-i586//proc/sys/fs/binfmt_misc", {st_mode=S_IFDIR|0555, st_size=0, ...}, 0) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [CHLD], 8) = 0
rt_sigprocmask(SIG_SETMASK, [CHLD], NULL, 8) = 0
newfstatat(AT_FDCWD, "/run/ob/openSUSE_Tumbleweed-i586//proc/sys/fs/binfmt_misc", {st_mode=S_IFDIR|0555, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(AT_FDCWD, "/run/ob/openSUSE_Tumbleweed-i586//proc/sys/fs/binfmt_misc", {st_mode=S_IFDIR|0555, st_size=0, ...}, 0) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [CHLD], 8) = 0
...

and it never gets to clean up the remaining mounts

none           proc          0     0     0    - /run/ob/openSUSE_Tumbleweed-i586/proc
none           proc          0     0     0    - /var/run/ob/openSUSE_Tumbleweed-i586/proc
none           -             -     -     -    - /run/ob/openSUSE_Tumbleweed-i586/sys
none           -             -     -     -    - /var/run/ob/openSUSE_Tumbleweed-i586/sys
none           devpts        0     0     0    - /run/ob/openSUSE_Tumbleweed-i586/dev/pts
none           devpts        0     0     0    - /var/run/ob/openSUSE_Tumbleweed-i586/dev/pts
none           tmpfs       32G     0   32G   0% /run/ob/openSUSE_Tumbleweed-i586/dev/shm
none           tmpfs       32G     0   32G   0% /var/run/ob/openSUSE_Tumbleweed-i586/dev/shm
marxin commented 1 year ago

I face the very same issue.

marxin commented 1 year ago

Any update on this, please? It's really annoying issue as it's leaving processes that consume 100% of a CPU :/

jengelh commented 1 year ago
++ buildroot_umount /proc/sys/fs/binfmt_misc
++ local d=/proc/sys/fs/binfmt_misc
++ local d2=//proc/sys/fs/binfmt_misc
++ echo 'Unmounting /proc/sys/fs/binfmt_misc'
Unmounting /proc/sys/fs/binfmt_misc
++ test -n //proc/sys/fs/binfmt_misc
++ test -L /run/ob/openSUSE_Factory-x86_64//proc/sys/fs/binfmt_misc
++ test -d /run/ob/openSUSE_Factory-x86_64//proc/sys/fs/binfmt_misc
++ d2=//proc/sys/fs/binfmt_misc
++ test -n //proc/sys/fs/binfmt_misc
++ test -L /run/ob/openSUSE_Factory-x86_64//proc/sys/fs/binfmt_misc
++ test -d /run/ob/openSUSE_Factory-x86_64//proc/sys/fs/binfmt_misc
++ d2=//proc/sys/fs/binfmt_misc
++ test -n //proc/sys/fs/binfmt_misc
++ test -L /run/ob/openSUSE_Factory-x86_64//proc/sys/fs/binfmt_misc
++ test -d /run/ob/openSUSE_Factory-x86_64//proc/sys/fs/binfmt_misc

There's only one place with d2=.

It looks like d2="${d2%/*}" has no effect, but then this is something of a bash bug. (Btw, a revert to older obs-build-2021xxxx also did not cure it, which strengths the idea that sh it at fault.)

jengelh commented 1 year ago

Reverting bash to 5.1.15 fixes the problem with obs-build. The git repository at https://git.savannah.gnu.org/git/bash.git is terrible—it's just tarball imports and thus not fine grained enough to pinpoint the problematic change.

4491c03014008046746b569108b7ea629d963e71 is the first bad commit
commit 4491c03014008046746b569108b7ea629d963e71
Author: Chet Ramey <chet.ramey@case.edu>
Date:   Thu Jan 20 15:06:05 2022 -0500

    bash-5.2-alpha release
marxin commented 1 year ago

I bisected the devel branch and the issue is fixed with:

commit 055a4552c901e43e6a6007f5bc664a33698dbf85 (HEAD)
Author: Chet Ramey <chet.ramey@case.edu>
Date:   Sat Oct 15 13:34:51 2022 -0400

    change to readline callback signal handling; fix message when popping shell function context; handle new terminating signals inside terminating signal handler; fix to pattern substitution with null replacement and null string; fix to alias expansion when we get a SIGINT while entering a command substitution in an interactive shell

Can we cherry-pick it for our bash package?

marxin commented 1 year ago

Apparently, one can't easily apply the patch on top of 5.2 release.

marxin commented 1 year ago

I was promised by the bash maintainer, that the fix will be included in the next minor release.

marxin commented 1 year ago

Fixed in Factory with an update of bash to Update to bash52-015. @jengelh Please close it.