search

bstansell / conserver

Logged, multi-user access to device consoles
https://www.conserver.com/
BSD 3-Clause "New" or "Revised" License
130 stars 38 forks source link

buffer overflow detected, Spawn() in group.c #93

Open agialluc opened 1 year ago

agialluc commented 1 year ago

Recently we have been seeing several: "conserver[22453]: buffer overflow detected : /usr/sbin/conserver terminated"

I have looked and I note that sprintf ( ) is being used in cutil.c in the functions FileOpenFD( ) FileOpenPipe( ) and FileOpen( ).

I believe using snprintf( ) would prevent this from happening and would be a simple fix.

Please note: I created a merge which I think will fix this, but I do not normally code so it should be check carefully: https://github.com/bstansell/conserver/pull/92

fweimer-rh commented 1 year ago

Have you actually attributed the crash to a specific sprintf call? What are its arguments?

agialluc commented 1 year ago

I should have thought to use snprintf(opt, sizeof(opt), .... ) it is much cleaner.

We had to rebuild the debuginfo rpm, there was a problem with it. We are waiting to see when it re-occurs.

Here is part of one of the Backtrace's that we got: From /var/log/conserver: [Wed Sep 20 08:57:32 2023] conserver (46864): [ibm-hs22-5.swcert.cee.pnq.redhat.com] exit(2) [Wed Sep 20 08:57:32 2023] conserver (46864): [ibm-hs22-5.swcert.cee.pnq.redhat.com] automatic reinitialization buffer overflow detected : /usr/sbin/conserver terminated [Wed Sep 20 08:57:32 2023] conserver (56606): [dell-per320-03.khw2.lab.eng.bos.redhat.com] exit(1) [Wed Sep 20 08:57:32 2023] conserver (56606): [dell-per320-03.khw2.lab.eng.bos.redhat.com] automatic reinitialization [Wed Sep 20 08:57:32 2023] conserver (47817): [dev203.mw.lab.eng.bos.redhat.com] exit(1) [Wed Sep 20 08:57:32 2023] conserver (47817): [dev203.mw.lab.eng.bos.redhat.com] automatic reinitialization ======= Backtrace: ========= /lib64/libc.so.6(__fortify_fail+0x37)[0x7facde1987a7] /lib64/libc.so.6(+0x116922)[0x7facde196922] /lib64/libc.so.6(+0x118707)[0x7facde198707] /usr/sbin/conserver(+0x158d2)[0x558ddb5468d2] /usr/sbin/conserver(+0x2581a)[0x558ddb55681a] /usr/sbin/conserver(+0x1944f)[0x558ddb54a44f] /usr/sbin/conserver(+0x78f8)[0x558ddb5388f8] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7facde0a2555] /usr/sbin/conserver(+0x7c79)[0x558ddb538c79] ======= Memory map: ======== 558ddb531000-558ddb56b000 r-xp 00000000 fd:00 50342149 /usr/sbin/conserver 558ddb76b000-558ddb76c000 r--p 0003a000 fd:00 50342149 /usr/sbin/conserver 558ddb76c000-558ddb76e000 rw-p 0003b000 fd:00 50342149 /usr/sbin/conserver 558ddb76e000-558ddb76f000 rw-p 00000000 00:00 0 558ddc853000-558ddd8b2000 rw-p 00000000 00:00 0 [heap] 558ddd8b2000-558dde1b6000 rw-p 00000000 00:00 0 [heap] 7facd5874000-7facd5877000 r-xp 00000000 fd:00 50752891 /usr/lib64/krb5/plugins/libkrb5/sssd_krb5_locator_plugin.so 7facd5877000-7facd5a76000 ---p 00003000 fd:00 50752891 /usr/lib64/krb5/plugins/libkrb5/sssd_krb5_locator_plugin.so

( Many more lines trucated here)

7facdf5fd000-7facdf5ff000 r-xp 00000000 fd:00 50413178 /usr/lib64/libutil-2.17.so 7facdf5ff000-7facdf7fe000 ---p 00002000 fd:00 50413178 /usr/lib64/libutil-2.17.so 7facdf7fe000-7facdf7ff000 r--p 00001000 fd:00 50413178 /usr/lib64/libutil-2.17.so 7facdf7ff000-7facdf800000 rw-p 00002000 fd:00 50413178 /usr/lib64/libutil-2.17.so 7facdf800000-7facdf822000 r-xp 00000000 fd:00 50353406 /usr/lib64/ld-2.17.so 7facdf8d4000-7facdf95c000 r--s 00000000 00:14 1072190834 /run/nscd/db8ERaBw (deleted) 7facdfa0a000-7facdfa18000 rw-p 00000000 00:00 0 7facdfa1d000-7facdfa1e000 rw-p 00000000 00:00 0 7facdfa1e000-7facdfa1f000 rw-p 00000000 00:00 0 7facdfa1f000-7facdfa21000 rw-p 00000000 00:00 0 7facdfa21000-7facdfa22000 r--p 00021000 fd:00 50353406 /usr/lib64/ld-2.17.so 7facdfa22000-7facdfa23000 rw-p 00022000 fd:00 50353406 /usr/lib64/ld-2.17.so 7facdfa23000-7facdfa24000 rw-p 00000000 00:00 0 7fffdfdb0000-7fffdfde0000 rw-p 00000000 00:00 0 [stack] 7fffdfdea000-7fffdfdec000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] [Wed Sep 20 08:57:32 2023] conserver (36831): child pid 64970: signal(6), restarting buffer overflow detected : /usr/sbin/conserver terminated ======= Backtrace: ========= /lib64/libc.so.6(__fortify_fail+0x37)[0x7facde1987a7] /lib64/libc.so.6(+0x116922)[0x7facde196922] /lib64/libc.so.6(+0x118707)[0x7facde198707] /usr/sbin/conserver(+0x158d2)[0x558ddb5468d2] /usr/sbin/conserver(+0x1960a)[0x558ddb54a60a] /usr/sbin/conserver(+0x78f8)[0x558ddb5388f8] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7facde0a2555] /usr/sbin/conserver(+0x7c79)[0x558ddb538c79]

( next buffer overflow here )

agialluc commented 1 year ago

I was away, however we did capture a backtrace from another buffer overflow, hopefully this is of some use:

egrep -m 1 -A 350 -B 10 'buffer over' /var/log/conserver

[Mon Oct 23 15:25:51 2023] conserver (43550): [[code]iot-r6s2-01.wlan.rhts.eng.bos.redhat.com[/code]] console initializing [Mon Oct 23 15:25:51 2023] conserver (37502): [[code]ibm-x3650m4-02.ibm2.lab.eng.bos.redhat.com[/code]] exit(1) [Mon Oct 23 15:25:51 2023] conserver (37502): [[code]ibm-x3650m4-02.ibm2.lab.eng.bos.redhat.com[/code]] automatic reinitialization [Mon Oct 23 15:25:51 2023] conserver (43550): [[code]iot-r6s1-01.wlan.rhts.eng.bos.redhat.com[/code]] console initializing [Mon Oct 23 15:25:51 2023] conserver (31669): [[code]ibm-p9b-40.ibm2.lab.eng.bos.redhat.com[/code]] exit(1) [Mon Oct 23 15:25:51 2023] conserver (31669): [[code]ibm-p9b-40.ibm2.lab.eng.bos.redhat.com[/code]] automatic reinitialization [Mon Oct 23 15:25:51 2023] conserver (41721): [[code]nec-em20.khw2.lab.eng.bos.redhat.com[/code]] exit(1) [Mon Oct 23 15:25:51 2023] conserver (41721): [[code]nec-em20.khw2.lab.eng.bos.redhat.com[/code]] automatic reinitialization [Mon Oct 23 15:25:51 2023] conserver (41814): [[code]jaguar.storage.lab.eng.bos.redhat.com[/code]] exit(1) [Mon Oct 23 15:25:51 2023] conserver (41814): [[code]jaguar.storage.lab.eng.bos.redhat.com[/code]] automatic reinitialization buffer overflow detected : /usr/sbin/conserver terminated ======= Backtrace: ========= /lib64/libc.so.6(__fortify_fail+0x37)[0x7f9caa6987a7] /lib64/libc.so.6(+0x116922)[0x7f9caa696922] /lib64/libc.so.6(+0x118707)[0x7f9caa698707] /usr/sbin/conserver(+0x158b2)[0x56388f4368b2] /usr/sbin/conserver(+0x2588a)[0x56388f44688a] /usr/sbin/conserver(+0x1942f)[0x56388f43a42f] /usr/sbin/conserver(+0x78d8)[0x56388f4288d8] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f9caa5a2555] /usr/sbin/conserver(+0x7c58)[0x56388f428c58] ======= Memory map: ======== 56388f421000-56388f45b000 r-xp 00000000 fd:00 50376691 /usr/sbin/conserver 56388f65b000-56388f65c000 r--p 0003a000 fd:00 50376691 /usr/sbin/conserver 56388f65c000-56388f65e000 rw-p 0003b000 fd:00 50376691 /usr/sbin/conserver 56388f65e000-56388f65f000 rw-p 00000000 00:00 0 56388f80d000-56389086b000 rw-p 00000000 00:00 0 [heap] 56389086b000-563891306000 rw-p 00000000 00:00 0 [heap] 7f9ca1d74000-7f9ca1d77000 r-xp 00000000 fd:00 50752891 /usr/lib64/krb5/plugins/libkrb5/sssd_krb5locatorplugin.so 7f9ca1d77000-7f9ca1f76000 ---p 00003000 fd:00 50752891 /usr/lib64/krb5/plugins/libkrb5/sssd_krb5locatorplugin.so 7f9ca1f76000-7f9ca1f77000 r--p 00002000 fd:00 50752891 /usr/lib64/krb5/plugins/libkrb5/sssd_krb5locatorplugin.so 7f9ca1f77000-7f9ca1f78000 rw-p 00003000 fd:00 50752891 /usr/lib64/krb5/plugins/libkrb5/sssd_krb5locatorplugin.so 7f9ca3ad7000-7f9ca3ade000 r-xp 00000000 fd:00 50413174 /usr/lib64/librt-2.17.so 7f9ca3ade000-7f9ca3cdd000 ---p 00007000 fd:00 50413174 /usr/lib64/librt-2.17.so 7f9ca3cdd000-7f9ca3cde000 r--p 00006000 fd:00 50413174 /usr/lib64/librt-2.17.so 7f9ca3cde000-7f9ca3cdf000 rw-p 00007000 fd:00 50413174 /usr/lib64/librt-2.17.so 7f9ca5b87000-7f9ca5b9c000 r-xp 00000000 fd:00 51225355 /usr/lib64/libgccs-4.8.5-20150702.so.1 7f9ca5b9c000-7f9ca5d9b000 ---p 00015000 fd:00 51225355 /usr/lib64/libgccs-4.8.5-20150702.so.1 7f9ca5d9b000-7f9ca5d9c000 r--p 00014000 fd:00 51225355 /usr/lib64/libgccs-4.8.5-20150702.so.1 7f9ca5d9c000-7f9ca5d9d000 rw-p 00015000 fd:00 51225355 /usr/lib64/libgccs-4.8.5-20150702.so.1 7f9ca5d9d000-7f9ca5da1000 r-xp 00000000 fd:00 17048267 /usr/lib64/krb5/plugins/authdata/sssdpacplugin.so 7f9ca5da1000-7f9ca5fa0000 ---p 00004000 fd:00 17048267 /usr/lib64/krb5/plugins/authdata/sssdpacplugin.so 7f9ca5fa0000-7f9ca5fa1000 r--p 00003000 fd:00 17048267 /usr/lib64/krb5/plugins/authdata/sssdpacplugin.so 7f9ca5fa1000-7f9ca5fa2000 rw-p 00004000 fd:00 17048267 /usr/lib64/krb5/plugins/authdata/sssdpacplugin.so 7f9ca5fa2000-7f9ca5fc0000 r-xp 00000000 fd:00 50413522 /usr/lib64/libgssrpc.so.4.2 7f9ca5fc0000-7f9ca61c0000 ---p 0001e000 fd:00 50413522 /usr/lib64/libgssrpc.so.4.2 7f9ca61c0000-7f9ca61c1000 r--p 0001e000 fd:00 50413522 /usr/lib64/libgssrpc.so.4.2 7f9ca61c1000-7f9ca61c2000 rw-p 0001f000 fd:00 50413522 /usr/lib64/libgssrpc.so.4.2 7f9ca61c2000-7f9ca61dd000 r-xp 00000000 fd:00 362057 /usr/lib64/gssproxy/proxymech.so 7f9ca61dd000-7f9ca63dd000 ---p 0001b000 fd:00 362057 /usr/lib64/gssproxy/proxymech.so 7f9ca63dd000-7f9ca63de000 r--p 0001b000 fd:00 362057 /usr/lib64/gssproxy/proxymech.so 7f9ca63de000-7f9ca63df000 rw-p 0001c000 fd:00 362057 /usr/lib64/gssproxy/proxymech.so 7f9ca63df000-7f9ca6a2f000 r--s 00000000 fd:00 50333058 /var/lib/sss/mc/group 7f9ca6a2f000-7f9ca6a3b000 r-xp 00000000 fd:00 50413162 /usr/lib64/libnssfiles-2.17.so 7f9ca6a3b000-7f9ca6c3a000 ---p 0000c000 fd:00 50413162 /usr/lib64/libnssfiles-2.17.so 7f9ca6c3a000-7f9ca6c3b000 r--p 0000b000 fd:00 50413162 /usr/lib64/libnssfiles-2.17.so 7f9ca6c3b000-7f9ca6c3c000 rw-p 0000c000 fd:00 50413162 /usr/lib64/libnssfiles-2.17.so 7f9ca6c3c000-7f9ca6c42000 rw-p 00000000 00:00 0 7f9ca6c42000-7f9ca74ad000 r--s 00000000 fd:00 50333049 /var/lib/sss/mc/passwd 7f9ca74ad000-7f9ca74b5000 r-xp 00000000 fd:00 50340076 /usr/lib64/libnsssss.so.2 7f9ca74b5000-7f9ca76b4000 ---p 00008000 fd:00 50340076 /usr/lib64/libnsssss.so.2 7f9ca76b4000-7f9ca76b5000 r--p 00007000 fd:00 50340076 /usr/lib64/libnsssss.so.2 7f9ca76b5000-7f9ca76b6000 rw-p 00008000 fd:00 50340076 /usr/lib64/libnsssss.so.2 7f9ca76b6000-7f9ca7716000 r-xp 00000000 fd:00 50413670 /usr/lib64/libpcre.so.1.2.0 7f9ca7716000-7f9ca7916000 ---p 00060000 fd:00 50413670 /usr/lib64/libpcre.so.1.2.0 7f9ca7916000-7f9ca7917000 r--p 00060000 fd:00 50413670 /usr/lib64/libpcre.so.1.2.0 7f9ca7917000-7f9ca7918000 rw-p 00061000 fd:00 50413670 /usr/lib64/libpcre.so.1.2.0 7f9ca7918000-7f9ca791c000 r-xp 00000000 fd:00 50413812 /usr/lib64/libcap-ng.so.0.0.0 7f9ca791c000-7f9ca7b1c000 ---p 00004000 fd:00 50413812 /usr/lib64/libcap-ng.so.0.0.0 7f9ca7b1c000-7f9ca7b1d000 r--p 00004000 fd:00 50413812 /usr/lib64/libcap-ng.so.0.0.0 7f9ca7b1d000-7f9ca7b1e000 rw-p 00005000 fd:00 50413812 /usr/lib64/libcap-ng.so.0.0.0 7f9ca7b1e000-7f9ca7b22000 r-xp 00000000 fd:00 50413781 /usr/lib64/libgpg-error.so.0.10.0 7f9ca7b22000-7f9ca7d21000 ---p 00004000 fd:00 50413781 /usr/lib64/libgpg-error.so.0.10.0 7f9ca7d21000-7f9ca7d22000 r--p 00003000 fd:00 50413781 /usr/lib64/libgpg-error.so.0.10.0 7f9ca7d22000-7f9ca7d23000 rw-p 00004000 fd:00 50413781 /usr/lib64/libgpg-error.so.0.10.0 7f9ca7d23000-7f9ca7d47000 r-xp 00000000 fd:00 50412459 /usr/lib64/libselinux.so.1 7f9ca7d47000-7f9ca7f46000 ---p 00024000 fd:00 50412459 /usr/lib64/libselinux.so.1 7f9ca7f46000-7f9ca7f47000 r--p 00023000 fd:00 50412459 /usr/lib64/libselinux.so.1 7f9ca7f47000-7f9ca7f48000 rw-p 00024000 fd:00 50412459 /usr/lib64/libselinux.so.1 7f9ca7f48000-7f9ca7f4a000 rw-p 00000000 00:00 0 7f9ca7f4a000-7f9ca7f68000 r-xp 00000000 fd:00 50413817 /usr/lib64/libaudit.so.1.0.0 7f9ca7f68000-7f9ca8167000 ---p 0001e000 fd:00 50413817 /usr/lib64/libaudit.so.1.0.0 7f9ca8167000-7f9ca8168000 r--p 0001d000 fd:00 50413817 /usr/lib64/libaudit.so.1.0.0 7f9ca8168000-7f9ca8169000 rw-p 0001e000 fd:00 50413817 /usr/lib64/libaudit.so.1.0.0 7f9ca8169000-7f9ca8173000 rw-p 00000000 00:00 0 7f9ca8173000-7f9ca8274000 r-xp 00000000 fd:00 50413152 /usr/lib64/libm-2.17.so 7f9ca8274000-7f9ca8473000 ---p 00101000 fd:00 50413152 /usr/lib64/libm-2.17.so 7f9ca8473000-7f9ca8474000 r--p 00100000 fd:00 50413152 /usr/lib64/libm-2.17.so 7f9ca8474000-7f9ca8475000 rw-p 00101000 fd:00 50413152 /usr/lib64/libm-2.17.so 7f9ca8475000-7f9ca84f2000 r-xp 00000000 fd:00 50413792 /usr/lib64/libgcrypt.so.11.8.2 7f9ca84f2000-7f9ca86f1000 ---p 0007d000 fd:00 50413792 /usr/lib64/libgcrypt.so.11.8.2 7f9ca86f1000-7f9ca86f2000 r--p 0007c000 fd:00 50413792 /usr/lib64/libgcrypt.so.11.8.2 7f9ca86f2000-7f9ca86f5000 rw-p 0007d000 fd:00 50413792 /usr/lib64/libgcrypt.so.11.8.2 7f9ca86f5000-7f9ca86f6000 rw-p 00000000 00:00 0 7f9ca86f6000-7f9ca870d000 r-xp 00000000 fd:00 50413170 /usr/lib64/libpthread-2.17.so 7f9ca870d000-7f9ca890c000 ---p 00017000 fd:00 50413170 /usr/lib64/libpthread-2.17.so 7f9ca890c000-7f9ca890d000 r--p 00016000 fd:00 50413170 /usr/lib64/libpthread-2.17.so 7f9ca890d000-7f9ca890e000 rw-p 00017000 fd:00 50413170 /usr/lib64/libpthread-2.17.so 7f9ca890e000-7f9ca8912000 rw-p 00000000 00:00 0 7f9ca8912000-7f9ca8c12000 r-xp 00000000 fd:00 50348982 /usr/lib64/libfreeipmi.so.17.1.4 7f9ca8c12000-7f9ca8e11000 ---p 00300000 fd:00 50348982 /usr/lib64/libfreeipmi.so.17.1.4 7f9ca8e11000-7f9ca8e74000 r--p 002ff000 fd:00 50348982 /usr/lib64/libfreeipmi.so.17.1.4 7f9ca8e74000-7f9ca8ffb000 rw-p 00362000 fd:00 50348982 /usr/lib64/libfreeipmi.so.17.1.4 7f9ca8ffb000-7f9ca9012000 r-xp 00000000 fd:00 50413154 /usr/lib64/libnsl-2.17.so 7f9ca9012000-7f9ca9211000 ---p 00017000 fd:00 50413154 /usr/lib64/libnsl-2.17.so 7f9ca9211000-7f9ca9212000 r--p 00016000 fd:00 50413154 /usr/lib64/libnsl-2.17.so 7f9ca9212000-7f9ca9213000 rw-p 00017000 fd:00 50413154 /usr/lib64/libnsl-2.17.so 7f9ca9213000-7f9ca9215000 rw-p 00000000 00:00 0 7f9ca9215000-7f9ca922b000 r-xp 00000000 fd:00 50413172 /usr/lib64/libresolv-2.17.so 7f9ca922b000-7f9ca942b000 ---p 00016000 fd:00 50413172 /usr/lib64/libresolv-2.17.so 7f9ca942b000-7f9ca942c000 r--p 00016000 fd:00 50413172 /usr/lib64/libresolv-2.17.so 7f9ca942c000-7f9ca942d000 rw-p 00017000 fd:00 50413172 /usr/lib64/libresolv-2.17.so 7f9ca942d000-7f9ca942f000 rw-p 00000000 00:00 0 7f9ca942f000-7f9ca9432000 r-xp 00000000 fd:00 50413825 /usr/lib64/libkeyutils.so.1.5 7f9ca9432000-7f9ca9631000 ---p 00003000 fd:00 50413825 /usr/lib64/libkeyutils.so.1.5 7f9ca9631000-7f9ca9632000 r--p 00002000 fd:00 50413825 /usr/lib64/libkeyutils.so.1.5 7f9ca9632000-7f9ca9633000 rw-p 00003000 fd:00 50413825 /usr/lib64/libkeyutils.so.1.5 7f9ca9633000-7f9ca9641000 r-xp 00000000 fd:00 50413542 /usr/lib64/libkrb5support.so.0.1 7f9ca9641000-7f9ca9841000 ---p 0000e000 fd:00 50413542 /usr/lib64/libkrb5support.so.0.1 7f9ca9841000-7f9ca9842000 r--p 0000e000 fd:00 50413542 /usr/lib64/libkrb5support.so.0.1 7f9ca9842000-7f9ca9843000 rw-p 0000f000 fd:00 50413542 /usr/lib64/libkrb5support.so.0.1 7f9ca9843000-7f9ca9858000 r-xp 00000000 fd:00 50413186 /usr/lib64/libz.so.1.2.7 7f9ca9858000-7f9ca9a57000 ---p 00015000 fd:00 50413186 /usr/lib64/libz.so.1.2.7 7f9ca9a57000-7f9ca9a58000 r--p 00014000 fd:00 50413186 /usr/lib64/libz.so.1.2.7 7f9ca9a58000-7f9ca9a59000 rw-p 00015000 fd:00 50413186 /usr/lib64/libz.so.1.2.7 7f9ca9a59000-7f9ca9a5b000 r-xp 00000000 fd:00 50413150 /usr/lib64/libdl-2.17.so 7f9ca9a5b000-7f9ca9c5b000 ---p 00002000 fd:00 50413150 /usr/lib64/libdl-2.17.so 7f9ca9c5b000-7f9ca9c5c000 r--p 00002000 fd:00 50413150 /usr/lib64/libdl-2.17.so 7f9ca9c5c000-7f9ca9c5d000 rw-p 00003000 fd:00 50413150 /usr/lib64/libdl-2.17.so 7f9ca9c5d000-7f9ca9c8e000 r-xp 00000000 fd:00 50413525 /usr/lib64/libk5crypto.so.3.1 7f9ca9c8e000-7f9ca9e8d000 ---p 00031000 fd:00 50413525 /usr/lib64/libk5crypto.so.3.1 7f9ca9e8d000-7f9ca9e8f000 r--p 00030000 fd:00 50413525 /usr/lib64/libk5crypto.so.3.1 7f9ca9e8f000-7f9ca9e90000 rw-p 00032000 fd:00 50413525 /usr/lib64/libk5crypto.so.3.1 7f9ca9e90000-7f9ca9e93000 r-xp 00000000 fd:00 50413591 /usr/lib64/libcomerr.so.2.1 7f9ca9e93000-7f9caa092000 ---p 00003000 fd:00 50413591 /usr/lib64/libcomerr.so.2.1 7f9caa092000-7f9caa093000 r--p 00002000 fd:00 50413591 /usr/lib64/libcomerr.so.2.1 7f9caa093000-7f9caa094000 rw-p 00003000 fd:00 50413591 /usr/lib64/libcomerr.so.2.1 7f9caa094000-7f9caa16d000 r-xp 00000000 fd:00 50413536 /usr/lib64/libkrb5.so.3.3 7f9caa16d000-7f9caa36c000 ---p 000d9000 fd:00 50413536 /usr/lib64/libkrb5.so.3.3 7f9caa36c000-7f9caa37a000 r--p 000d8000 fd:00 50413536 /usr/lib64/libkrb5.so.3.3 7f9caa37a000-7f9caa37d000 rw-p 000e6000 fd:00 50413536 /usr/lib64/libkrb5.so.3.3 7f9caa37d000-7f9caa37f000 r-xp 00000000 fd:00 50413068 /usr/lib64/libfreebl3.so 7f9caa37f000-7f9caa57e000 ---p 00002000 fd:00 50413068 /usr/lib64/libfreebl3.so 7f9caa57e000-7f9caa57f000 r--p 00001000 fd:00 50413068 /usr/lib64/libfreebl3.so 7f9caa57f000-7f9caa580000 rw-p 00002000 fd:00 50413068 /usr/lib64/libfreebl3.so 7f9caa580000-7f9caa744000 r-xp 00000000 fd:00 50413144 /usr/lib64/libc-2.17.so 7f9caa744000-7f9caa943000 ---p 001c4000 fd:00 50413144 /usr/lib64/libc-2.17.so 7f9caa943000-7f9caa947000 r--p 001c3000 fd:00 50413144 /usr/lib64/libc-2.17.so 7f9caa947000-7f9caa949000 rw-p 001c7000 fd:00 50413144 /usr/lib64/libc-2.17.so 7f9caa949000-7f9caa94e000 rw-p 00000000 00:00 0 7f9caa94e000-7f9caa95b000 r-xp 00000000 fd:00 50341538 /usr/lib64/libpam.so.0.83.1 7f9caa95b000-7f9caab5b000 ---p 0000d000 fd:00 50341538 /usr/lib64/libpam.so.0.83.1 7f9caab5b000-7f9caab5c000 r--p 0000d000 fd:00 50341538 /usr/lib64/libpam.so.0.83.1 7f9caab5c000-7f9caab5d000 rw-p 0000e000 fd:00 50341538 /usr/lib64/libpam.so.0.83.1 7f9caab5d000-7f9caab98000 r-xp 00000000 fd:00 50348984 /usr/lib64/libipmiconsole.so.2.3.4 7f9caab98000-7f9caad97000 ---p 0003b000 fd:00 50348984 /usr/lib64/libipmiconsole.so.2.3.4 7f9caad97000-7f9caad98000 r--p 0003a000 fd:00 50348984 /usr/lib64/libipmiconsole.so.2.3.4 7f9caad98000-7f9caad99000 rw-p 0003b000 fd:00 50348984 /usr/lib64/libipmiconsole.so.2.3.4 7f9caad99000-7f9caada2000 r-xp 00000000 fd:00 50413823 /usr/lib64/libwrap.so.0.7.6 7f9caada2000-7f9caafa1000 ---p 00009000 fd:00 50413823 /usr/lib64/libwrap.so.0.7.6 7f9caafa1000-7f9caafa2000 r--p 00008000 fd:00 50413823 /usr/lib64/libwrap.so.0.7.6 7f9caafa2000-7f9caafa3000 rw-p 00009000 fd:00 50413823 /usr/lib64/libwrap.so.0.7.6 7f9caafa3000-7f9caafa4000 rw-p 00000000 00:00 0 7f9caafa4000-7f9caafee000 r-xp 00000000 fd:00 50413520 /usr/lib64/libgssapikrb5.so.2.2 7f9caafee000-7f9cab1ee000 ---p 0004a000 fd:00 50413520 /usr/lib64/libgssapikrb5.so.2.2 7f9cab1ee000-7f9cab1ef000 r--p 0004a000 fd:00 50413520 /usr/lib64/libgssapikrb5.so.2.2 7f9cab1ef000-7f9cab1f1000 rw-p 0004b000 fd:00 50413520 /usr/lib64/libgssapikrb5.so.2.2 7f9cab1f1000-7f9cab428000 r-xp 00000000 fd:00 50413549 /usr/lib64/libcrypto.so.1.0.2k 7f9cab428000-7f9cab627000 ---p 00237000 fd:00 50413549 /usr/lib64/libcrypto.so.1.0.2k 7f9cab627000-7f9cab643000 r--p 00236000 fd:00 50413549 /usr/lib64/libcrypto.so.1.0.2k 7f9cab643000-7f9cab650000 rw-p 00252000 fd:00 50413549 /usr/lib64/libcrypto.so.1.0.2k 7f9cab650000-7f9cab654000 rw-p 00000000 00:00 0 7f9cab654000-7f9cab6bb000 r-xp 00000000 fd:00 50413554 /usr/lib64/libssl.so.1.0.2k 7f9cab6bb000-7f9cab8bb000 ---p 00067000 fd:00 50413554 /usr/lib64/libssl.so.1.0.2k 7f9cab8bb000-7f9cab8bf000 r--p 00067000 fd:00 50413554 /usr/lib64/libssl.so.1.0.2k 7f9cab8bf000-7f9cab8c6000 rw-p 0006b000 fd:00 50413554 /usr/lib64/libssl.so.1.0.2k 7f9cab8c6000-7f9cab8ce000 r-xp 00000000 fd:00 50413148 /usr/lib64/libcrypt-2.17.so 7f9cab8ce000-7f9cabacd000 ---p 00008000 fd:00 50413148 /usr/lib64/libcrypt-2.17.so 7f9cabacd000-7f9cabace000 r--p 00007000 fd:00 50413148 /usr/lib64/libcrypt-2.17.so 7f9cabace000-7f9cabacf000 rw-p 00008000 fd:00 50413148 /usr/lib64/libcrypt-2.17.so 7f9cabacf000-7f9cabafd000 rw-p 00000000 00:00 0 7f9cabafd000-7f9cabaff000 r-xp 00000000 fd:00 50413178 /usr/lib64/libutil-2.17.so 7f9cabaff000-7f9cabcfe000 ---p 00002000 fd:00 50413178 /usr/lib64/libutil-2.17.so 7f9cabcfe000-7f9cabcff000 r--p 00001000 fd:00 50413178 /usr/lib64/libutil-2.17.so 7f9cabcff000-7f9cabd00000 rw-p 00002000 fd:00 50413178 /usr/lib64/libutil-2.17.so 7f9cabd00000-7f9cabd22000 r-xp 00000000 fd:00 50353406 /usr/lib64/ld-2.17.so 7f9cabe5c000-7f9cabed5000 r--s 00000000 00:14 18176394 /run/nscd/db8O03r8 (deleted) 7f9cabf0a000-7f9cabf18000 rw-p 00000000 00:00 0 7f9cabf1d000-7f9cabf1e000 rw-p 00000000 00:00 0 7f9cabf1e000-7f9cabf1f000 rw-p 00000000 00:00 0 7f9cabf1f000-7f9cabf21000 rw-p 00000000 00:00 0 7f9cabf21000-7f9cabf22000 r--p 00021000 fd:00 50353406 /usr/lib64/ld-2.17.so 7f9cabf22000-7f9cabf23000 rw-p 00022000 fd:00 50353406 /usr/lib64/ld-2.17.so 7f9cabf23000-7f9cabf24000 rw-p 00000000 00:00 0 7fff947c4000-7fff947f6000 rw-p 00000000 00:00 0 [stack] 7fff947fe000-7fff94800000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] [Mon Oct 23 15:25:51 2023] conserver (40927): [[code]ibm-z-111.rhts.eng.bos.redhat.com[/code]] exit(2) [Mon Oct 23 15:25:51 2023] conserver (40927): [[code]ibm-z-111.rhts.eng.bos.redhat.com[/code]] automatic reinitialization [Mon Oct 23 15:25:51 2023] conserver (44478): [[code]adria.lab.bos.redhat.com[/code]] console initializing [Mon Oct 23 15:25:51 2023] conserver (44478): [[code]lenovo-sd630v2-01.khw2.lab.eng.bos.redhat.com[/code]] exit(1) [Mon Oct 23 15:25:51 2023] conserver (44478): [[code]lenovo-sd630v2-01.khw2.lab.eng.bos.redhat.com[/code]] automatic reinitialization [Mon Oct 23 15:25:51 2023] conserver (43550): [[code]gsm-r6s18-01.wlan.rhts.eng.bos.redhat.com[/code]] console initializing [Mon Oct 23 15:25:51 2023] conserver (26154): child pid 54394: signal(6), restarting

agialluc commented 12 months ago

Here is an analysis on what may be happening by a developer who kindly looked at the backtrace:

======= Backtrace: ========= /lib64/libc.so.6(__fortify_fail+0x37)[0x7f9caa6987a7] /lib64/libc.so.6(+0x116922)[0x7f9caa696922] /lib64/libc.so.6(+0x118707)[0x7f9caa698707] /usr/sbin/conserver(+0x158b2)[0x56388f4368b2] /usr/sbin/conserver(+0x2588a)[0x56388f44688a] /usr/sbin/conserver(+0x1942f)[0x56388f43a42f] /usr/sbin/conserver(+0x78d8)[0x56388f4288d8] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f9caa5a2555] /usr/sbin/conserver(+0x7c58)[0x56388f428c58]

As far as I can tell, it's the number within parenthesis that should be passed to addr2line:

$ addr2line -e ./conserver.debug 0x158b2 /usr/src/debug/conserver-8.2.1/conserver/group.c:5282 (discriminator 1) $ addr2line -e ./conserver.debug 0x2588a /usr/src/debug/conserver-8.2.1/conserver/readcfg.c:5345 [jmoyer@segfault sbin]$ addr2line -e ./conserver.debug 0x1942f /usr/src/debug/conserver-8.2.1/conserver/master.c:872 $ addr2line -e ./conserver.debug 0x78d8 /usr/src/debug/conserver-8.2.1/conserver/main.c:1773

Connecting that to the code, you get the following, starting at the top of the stack (a '*' after the line number indicates the line number from addr2line):

group.c:Spawn() 5281: close(fd); 5282:* FD_CLR(fd, &rinit);

readcfg.c:ReReadCfg() 5343: Spawn(pGE, msfd); 5344: 5345:* Verbose("group #%d pid %lu on port %hu", pGE->id, (unsigned long)pGE->pid, pGE->port);

master.c:Master() 863: if (fSawHUP) { 864: fSawHUP = 0; 865: Msg("processing SIGHUP"); 866: ReopenLogfile(); 867: ReopenUnifiedlog(); 868: SignalKids(SIGHUP); 869: ReReadCfg(msfd, msfd); 870: / fix up the client descriptors since ReReadCfg() doesn't 871: see them like it can in the child processes / 872: for (pCL = pCLmall; pCL != (CONSCLIENT *)0; pCL = pCL->pCLscan) {

main.c:1773 Master();

It looks like we've overflowed the fd_set. Here's more context around the FD_CLR:

/* clean out the master client lists - they aren't useful here and just

From the select(2) man page:

NOTES An fd_set is a fixed size buffer. Executing FD_CLR() or FD_SET() with a value of fd that is negative or is equal to or larger than FD_SETSIZE will result in undefined behavior. Moreover, POSIX requires fd to be a valid file descriptor.

It is certainly possible that fd is greater than 1024. It may also be possible for FileUnopen to return -1, I'm not 100% sure on that. Enabling debugging would help verify which (if either) of those cases we stumbled upon.

Assuming this is the problem we are hitting, one fix would be to convert to poll or epoll, as suggested by the select(2) man page:

WARNING: select() can monitor only file descriptors numbers that are less than FD_SETSIZE (1024)—an unreasonably low limit for many modern applications—and this limitation will not change. All modern applica‐ tions should instead use poll(2) or epoll(7), which do not suffer this limitation.

But first, we should verify this is the problem we are hitting.

agialluc commented 12 months ago

FYI: We will be trying a modified conserver binary to try to nail this down further.

agialluc commented 11 months ago

We created a patch to try to narrow this down: ( I had to put a | char in the code block at the first char due to formatting issues)

`` diff -up conserver-8.2.1/conserver/cutil.h.orig conserver-8.2.1/conserver/cutil.h --- conserver-8.2.1/conserver/cutil.h.orig 2023-10-24 13:39:25.955849733 -0400 +++ conserver-8.2.1/conserver/cutil.h 2023-10-24 13:40:02.335721870 -0400 @@ -214,3 +214,11 @@ extern int SSLVerifyCallback(int, X509_S extern int FileSSLAccept(CONSFILE *); extern int FileCanSSLAccept(CONSFILE , fd_set , fd_set *); #endif + +#define ErrorRatelimited(fmt, ...) \ +{ \ + static int _printed_rl; \ + \ + if (_printed_rl++ % 100 == 0) \ + Error(fmt, ##__VA_ARGS__); \ +} diff -up conserver-8.2.1/conserver/group.c.orig conserver-8.2.1/conserver/group.c --- conserver-8.2.1/conserver/group.c.orig 2023-10-24 13:39:25.956849757 -0400 +++ conserver-8.2.1/conserver/group.c 2023-10-24 13:39:31.126973698 -0400 @@ -5277,6 +5277,8 @@ Spawn(GRPENT *pGE, int msfd) int fd; fd = FileUnopen(pCLmall->fd); pCLmall->fd = (CONSFILE *)0; + if (fd < 0 fd >= FD_SETSIZE) + ErrorRatelimited("Spawn(): FileUnopen() returned fd %d", fd); CONDDEBUG((1, "Spawn(): closing Master() client fd %d", fd)); close(fd); FD_CLR(fd, &rinit); diff -up conserver-8.2.1/conserver/master.c.orig conserver-8.2.1/conserver/master.c --- conserver-8.2.1/conserver/master.c.orig 2023-10-24 13:39:25.956849757 -0400 +++ conserver-8.2.1/conserver/master.c 2023-10-24 13:39:31.126973698 -0400 @@ -816,6 +816,10 @@ Master(void) strerror(errno)); return; } + + if (msfd >= FD_SETSIZE) + ErrorRatelimited("Master(): socket() returned fd (%d) which will overflow fd_set", msfd); + # if HAVE_SETSOCKOPT if (setsockopt (msfd, SOL_SOCKET, SO_REUSEADDR, (char *)&true, @@ -1029,6 +1033,8 @@ Master(void) } pCLmall = pCL;
+ if (cfd >= FD_SETSIZE)
+ ErrorRatelimited("Master(): accept()ed fd (%d) will overflow fd_set", cfd);
FD_SET(cfd, &rinit);
if (maxfd < cfd + 1)
maxfd = cfd + 1;

This time when the buffer overrun happened we see multiple lines like: egrep 'Spawn|Master' /var/log/conserver | head -20 [Wed Nov 8 14:38:41 2023] conserver (110982): ERROR: Spawn(): FileUnopen() returned fd -1 [Wed Nov 8 14:38:41 2023] conserver (110987): ERROR: Spawn(): FileUnopen() returned fd -1

Its still being looked at but it seems tied to Spawn() call in group.c

agialluc commented 11 months ago

The developer I was working with said:

I think we can just add a check for -1 and it should be fine. This code appears to be cleaning up any file descriptors that were open in the parent process.

I'll post a diff of what he provides as the possible solution.

JeffMoyer commented 11 months ago

Is there a reason close() is used instead of FileClose() in Spawn()? Specifically, here:

    /* clean out the master client lists - they aren't useful here and just
     * cause extra file descriptors and memory allocation to lie around,
     * not a very good thing!
     */
    while (pCLmall != (CONSCLIENT *)0) {
    CONSCLIENT *pCL;
    if (pCLmall->fd != (CONSFILE *)0) {
        int fd;
        fd = FileUnopen(pCLmall->fd);
        pCLmall->fd = (CONSFILE *)0;
        CONDDEBUG((1, "Spawn(): closing Master() client fd %d", fd));
        close(fd);
        FD_CLR(fd, &rinit);
        FD_CLR(fd, &winit);
    }
    pCL = pCLmall->pCLscan;
    DestroyClient(pCLmall);
    pCLmall = pCL;
    }

It seems to me, something like this would work better, as it wouldn't leak ssl sockets (and would fix the buffer overflow we've been seeing):

--- conserver/group.c.orig  2023-11-08 16:08:37.684566585 -0500
+++ conserver/group.c   2023-11-08 16:47:14.026563054 -0500
@@ -5275,12 +5275,16 @@ Spawn(GRPENT *pGE, int msfd)
    CONSCLIENT *pCL;
    if (pCLmall->fd != (CONSFILE *)0) {
        int fd;
-       fd = FileUnopen(pCLmall->fd);
-       pCLmall->fd = (CONSFILE *)0;
-       CONDDEBUG((1, "Spawn(): closing Master() client fd %d", fd));
-       close(fd);
-       FD_CLR(fd, &rinit);
-       FD_CLR(fd, &winit);
+       fd = FileFDNum(pCLmall->fd);
+       if (fd >= 0) {
+           CONDDEBUG((1, "Spawn(): closing Master() client fd %d", fd));
+           FileClose(&pCLmall->fd);
+           FD_CLR(fd, &rinit);
+           FD_CLR(fd, &winit);
+       } else {
+           FileUnopen(pCLmall->fd);
+           pCLmall->fd = (CONSFILE *)0;
+       }
    }
    pCL = pCLmall->pCLscan;
    DestroyClient(pCLmall);

What do you think?

agialluc commented 11 months ago

"Is there a reason close() is used instead of FileClose() in Spawn()? Specifically, here:"

I couldn't answer this. What little programming I do is just 'on the side', perhaps @bstansell can address this.

JeffMoyer commented 11 months ago

I answered my own question. FileClose would shutdown the socket, which isn't what we want. I created a pull request for this issue here: https://github.com/bstansell/conserver/pull/95