*** Error in `ntttcp': realloc(): invalid pointer: 0x00007ffd842ffb80 ***

[closertotheheart]

test script running between mellanox X5 and X6-DX on a supermicro Xeon Gold SP dual socket and intel R1800 Xeon Gold SP 2nd Gen dual socket.

recently upgraded to HEAD as of 1/21/2021. Don't recall seeing this issue before (last clone was likely >= 2 weeks ago)--but this script has changed over time.

X5 is running as Gen3x8, X6DX as Gen3x16.

Basically just a stress test of link bounce & link saturation testing out some new physical plant between the two cards.

compiled with:

CFLAGS="-march=native -flto -Wl,-flto -O2" AR="gcc-ar" NM="gcc-nm" RANLIB="gcc-ranlib"

using devtoolset-9 on RHEL 7.7

WARMUP=2 COOLDOWN=2 DURATION=60 ARGS="-t $DURATION -W $WARMUP -C $COOLDOWN" while true; do
ifconfig enp24s0f0 down sleep 1 ifconfig enp24s0f0 up sleep 1 ifconfig enp24s0f0 | grep UP,BROADCAST,RUNNING,MULTICAST || exit 1 TIMESTAMP=$(date +%Y-%m-%dT%H:%M:%S%z) ntttcp $ARGS -O log.$TIMESTAMP & ssh user>@<host ntttcp -s 192.168.100.1 $ARGS -O ntttcp.enp24s0f0.$TIMESTAMP.log -x ntttcp.enp24s0f0.$TIMESTAMP.xml wait TIMESTAMP=$(date +%Y-%m-%dT%H:%M:%S%z) ntttcp $ARGS -O log.$TIMESTAMP & ssh user>@<host ntttcp -s 192.168.101.1 $ARGS -O ntttcp.enp24s0f0.$TIMESTAMP.log -x ntttcp.enp24s0f0.$TIMESTAMP.xml wait ifconfig enp24s0f1 down sleep 1 ifconfig enp24s0f1 up sleep 1 ifconfig enp24s0f1 | grep UP,BROADCAST,RUNNING,MULTICAST || exit 1 done

---

======= Backtrace: ========= /lib64/libc.so.6(+0x7f3e4)[0x7fa76f86e3e4] /lib64/libc.so.6(realloc+0x389)[0x7fa76f874f39] /lib64/libc.so.6(getdelim+0x10b)[0x7fa76f85e96b] ntttcp[0x406d9e] ntttcp[0x409e40] ntttcp[0x4039f7] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7fa76f811555] ntttcp[0x4043bd] ======= Memory map: ======== 00400000-00402000 r--p 00000000 fd:00 68462831 /usr/local/bin/ntttcp 00402000-0040b000 r-xp 00002000 fd:00 68462831 /usr/local/bin/ntttcp 0040b000-00410000 r--p 0000b000 fd:00 68462831 /usr/local/bin/ntttcp 00410000-00411000 r--p 0000f000 fd:00 68462831 /usr/local/bin/ntttcp 00411000-00412000 rw-p 00010000 fd:00 68462831 /usr/local/bin/ntttcp 0105d000-0107e000 rw-p 00000000 00:00 0 [heap] 7fa668000000-7fa668021000 rw-p 00000000 00:00 0 7fa668021000-7fa66c000000 ---p 00000000 00:00 0 7fa670000000-7fa670021000 rw-p 00000000 00:00 0 7fa670021000-7fa674000000 ---p 00000000 00:00 0 7fa674000000-7fa674021000 rw-p 00000000 00:00 0 7fa674021000-7fa678000000 ---p 00000000 00:00 0 7fa678000000-7fa678021000 rw-p 00000000 00:00 0 7fa678021000-7fa67c000000 ---p 00000000 00:00 0 7fa67c000000-7fa67c021000 rw-p 00000000 00:00 0 7fa67c021000-7fa680000000 ---p 00000000 00:00 0 7fa680000000-7fa680021000 rw-p 00000000 00:00 0 7fa680021000-7fa684000000 ---p 00000000 00:00 0 7fa684000000-7fa684021000 rw-p 00000000 00:00 0 7fa684021000-7fa688000000 ---p 00000000 00:00 0 7fa688000000-7fa688021000 rw-p 00000000 00:00 0 7fa688021000-7fa68c000000 ---p 00000000 00:00 0 7fa68c000000-7fa68c021000 rw-p 00000000 00:00 0 7fa68c021000-7fa690000000 ---p 00000000 00:00 0 7fa690000000-7fa690021000 rw-p 00000000 00:00 0 7fa690021000-7fa694000000 ---p 00000000 00:00 0 7fa694000000-7fa694021000 rw-p 00000000 00:00 0 7fa694021000-7fa698000000 ---p 00000000 00:00 0 7fa698000000-7fa698021000 rw-p 00000000 00:00 0 7fa698021000-7fa69c000000 ---p 00000000 00:00 0 7fa69c000000-7fa69c021000 rw-p 00000000 00:00 0 7fa69c021000-7fa6a0000000 ---p 00000000 00:00 0 7fa6a0000000-7fa6a0021000 rw-p 00000000 00:00 0 7fa6a0021000-7fa6a4000000 ---p 00000000 00:00 0 7fa6a4000000-7fa6a4021000 rw-p 00000000 00:00 0 7fa6a4021000-7fa6a8000000 ---p 00000000 00:00 0 7fa6a8000000-7fa6a8021000 rw-p 00000000 00:00 0 7fa6a8021000-7fa6ac000000 ---p 00000000 00:00 0 7fa6ac000000-7fa6ac021000 rw-p 00000000 00:00 0 7fa6ac021000-7fa6b0000000 ---p 00000000 00:00 0 7fa6b0000000-7fa6b0021000 rw-p 00000000 00:00 0 7fa6b0021000-7fa6b4000000 ---p 00000000 00:00 0 7fa6b4000000-7fa6b4021000 rw-p 00000000 00:00 0 7fa6b4021000-7fa6b8000000 ---p 00000000 00:00 0 7fa6b8000000-7fa6b8021000 rw-p 00000000 00:00 0 7fa6b8021000-7fa6bc000000 ---p 00000000 00:00 0 7fa6bc000000-7fa6bc021000 rw-p 00000000 00:00 0 7fa6bc021000-7fa6c0000000 ---p 00000000 00:00 0 7fa6c0000000-7fa6c0021000 rw-p 00000000 00:00 0 7fa6c0021000-7fa6c4000000 ---p 00000000 00:00 0 7fa6c4000000-7fa6c4021000 rw-p 00000000 00:00 0 7fa6c4021000-7fa6c8000000 ---p 00000000 00:00 0 7fa6c8000000-7fa6c8021000 rw-p 00000000 00:00 0 7fa6c8021000-7fa6cc000000 ---p 00000000 00:00 0 7fa6cc000000-7fa6cc021000 rw-p 00000000 00:00 0 7fa6cc021000-7fa6d0000000 ---p 00000000 00:00 0 7fa6d0000000-7fa6d0021000 rw-p 00000000 00:00 0 7fa6d0021000-7fa6d4000000 ---p 00000000 00:00 0 7fa6d4000000-7fa6d4021000 rw-p 00000000 00:00 0 7fa6d4021000-7fa6d8000000 ---p 00000000 00:00 0 7fa6d8000000-7fa6d8021000 rw-p 00000000 00:00 0 7fa6d8021000-7fa6dc000000 ---p 00000000 00:00 0 7fa6dc000000-7fa6dc021000 rw-p 00000000 00:00 0 7fa6dc021000-7fa6e0000000 ---p 00000000 00:00 0 7fa6e0000000-7fa6e0021000 rw-p 00000000 00:00 0 7fa6e0021000-7fa6e4000000 ---p 00000000 00:00 0 7fa6e4000000-7fa6e4021000 rw-p 00000000 00:00 0 7fa6e4021000-7fa6e8000000 ---p 00000000 00:00 0 7fa6e8000000-7fa6e8021000 rw-p 00000000 00:00 0 7fa6e8021000-7fa6ec000000 ---p 00000000 00:00 0 7fa6ec000000-7fa6ec021000 rw-p 00000000 00:00 0 7fa6ec021000-7fa6f0000000 ---p 00000000 00:00 0 7fa6f0000000-7fa6f0021000 rw-p 00000000 00:00 0 7fa6f0021000-7fa6f4000000 ---p 00000000 00:00 0 7fa6f4000000-7fa6f4021000 rw-p 00000000 00:00 0 7fa6f4021000-7fa6f8000000 ---p 00000000 00:00 0 7fa6f8000000-7fa6f8021000 rw-p 00000000 00:00 0 7fa6f8021000-7fa6fc000000 ---p 00000000 00:00 0 7fa6fc000000-7fa6fc021000 rw-p 00000000 00:00 0 7fa6fc021000-7fa700000000 ---p 00000000 00:00 0 7fa700000000-7fa700021000 rw-p 00000000 00:00 0 7fa700021000-7fa704000000 ---p 00000000 00:00 0 7fa704000000-7fa704021000 rw-p 00000000 00:00 0 7fa704021000-7fa708000000 ---p 00000000 00:00 0 7fa708000000-7fa708021000 rw-p 00000000 00:00 0 7fa708021000-7fa70c000000 ---p 00000000 00:00 0 7fa70c000000-7fa70c021000 rw-p 00000000 00:00 0 7fa70c021000-7fa710000000 ---p 00000000 00:00 0 7fa710000000-7fa710021000 rw-p 00000000 00:00 0 7fa710021000-7fa714000000 ---p 00000000 00:00 0 7fa714000000-7fa714021000 rw-p 00000000 00:00 0 7fa714021000-7fa718000000 ---p 00000000 00:00 0 7fa718000000-7fa718021000 rw-p 00000000 00:00 0 7fa718021000-7fa71c000000 ---p 00000000 00:00 0 7fa71c000000-7fa71c021000 rw-p 00000000 00:00 0 7fa71c021000-7fa720000000 ---p 00000000 00:00 0 7fa720000000-7fa720021000 rw-p 00000000 00:00 0 7fa720021000-7fa724000000 ---p 00000000 00:00 0 7fa724000000-7fa724021000 rw-p 00000000 00:00 0 7fa724021000-7fa728000000 ---p 00000000 00:00 0 7fa728000000-7fa728021000 rw-p 00000000 00:00 0 7fa728021000-7fa72c000000 ---p 00000000 00:00 0 7fa72c000000-7fa72c021000 rw-p 00000000 00:00 0 7fa72c021000-7fa730000000 ---p 00000000 00:00 0 7fa730000000-7fa730021000 rw-p 00000000 00:00 0 7fa730021000-7fa734000000 ---p 00000000 00:00 0 7fa734000000-7fa734021000 rw-p 00000000 00:00 0 7fa734021000-7fa738000000 ---p 00000000 00:00 0 7fa738000000-7fa738021000 rw-p 00000000 00:00 0 7fa738021000-7fa73c000000 ---p 00000000 00:00 0 7fa73c000000-7fa73c021000 rw-p 00000000 00:00 0 7fa73c021000-7fa740000000 ---p 00000000 00:00 0 7fa740000000-7fa740021000 rw-p 00000000 00:00 0 7fa740021000-7fa744000000 ---p 00000000 00:00 0 7fa744000000-7fa744021000 rw-p 00000000 00:00 0 7fa744021000-7fa748000000 ---p 00000000 00:00 0 7fa748000000-7fa748021000 rw-p 00000000 00:00 0 7fa748021000-7fa74c000000 ---p 00000000 00:00 0 7fa74c000000-7fa74c021000 rw-p 00000000 00:00 0 7fa74c021000-7fa750000000 ---p 00000000 00:00 0 7fa750000000-7fa750021000 rw-p 00000000 00:00 0 7fa750021000-7fa754000000 ---p 00000000 00:00 0 7fa754000000-7fa754021000 rw-p 00000000 00:00 0 7fa754021000-7fa758000000 ---p 00000000 00:00 0 7fa758000000-7fa758021000 rw-p 00000000 00:00 0 7fa758021000-7fa75c000000 ---p 00000000 00:00 0 7fa75c000000-7fa75c021000 rw-p 00000000 00:00 0 7fa75c021000-7fa760000000 ---p 00000000 00:00 0 7fa760000000-7fa760021000 rw-p 00000000 00:00 0 7fa760021000-7fa764000000 ---p 00000000 00:00 0 7fa764000000-7fa764021000 rw-p 00000000 00:00 0 7fa764021000-7fa768000000 ---p 00000000 00:00 0 7fa768000000-7fa768021000 rw-p 00000000 00:00 0 7fa768021000-7fa76c000000 ---p 00000000 00:00 0 7fa76f36f000-7fa76f384000 r-xp 00000000 fd:00 33753820 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7fa76f384000-7fa76f583000 ---p 00015000 fd:00 33753820 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7fa76f583000-7fa76f584000 r--p 00014000 fd:00 33753820 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7fa76f584000-7fa76f585000 rw-p 00015000 fd:00 33753820 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7fa76f59c000-7fa76f59d000 ---p 00000000 00:00 0 7fa76f59d000-7fa76f5ad000 rw-p 00000000 00:00 0 7fa76f5ad000-7fa76f5ae000 ---p 00000000 00:00 0 7fa76f5ae000-7fa76f5be000 rw-p 00000000 00:00 0 7fa76f5be000-7fa76f5bf000 ---p 00000000 00:00 0 7fa76f5bf000-7fa76f5cf000 rw-p 00000000 00:00 0 7fa76f5cf000-7fa76f5d0000 ---p 00000000 00:00 0 7fa76f5d0000-7fa76f5e0000 rw-p 00000000 00:00 0 7fa76f5e0000-7fa76f5e1000 ---p 00000000 00:00 0 7fa76f5e1000-7fa76f5f1000 rw-p 00000000 00:00 0 7fa76f5f1000-7fa76f5f2000 ---p 00000000 00:00 0 7fa76f5f2000-7fa76f602000 rw-p 00000000 00:00 0 7fa76f602000-7fa76f603000 ---p 00000000 00:00 0 7fa76f603000-7fa76f613000 rw-p 00000000 00:00 0 7fa76f613000-7fa76f614000 ---p 00000000 00:00 0 7fa76f614000-7fa76f624000 rw-p 00000000 00:00 0 7fa76f624000-7fa76f625000 ---p 00000000 00:00 0 7fa76f625000-7fa76f635000 rw-p 00000000 00:00 0 7fa76f635000-7fa76f636000 ---p 00000000 00:00 0 7fa76f636000-7fa76f646000 rw-p 00000000 00:00 0 7fa76f646000-7fa76f647000 ---p 00000000 00:00 0 7fa76f647000-7fa76f657000 rw-p 00000000 00:00 0 7fa76f657000-7fa76f658000 ---p 00000000 00:00 0 7fa76f658000-7fa76f668000 rw-p 00000000 00:00 0 7fa76f668000-7fa76f669000 ---p 00000000 00:00 0 7fa76f669000-7fa76f679000 rw-p 00000000 00:00 0 7fa76f679000-7fa76f67a000 ---p 00000000 00:00 0 7fa76f67a000-7fa76f68a000 rw-p 00000000 00:00 0 7fa76f68a000-7fa76f68b000 ---p 00000000 00:00 0 7fa76f68b000-7fa76f69b000 rw-p 00000000 00:00 0 7fa76f69b000-7fa76f69c000 ---p 00000000 00:00 0 7fa76f69c000-7fa76f6ac000 rw-p 00000000 00:00 0 7fa76f6ac000-7fa76f6ad000 ---p 00000000 00:00 0 7fa76f6ad000-7fa76f6bd000 rw-p 00000000 00:00 0 7fa76f6bd000-7fa76f6be000 ---p 00000000 00:00 0 7fa76f6be000-7fa76f6ce000 rw-p 00000000 00:00 0 7fa76f6ce000-7fa76f6cf000 ---p 00000000 00:00 0 7fa76f6cf000-7fa76f6df000 rw-p 00000000 00:00 0 7fa76f6df000-7fa76f6e0000 ---p 00000000 00:00 0 7fa76f6e0000-7fa76f6f0000 rw-p 00000000 00:00 0 7fa76f6f0000-7fa76f6f1000 ---p 00000000 00:00 0 7fa76f6f1000-7fa76f701000 rw-p 00000000 00:00 0 7fa76f701000-7fa76f702000 ---p 00000000 00:00 0 7fa76f702000-7fa76f712000 rw-p 00000000 00:00 0 7fa76f712000-7fa76f713000 ---p 00000000 00:00 0 7fa76f713000-7fa76f723000 rw-p 00000000 00:00 0 7fa76f723000-7fa76f724000 ---p 00000000 00:00 0 7fa76f724000-7fa76f734000 rw-p 00000000 00:00 0 7fa76f734000-7fa76f735000 ---p 00000000 00:00 0 7fa76f735000-7fa76f745000 rw-p 00000000 00:00 0 7fa76f745000-7fa76f746000 ---p 00000000 00:00 0 7fa76f746000-7fa76f756000 rw-p 00000000 00:00 0 7fa76f756000-7fa76f757000 ---p 00000000 00:00 0 7fa76f757000-7fa76f767000 rw-p 00000000 00:00 0 7fa76f767000-7fa76f768000 ---p 00000000 00:00 0 7fa76f768000-7fa76f778000 rw-p 00000000 00:00 0 7fa76f778000-7fa76f779000 ---p 00000000 00:00 0 7fa76f779000-7fa76f789000 rw-p 00000000 00:00 0 7fa76f789000-7fa76f78a000 ---p 00000000 00:00 0 7fa76f78a000-7fa76f79a000 rw-p 00000000 00:00 0 7fa76f79a000-7fa76f79b000 ---p 00000000 00:00 0 7fa76f79b000-7fa76f7ab000 rw-p 00000000 00:00 0 7fa76f7ab000-7fa76f7ac000 ---p 00000000 00:00 0 7fa76f7ac000-7fa76f7bc000 rw-p 00000000 00:00 0 7fa76f7bc000-7fa76f7bd000 ---p 00000000 00:00 0 7fa76f7bd000-7fa76f7cd000 rw-p 00000000 00:00 0 7fa76f7cd000-7fa76f7ce000 ---p 00000000 00:00 0 7fa76f7ce000-7fa76f7de000 rw-p 00000000 00:00 0 7fa76f7de000-7fa76f7df000 ---p 00000000 00:00 0 7fa76f7df000-7fa76f7ef000 rw-p 00000000 00:00 0 7fa76f7ef000-7fa76f9b3000 r-xp 00000000 fd:00 33796604 /usr/lib64/libc-2.17.so 7fa76f9b3000-7fa76fbb2000 ---p 001c4000 fd:00 33796604 /usr/lib64/libc-2.17.so 7fa76fbb2000-7fa76fbb6000 r--p 001c3000 fd:00 33796604 /usr/lib64/libc-2.17.so 7fa76fbb6000-7fa76fbb8000 rw-p 001c7000 fd:00 33796604 /usr/lib64/libc-2.17.so 7fa76fbb8000-7fa76fbbd000 rw-p 00000000 00:00 0 7fa76fbbd000-7fa76fbd4000 r-xp 00000000 fd:00 33838607 /usr/lib64/libpthread-2.17.so 7fa76fbd4000-7fa76fdd3000 ---p 00017000 fd:00 33838607 /usr/lib64/libpthread-2.17.so 7fa76fdd3000-7fa76fdd4000 r--p 00016000 fd:00 33838607 /usr/lib64/libpthread-2.17.so 7fa76fdd4000-7fa76fdd5000 rw-p 00017000 fd:00 33838607 /usr/lib64/libpthread-2.17.so 7fa76fdd5000-7fa76fdd9000 rw-p 00000000 00:00 0 7fa76fdd9000-7fa76fdfb000 r-xp 00000000 fd:00 33796588 /usr/lib64/ld-2.17.so 7fa76fe03000-7fa76fe04000 ---p 00000000 00:00 0 7fa76fe04000-7fa76fe14000 rw-p 00000000 00:00 0 7fa76fe14000-7fa76fe15000 ---p 00000000 00:00 0 7fa76fe15000-7fa76fe25000 rw-p 00000000 00:00 0 7fa76fe25000-7fa76fe26000 ---p 00000000 00:00 0 7fa76fe26000-7fa76fe36000 rw-p 00000000 00:00 0 7fa76fe36000-7fa76fe37000 ---p 00000000 00:00 0 7fa76fe37000-7fa76fe47000 rw-p 00000000 00:00 0 7fa76fe47000-7fa76fe48000 ---p 00000000 00:00 0 7fa76fe48000-7fa76fe58000 rw-p 00000000 00:00 0 7fa76fe58000-7fa76fe59000 ---p 00000000 00:00 0 7fa76fe59000-7fa76fe69000 rw-p 00000000 00:00 0 7fa76fe69000-7fa76fe6a000 ---p 00000000 00:00 0 7fa76fe6a000-7fa76fe7a000 rw-p 00000000 00:00 0 7fa76fe7a000-7fa76fe7b000 ---p 00000000 00:00 0 7fa76fe7b000-7fa76fe8b000 rw-p 00000000 00:00 0 7fa76fe8b000-7fa76fe8c000 ---p 00000000 00:00 0 7fa76fe8c000-7fa76fe9c000 rw-p 00000000 00:00 0 7fa76fe9c000-7fa76fe9d000 ---p 00000000 00:00 0 7fa76fe9d000-7fa76fead000 rw-p 00000000 00:00 0 7fa76fead000-7fa76feae000 ---p 00000000 00:00 0 7fa76feae000-7fa76febe000 rw-p 00000000 00:00 0 7fa76febe000-7fa76febf000 ---p 00000000 00:00 0 7fa76febf000-7fa76fecf000 rw-p 00000000 00:00 0 7fa76fecf000-7fa76fed0000 ---p 00000000 00:00 0 7fa76fed0000-7fa76fee0000 rw-p 00000000 00:00 0 7fa76fee0000-7fa76fee1000 ---p 00000000 00:00 0 7fa76fee1000-7fa76fef1000 rw-p 00000000 00:00 0 7fa76fef1000-7fa76fef2000 ---p 00000000 00:00 0 7fa76fef2000-7fa76ff02000 rw-p 00000000 00:00 0 7fa76ff02000-7fa76ff03000 ---p 00000000 00:00 0 7fa76ff03000-7fa76ff13000 rw-p 00000000 00:00 0 7fa76ff13000-7fa76ff14000 ---p 00000000 00--------------------------------------------------------- :00 0 7fa76ff14000-7fa76ff24000 rw-p 00000000 00:00 0 7fa76ff24000-7fa76ff25000 ---p 00000000 00:00 0 7fa76ff25000-7fa76ff35000 rw-p 00000000 00:00 0 7fa76ff35000-7fa76ff36000 ---p 00000000 00:00 0 7fa76ff36000-7fa76ff46000 rw-p 00000000 00:00 0 7fa76ff46000-7fa76ff47000 ---p 00000000 00:00 0 7fa76ff47000-7fa76ff57000 rw-p 00000000 00:00 0 7fa76ff57000-7fa76ff58000 ---p 00000000 00:00 0 7fa76ff58000-7fa76ff68000 rw-p 00000000 00:00 0 7fa76ff68000-7fa76ff69000 ---p 00000000 00:00 0 7fa76ff69000-7fa76ff79000 rw-p 00000000 00:00 0 7fa76ff79000-7fa76ff7a000 ---p 00000000 00:00 0 7fa76ff7a000-7fa76ff8a000 rw-p 00000000 00:00 0 7fa76ff8a000-7fa76ff8b000 ---p 00000000 00:00 0 7fa76ff8b000-7fa76ff9b000 rw-p 00000000 00:00 0 7fa76ff9b000-7fa76ff9c000 ---p 00000000 00:00 0 7fa76ff9c000-7fa76ffac000 rw-p 00000000 00:00 0 7fa76ffac000-7fa76ffad000 ---p 00000000 00:00 0 7fa76ffad000-7fa76ffbd000 rw-p 00000000 00:00 0 7fa76ffbd000-7fa76ffbe000 ---p 00000000 00:00 0 7fa76ffbe000-7fa76ffce000 rw-p 00000000 00:00 0 7fa76ffce000-7fa76ffcf000 ---p 00000000 00:00 0 7fa76ffcf000-7fa76ffe2000 rw-p 00000000 00:00 0 7fa76ffe4000-7fa76ffe7000 rw-p 00000000 00:00 0 7fa76ffe7000-7fa76ffe8000 ---p 00000000 00:00 0 7fa76ffe8000-7fa76fffa000 rw-p 00000000 00:00 0 7fa76fffa000-7fa76fffb000 r--p 00021000 fd:00 33796588 /usr/lib64/ld-2.17.so 7fa76fffb000-7fa76fffc000 rw-p 00022000 fd:00 33796588 /usr/lib64/ld-2.17.so 7fa76fffc000-7fa76fffd000 rw-p 00000000 00:00 0 7fff70000000-7fff706d8000 rw-p 00000000 00:00 0 [stack] 7fff707d4000-7fff707d6000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]

local system is

lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 48 On-line CPU(s) list: 0-47 Thread(s) per core: 2 Core(s) per socket: 12 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Gold 6146 CPU @ 3.20GHz Stepping: 4 CPU MHz: 3781.835 CPU max MHz: 4200.0000 CPU min MHz: 1200.0000 BogoMIPS: 6400.00 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 1024K L3 cache: 25344K NUMA node0 CPU(s): 0-11,24-35 NUMA node1 CPU(s): 12-23,36-47 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 invpcid_single intel_ppin intel_pt ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke md_clear spec_ctrl intel_stibp flush_l1d

remote system is

lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 80 On-line CPU(s) list: 0-79 Thread(s) per core: 2 Core(s) per socket: 20 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz Stepping: 7 CPU MHz: 2501.000 CPU max MHz: 2501.0000 CPU min MHz: 1000.0000 BogoMIPS: 5000.00 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 1024K L3 cache: 28160K NUMA node0 CPU(s): 0-19,40-59 NUMA node1 CPU(s): 20-39,60-79 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 invpcid_single intel_pt ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke avx512_vnni md_clear spec_ctrl intel_stibp flush_l1d arch_capabilities

[closertotheheart]

backed off one commit to 213371f3b0f1b134a80c0157677d8c1cf640bbf1 & am rerunning overnight.

[closertotheheart]

happens on https://github.com/microsoft/ntttcp-for-linux/commit/213371f3b0f1b134a80c0157677d8c1cf640bbf1 and on tag v1.4.0.

Also happens without -O flag (had to delete that for v1.4.0).

Now trying without the -x flag on v1.4.0. None of my earlier usage included -x; want to rule that out as contributor.

[closertotheheart]

failure appears to depend entirely on the presence of the -x flag.