holiman
commented
3 weeks ago Tried to repro it
root@a04949b121fe:/# yes "/fuzztmp/10573375-mixed-8.json" | /neth/nethtest --trace -m -x 2>/dev/null | grep "time" | xargs -L1 bash checkmem.sh nethtest
PID 456170 proc nethtest | Rss: 728460 kB | Pss: 1417619 kB | Shared Clean 7348 kB | Shared Dirty 40 kB | Private 732420 kB
PID 456170 proc nethtest | Rss: 751540 kB | Pss: 1464707 kB | Shared Clean 7348 kB | Shared Dirty 48 kB | Private 754788 kB
PID 456170 proc nethtest | Rss: 775672 kB | Pss: 1512089 kB | Shared Clean 7292 kB | Shared Dirty 32 kB | Private 778992 kB
PID 456170 proc nethtest | Rss: 799328 kB | Pss: 1559395 kB | Shared Clean 7360 kB | Shared Dirty 32 kB | Private 802492 kB
PID 456170 proc nethtest | Rss: 823460 kB | Pss: 1608071 kB | Shared Clean 7364 kB | Shared Dirty 32 kB | Private 826936 kB
PID 456170 proc nethtest | Rss: 847404 kB | Pss: 1658023 kB | Shared Clean 7356 kB | Shared Dirty 32 kB | Private 853024 kB
PID 456170 proc nethtest | Rss: 874820 kB | Pss: 1713117 kB | Shared Clean 7412 kB | Shared Dirty 28 kB | Private 880676 kBAfter a couple of minutes:
PID 456170 proc nethtest | Rss: 4451112 kB | Pss: 8861466 kB | Shared Clean 7420 kB | Shared Dirty 28 kB | Private 4443600 kB
PID 456170 proc nethtest | Rss: 4451112 kB | Pss: 8861496 kB | Shared Clean 7436 kB | Shared Dirty 28 kB | Private 4443584 kBA few minutes later it has gone down a bit again:
PID 456170 proc nethtest | Rss: 4329912 kB | Pss: 8619118 kB | Shared Clean 7540 kB | Shared Dirty 32 kB | Private 4322404 kB
PID 456170 proc nethtest | Rss: 4329912 kB | Pss: 8619137 kB | Shared Clean 7496 kB | Shared Dirty 32 kB | Private 4322384 kBSeems to stabilize around here
PID 456170 proc nethtest | Rss: 4351724 kB | Pss: 8662759 kB | Shared Clean 7492 kB | Shared Dirty 36 kB | Private 4344196 kBSo yeah, no obvious easily repdocucible leak.
memchecker-script:
#!/bin/bash
if [[ -z "$2" ]]; then
exit 0
fi
for pid in $(ps -ef | awk '{print $2}'); do
if [[ -z "$pid" ]]; then
continue
fi
if [[ $pid == "PID" ]]; then
continue
fi
a=$(ps -p $pid -o comm=)
if [[ $a != "nethtest" ]]; then
continue
fi
if [ -f /proc/$pid/smaps ]; then
rss=$(awk 'BEGIN {i=0} /^Rss/ {i = i + $2} END {print i}' /proc/$pid/smaps)
pss=$(awk 'BEGIN {i=0} /^Pss/ {i = i + $2 + 0.5} END {print i}' /proc/$pid/smaps)
sc=$(awk 'BEGIN {i=0} /^Shared_Clean/ {i = i + $2} END {print i}' /proc/$pid/smaps)
sd=$(awk 'BEGIN {i=0} /^Shared_Dirty/ {i = i + $2} END {print i}' /proc/$pid/smaps)
pc=$(awk 'BEGIN {i=0} /^Private_Clean/ {i = i + $2} END {print i}' /proc/$pid/smaps)
pd=$(awk 'BEGIN {i=0} /^Private_Dirty/ {i = i + $2} END {print i}' /proc/$pid/smaps)
echo "PID $pid proc $a | Rss: $rss kB | Pss: $pss kB | Shared Clean $sc kB | Shared Dirty $sd kB | Private $(($pd + $pc)) kB"
fi
done
LukaszRozmej
Ran into this crash with the fuzzer:
Somewhere after
42Minvocations, the nethermind instance went OOM (the testcase was nothing special). Possibly nethermind in batch mode has some memory leak.TODO investigate.
file which potentially triggers the leak,
10573375-mixed-8.json: