Closed holiman closed 1 year ago
It can be reproed within a few minutes by:
yes /home/martin/workspace/goevmlab/evms/testdata/cases/00000006-naivefuzz-0.json | /home/martin/workspace/besu-vm state-test --nomemory
When it is 'bad', the mem consumption keeps rising
$ while [ true ]; do ps -o rss= `pidof java` | awk '{printf "%.0f MB\n", $1 / 1024}'; sleep 1; done
32 MB
137 MB
166 MB
175 MB
175 MB
220 MB
247 MB
257 MB
272 MB
286 MB
303 MB
320 MB
338 MB
351 MB
369 MB
551 MB
565 MB
583 MB
604 MB
623 MB
643 MB
671 MB
689 MB
704 MB
719 MB
720 MB
734 MB
748 MB
766 MB
782 MB
798 MB
815 MB
830 MB
841 MB
856 MB
870 MB
886 MB
899 MB
913 MB
928 MB
942 MB
961 MB
979 MB
Whereas the 'good' becomes stable somewhere below 600 MB
:
$ while [ true ]; do ps -o rss= `pidof java` | awk '{printf "%.0f MB\n", $1 / 1024}'; sleep 1; done
331 MB
415 MB
481 MB
563 MB
560 MB
561 MB
563 MB
576 MB
578 MB
578 MB
578 MB
578 MB
583 MB
583 MB
583 MB
583 MB
584 MB
577 MB
577 MB
577 MB
577 MB
577 MB
579 MB
579 MB
579 MB
579 MB
581 MB
575 MB
575 MB
575 MB
575 MB
575 MB
575 MB
575 MB
575 MB
575 MB
So, according to git bisect
, the culprit is...
f8a0f874057a87e0f7ef864c9f289b39d370c0ae is the first bad commit
commit f8a0f874057a87e0f7ef864c9f289b39d370c0ae
Author: Justin Florentine <justin+github@florentine.us>
Date: Mon Jan 23 09:01:39 2023 -0500
KZG point eval precompile (#4860)
* KZG implementation
Signed-off-by: Justin Florentine <justin+github@florentine.us>
So https://github.com/hyperledger/besu/commit/f8a0f874057a87e0f7ef864c9f289b39d370c0ae ..
How, or why, I have no idea. I even extra-checked that commit and the one before, and it is indeed that commit which leaks memory.
@jflo wanna take a look at this one?
It is loading this resource every time a test is executed: https://github.com/hyperledger/besu/commit/f8a0f874057a87e0f7ef864c9f289b39d370c0ae#diff-e614ff91ff732e88484a9e612a16f3bb91fd9889b3e06f0223cc33399e158fa9R52
yeah, @shemnon ran into this last week and has a fix for it pending: https://github.com/hyperledger/besu/pull/5030/commits/1aa8fb28f56a236404fbe661ce809adc4769f513. Will keep this issue open for retest after that PR hits main.
PR #5030 merged, so I assume this is fixed. I'll test later this evening, closing in the meanwhile
If I use besu-vm in batch-mode for a couple of hours, or ~15-20K testcases, it eventually runs out of memory.
Some chosen portions of the crash log:
THis happens with commit
56ae1fcf5b59ccf1f4a0855133110e53354fec38
-- previously it worked fine, in december, so I think it's some memory leak that's been created between mid december and now.I'll try to repro it better, with a bit of luck I can just feed the same testcase over and over in a loop and trigger it, and do a git bisect.
Invocation is
and feeding filenames via
stdin
.