Apparently when CPU cores are all busy with threads the gc cycles are planned but not executed, so the memory occupation showed in pprof is low, but the virtual memory pages aren't marked yet as MADV_FREE_REUSABLE (osx) until the cycle happens and the OS can't reclaim them.
Reducing CPU consumption allowed the cycles to happen more frequently:
Before (nuclei)
$ /usr/bin/time -a -p -h go run . -mhe 99999999999999999 -itags high,critical -etags dns -jsc 110 -rl 0 -l target.txt -duc -rl 0
...
[INF] Targets loaded for current scan: 10000
[INF] Templates clustered: 1270 (Reduced 12380000 Requests)
[INF] Using Interactsh Server: oast.site
...
After (nuclei):
$ /usr/bin/time -a -p -h go run . -mhe 99999999999999999 -itags high,critical -etags dns -jsc 110 -rl 0 -l target.txt -duc -rl 0
...
gc 80 @42.776s 5%: 0.19+250+0.089 ms clock, 1.5+127/496/0+0.71 ms cpu, 315->370->185 MB, 371 MB goal, 0 MB stacks, 1 MB globals, 8 P
gc 81 @43.313s 5%: 0.24+152+0.12 ms clock, 1.9+4.5/268/0+1.0 ms cpu, 316->379->192 MB, 371 MB goal, 0 MB stacks, 1 MB globals, 8 P
gc 82 @43.709s 5%: 0.20+147+0.067 ms clock, 1.6+5.1/285/0+0.54 ms cpu, 329->386->187 MB, 386 MB goal, 0 MB stacks, 1 MB globals, 8 P
Apparently when CPU cores are all busy with threads the gc cycles are planned but not executed, so the memory occupation showed in pprof is low, but the virtual memory pages aren't marked yet as
MADV_FREE_REUSABLE(osx) until the cycle happens and the OS can't reclaim them. Reducing CPU consumption allowed the cycles to happen more frequently: Before (nuclei)After (nuclei):