Closed andres-erbsen closed 1 year ago
the memory usage solely comes from keeping track of the ratios along the optimisation. Those are then put into the csv files, too (hence they are so big). Those are used to generate the plot of how the optimisation ran after. Seems like you've given each process 100M evaluations. (I've never done so many, max was around 10M I believe.) However, opitimisation depends heavily on Curve and machine. Maybe check the generated pdf's on how the optimisation went, and then see if it feels that there is much more to improve after say 1M/5M evals. Curve25519 is not too big of a curve, and there is not too much to explore (as compared to 448, or 521 for instance). I could also see not keeping track of the ratios, which should keep the memory/storage down.
Do you have a good sense of how much RAM and disk CryptOpt should use? I haphazardly attempted a week-long CryptOpt run and saw it reach memory exhaustion on day 2. Here's an output from the rare instance that got killed by node instead of Linux:
Does 4GB/process look like expected memory usage to you? I could provision that much, but I can't think of a reason why it would be needed.
JS stacktrace:
I believe here is the invocation I used:
Additionally, the results directory seems to have acquired 49GB of csv files (and some asm and json). Are these something I'd may want to look at, or perhaps I should not be collecting them at all?