search

mikemccand / luceneutil

Various utility scripts for running Lucene performance tests
Apache License 2.0
203 stars 113 forks source link

Run nightly benchmarks in memory-constrained fashion? #269

Open jpountz opened 5 months ago

jpountz commented 5 months ago

We're currently working on better parallelizing I/O for Lucene queries. To be able to track improvements (and regressions!), it would be nice if we could run nightly benchmarks with a page cache size that is only a fraction of the total index size.

rmuir commented 5 months ago

For now we can use ramhog for this if needed. In the future we could specify cgroupv2 limits in the systemd service/timer for all the benchmarks, or run particular benchmarks with systemd-run, specifying lower limits.

mikemccand commented 5 months ago

@rmuir also suggested using fincore --output-all /path/to/index/* to monitor how many hot/cold pages we see in the index while/after benchmarking.