Remove bad defaults for java options

[uschindler]

While working on https://github.com/apache/lucene/pull/177, I had to fight against bad defaults for Java's command line parameters.

Please don't set any default command line options like -Xbatch or disable tiered compilation with -XX:-TieredCompilation. Both options behave very badly for long running processes like those testing real-world benchmarking. In fact without tiered compilation, the JVM takes like 6 seconds (out of 50 seconds total runtime) until it was fully optimized, because the JVM wasn't able to do it in parallel (due to batch) and only had one chance to optimize at all (non-tiered). During that time the new Panama Java API produced so many garbage instances on heap that 15 seconds following the 6 seconds, the garbage collector was driving crazy to cleanup.

In addition, using batch and no tiered compilation is not reflecting reality. Nobody would run JDK like this, except for very short-running processes that need no optimizations (like unit test cases).

In addition with tiered compilation the benchmark runs 20% faster (except the GC problems introduced by project panama). Running a single benchmark round on main branch with MappedByteBuffer directory ran 57 seconds with tiered and batch, but took <40 seconds on all java defaults. WIth the GC problems mentioned before it was 75 seconds total.

[uschindler]

Here is a more detailed explanation: https://github.com/openjdk/panama-foreign/pull/555#issuecomment-865672909

Please also read the benchamrking output (especially JFR output before that comment).

[dsmiley]

My suspicion is that batch & disable tiered compilation is trying to reduce the number of ancillary activities the JVM is doing that might interfere with the measurements (introduce sporadic noise). The settings need not be reflective of a production system; it's only about measuring and trying to minimize jitter/noise.

[uschindler]

My suspicion is that batch & disable tiered compilation is trying to reduce the number of ancillary activities the JVM is doing that might interfere with the measurements (introduce sporadic noise). The settings need not be reflective of a production system; it's only about measuring and trying to minimize jitter/noise.

Because of this we repeat the whole thing 20 times :-)

In fact disabling tiered makes jitter worse, same applies to Xbatch. @dsmiley: Mike already disabled those settings on the nightly benchmark, it is a relic in this repo, but leads to unrelaistic benchmarks for newcomers or those not adapt the defaults. The results with those options added have huge jitter. Disabling tiered and enabling batch schould only run on microbenchmarks and short-running tasks.

(I opened this issue just for reference).

[dsmiley]

If I recall (at Berlin Buzzwords "spatial chat") did you say the runs were using separate JVMs? (I was surprised to hear that). If it's the same JVM and then it has plenty of time to warm up after 20 runs and so I agree we might as well use realistic settings.

[uschindler]

If I recall (at Berlin Buzzwords "spatial chat") did you say the runs were using separate JVMs? (I was surprised to hear that).

Yes it does. But a single run is still long enough (number of iteration) to have warmup.

Another problem is that you always need a long warmup, because Lucene now uses many Lambdas and method references, which are not affected by those settings. Those need warmup, too - but this is happending in Java-land (invokedynamic and lambda transformations). If you add batch or diable tiered, then it takes even longer and that's the problem!