bturner
commented
6 years ago We're investigating using NuProcess in Bitbucket Server, so I put together a simple jmh benchmark to try and get a feel for its performance when pumping a large amount of data from a process (serving Git hosting operations can require pumping an enormous amount of data). My benchmark was very simple:
public class NuProcessBenchmark {
public static void main(String[] args) throws Exception {
System.setProperty("com.zaxxer.nuprocess.threads", "cores");
Options options = new OptionsBuilder()
.addProfiler(GCProfiler.class)
//.forks(0) //Set when running with JProfiler
.forks(2)
.jvmArgsAppend("-Dcom.zaxxer.nuprocess.threads=cores")
.include(NuProcessBenchmark.class.getSimpleName())
//.measurementIterations(20) // Set when running with JProfiler
.measurementIterations(50)
//.threads(5) // Set when running with JProfiler
.threads(10)
.timeUnit(TimeUnit.SECONDS)
.warmupIterations(1)
.build();
new Runner(options).run();
}
@Benchmark
@Test
public void cat() throws Exception {
NuProcessBuilder builder = new NuProcessBuilder("cat", "/tmp/random.dat");
builder.setProcessListener(new NuAbstractProcessHandler() {
private final CRC32 crc32 = new CRC32();
@Override
public void onExit(int statusCode) {
assertEquals(3407529827L, crc32.getValue());
}
@Override
public void onStdout(ByteBuffer buffer, boolean closed) {
crc32.update(buffer);
}
});
NuProcess process = builder.start();
assertEquals(0, process.waitFor(1L, TimeUnit.DAYS));
}
}(/tmp/random.dat is a 250MB file created using using head -c 250M /dev/urandom >/tmp/random.dat)
When running the benchmark, I noticed that the GC churn appeared to be extremely high, triggering heavy GC. (jmh showed ~34MB/s churn, compared to <200KB/s churn from a basically identical benchmark using our existing processutils library)
Running JProfiler (my benchmark above has some comments with the values I used when profiling), analyzing allocation hotspots showed ProcessEpoll.process, with a graph like this:
JNA is the primary culprit, triggering well over 500MB in allocations, as well as allocating a fair number of Integers.
With the changes above, profiling the same benchmark now shows:
Notice that:
- The number of
Integerallocations has been massively reduced - The number of JNA allocations has been cut approximately in half, except for
Pointerallocations have been entirely removed
This reduces the reported churn rate from 34MB/s to 14MB/s, which results in significantly less GC runs.
The churn rate is extremely important to Bitbucket Server because we run an enormous number of Git processes and, on larger servers, can easily have over 100 git processes running in parallel. Not all of the processes will be pumping hundreds of megabytes of data, but such processes are anything but rare (and 250MB is far from the largest; repositories that are tens of gigabytes, while not common, are not exactly rare, either).
(Edit: For reference, the benchmark was run on a Linux Mint 18.3 x86_64 VM, with 8 VCPUs (AMD Threadripper 1950X) and 16GB of RAM, using Oracle Java 8u151 and Maven 3.3.9.)
lfbayer
brettwooldridge
autoReadand/orautoWriteonEpollEventstructs depending on how they're used, inProcessEpollEpollEvents in theeventPoolhaveautoReaddisabled because they are used a write-only context (Callingepoll_ctl)triggeredEventEpollEventhas bothautoRead/autoWritedisabledepoll_waitreturns 1,read()is called explicitly, but if it returns 0 native memory is not readepoll_ctlto register for further stdin events,write()is called explicitlyEpollEventand added a unit test to verify the size of the struct is 12 bytesEpollEvent.EpollData.ptrbecause, even though it's never used, JNA still allocates itIntegers withints usingInteger.MIN_VALUEas a marker, when acquiring descriptors, to avoid allocating 3 perprocess()loopIntegers will still be created thenExtra bits:
.gitignore