search

onyx-platform / onyx

Distributed, masterless, high performance, fault tolerant data processing
http://www.onyxplatform.org
Eclipse Public License 1.0
2.05k stars 205 forks source link

Evaluate coordinated JVM gc to improve latency/throughput #178

Open lbradstreet opened 9 years ago

lbradstreet commented 9 years ago

See paper at http://blog.acolyer.org/2015/05/06/blade-a-data-center-garbage-collector/

This might be fairly easy to achieve, host System/GCs on a timer. Before doing so, it writes out to ZooKeeper that it's GC'ing (maybe to secondary log since this data is very transitory). If peers pick it at random and there are other peers, then write out to the other peers at this time.

Obviously grouping tasks would just have to wait.

This could really help latency, while allowing us to keep the default Java8 garbage collector that has better throughput but can stop the world for 1+ seconds (unlike the G1GC that will be the default in Java9).

lbradstreet commented 9 years ago

Another paper that might be useful reading before we implement this:

https://www.usenix.org/system/files/conference/hotos15/hotos15-paper-maas.pdf