REST job server for Spark. Note that this is *not* the mainline open source version. For that, go to https://github.com/spark-jobserver/spark-jobserver. This fork now serves as a semi-private repo for Ooyala.
Other
344
stars
135
forks
source link
Migration to LRU+TTL Guava cache in JobResultActor + metrics #123
This PR addresses the issue with using LRU only cache of large size. When multiple job results of bigger size are stored in memory there's no way to evict them until new results are placed into the cache. High memory pressure caused by large cache leads to slow GC and inability to place new results into the cache. Job Server gets stuck and hangs trying to perform GC.
This PR introduces LRU+TTL cache which allows limiting not only number of entries in the cache but keys TTL and leads to a smoother eviction and lower memory pressure. The cache implementation used is Guava cache which comes with a set of useful cache statistics useful to monitor (cache metrics reporting is a part of this PR too).
This PR addresses the issue with using LRU only cache of large size. When multiple job results of bigger size are stored in memory there's no way to evict them until new results are placed into the cache. High memory pressure caused by large cache leads to slow GC and inability to place new results into the cache. Job Server gets stuck and hangs trying to perform GC.
This PR introduces LRU+TTL cache which allows limiting not only number of entries in the cache but keys TTL and leads to a smoother eviction and lower memory pressure. The cache implementation used is Guava cache which comes with a set of useful cache statistics useful to monitor (cache metrics reporting is a part of this PR too).