trautmane
commented
4 years ago Hi Martin,
Here are the key parameters we typically run with at Janelia:
--conf spark.executor.cores=30
--conf spark.executor.memory=500g
--conf spark.driver.memory=14g
--maxFeatureSourceCacheGb 2
--maxFeatureCacheGb 15At Janelia, the Spark executors run on nodes with 32 cores, so for each node we give 30 cores to the executor and leave 2 cores for the JVM. We typically run the Spark driver on just one core. We are allocated 15g of memory per core.
Each Spark executor JVM maintains one shared feature source cache containing the source image pixels for each tile and one shared feature cache containing the SIFT features extracted for each tile (or for each tile edge when montage matching). Both --maxFeatureSourceCacheGb and --maxFeatureCacheGb have a default value of 2. My guess is that this is the source of your problem since you are only allocating 4G per worker and it is important leave some memory for non-cache elements.
I suggest you try increasing your worker memory allocation or decreasing the cache sizes.
Here is a little more context to help with deciding how to size the caches ...
The feature source cache improves performance for same layer (montage) matching since tiles can be loaded once and then pulled from cache when rendering each of the potentially four clipped edges. The feature source cache is not as important for cross layer matching since tiles are not clipped - cross layer tiles get rendered once and the extracted features then get cached.
The feature cache improves performance for cross layer matching since features can be extracted once and then reused when looking for matches across different layers.
Technically, you could measure the impact of different cache sizes by comparing runs. I have not done that, but it would be interesting to see. Setting the max cache sizes to 0 will disable caching altogether.
Finally, the caches do have to be cleaned-up when they fill. So it is a good idea to only make them as large as you really need. Output from the tile pair client is sorted to keep similar pairs together with the hope of improving cache hits and allowing smaller cache sizes to be sufficient.
Hope this helps, Eric
martinschorb
Hi,
I am running into
java.lang.OutOfMemoryError: Java heap spacewhen trying to run the SIFT client on spark. Even though every worker has 4GB of memory assigned, it only initializes them with 1024M and shows them with much less (366M) when running.Any idea what could cause this?