Open davidonlaptop opened 9 years ago
Avocado is always failing at the moment...
what's the error message?
(let's leave comments here, for future tracking)
François Langelier 2:35 PM (22 minutes ago) to David 2015-03-22 17:29:25 ERROR Executor:96 - Exception in task 0.0 in stage 1.0 (TID 16) java.lang.OutOfMemoryError: GC overhead limit exceeded
Potential solution: https://plumbr.eu/outofmemoryerror/gc-overhead-limit-exceeded
If you can run it with jvisualvm attached you will have more idea of what is using the memory and what is the usage of the heap in general.
What JVM parameters you are currently using ?
-tony
On Mon, Mar 23, 2015 at 3:13 PM, David Lauzon notifications@github.com wrote:
Potential solution: https://plumbr.eu/outofmemoryerror/gc-overhead-limit-exceeded
— Reply to this email directly or view it on GitHub https://github.com/GELOG/adamcloud/issues/11#issuecomment-85154098.
Currently, it is the default settings. I think there is no enough memory allocated to the spark workers.
Francois will is describing the steps-by-steps to reproduce the problem.
@flangelier I think I encountered the same problem as you with Avocado! Change the values of the two lines below in the file bin/avocado-submit. 4g is probably more than what you have available in memory and the JVM can't start.
--conf spark.executor.memory=${AVOCADO_EXECUTOR_MEMORY:-4g} \
--driver-memory ${AVOCADO_DRIVER_MEMORY:-4g} \
See: https://github.com/bigdatagenomics/avocado/blob/master/bin/avocado-submit#L56-58
With updated versions of the BDGenomics pipeline (snap, adam, avocado), use a small / medium / large genome to validate the new images and the orchestration scripts. We'll compare these results with the results from S. Bonami and the BDGenomics papers.
References
TODO: Find the data used in the Snap / Adam / Avocado papers.