broadinstitute / gatk

Official code repository for GATK versions 4 and up
https://software.broadinstitute.org/gatk
Other
1.69k stars 588 forks source link

GatherVcfsIntegrationTest.testBlockGather() passes locally, but runs out of memory and gets killed on travis #3208

Open droazen opened 7 years ago

droazen commented 7 years ago

GatherVcfsIntegrationTest.testBlockGather() is currently disabled in master, since when running on travis with it enabled the JVM appears to run out of memory, resulting in errors like the following:

OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f5293240000, 65536, 1) failed; error='Cannot allocate memory' (errno=12)

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 65536 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /home/travis/build/broadinstitute/gatk/hs_err_pid11418.log

[error occurred during error reporting , id 0xb]
:test[M::bwa_idx_load_from_disk] read 0 ALT contigs
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000715180000, 719847424, 0) failed; error='Cannot allocate memory' (errno=12)

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 719847424 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /home/travis/build/broadinstitute/gatk/hs_err_pid11513.log

This often manifests as a generic "exited with code 137" error, which appears to mean "JVM killed with signal 9":

FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':test'.
> Process 'Gradle Test Executor 1' finished with non-zero exit value 137

See https://travis-ci.org/broadinstitute/gatk/builds/248893875 for an example failed travis build.

We should either modify the test to use less memory, or make more memory available to it on travis. I've tried increasing maxHeapSize for the test suite in build.gradle from 4G to 6G, but this did not help (see https://travis-ci.org/broadinstitute/gatk/builds/249355697)

droazen commented 7 years ago

For @lbergelson, since he wrote the test in question