search

broadinstitute / wdl-runner

Easily run WDL workflows on GCP
BSD 3-Clause "New" or "Revised" License
13 stars 11 forks source link

Configuration of the Cromwell server #8

Closed carbocation closed 5 years ago

carbocation commented 5 years ago

When using this tool to monitor a few hundred jobs, my jobs failed because of an error that seems to be related to Cromwell running out of memory (if digging through prior forum posts led me to the right conclusion).

2019-04-27 21:24:27,485 cromwell-system-akka.dispatchers.backend-dispatcher-34 ERROR - The JES API worker actor Actor[akka://cromwell-system/user/cromwell-service/JES-Singleton/PAPIQueryManager/PAPIQueryWorker-3e9e46a9-342d-43df-a969-da132ac1b9ad#860159558] unexpectedly terminated while conducting 100 polls. Making a new one...
java.net.SocketTimeoutException: Read timed out
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
    at java.net.SocketInputStream.read(SocketInputStream.java:171)
    at java.net.SocketInputStream.read(SocketInputStream.java:141)
    at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
    at sun.security.ssl.InputRecord.read(InputRecord.java:503)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975)
    at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:933)
    at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
    at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735)
    at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
    at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:347)
    at com.google.api.client.http.javanet.NetHttpResponse.<init>(NetHttpResponse.java:37)
    at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:143)
    at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84)
    at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1040)
    at com.google.api.client.googleapis.batch.BatchRequest.execute(BatchRequest.java:233)
    at cromwell.backend.google.pipelines.common.api.PipelinesApiRequestWorker.runBatch(PipelinesApiRequestWorker.scala:56)
    at cromwell.backend.google.pipelines.common.api.PipelinesApiRequestWorker.cromwell$backend$google$pipelines$common$api$PipelinesApiRequestWorker$$handleBatch(PipelinesApiRequestWorker.scala:50)
    at cromwell.backend.google.pipelines.common.api.PipelinesApiRequestWorker$$anonfun$receive$1.applyOrElse(PipelinesApiRequestWorker.scala:35)
    at akka.actor.Actor.aroundReceive(Actor.scala:517)
    at akka.actor.Actor.aroundReceive$(Actor.scala:515)
    at cromwell.backend.google.pipelines.common.api.PipelinesApiRequestWorker.aroundReceive(PipelinesApiRequestWorker.scala:19)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:588)
    at akka.actor.ActorCell.invoke(ActorCell.scala:557)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
    at akka.dispatch.Mailbox.run(Mailbox.scala:225)
    at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
    at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2019-04-27 21:24:27,493 cromwell-system-akka.dispatchers.backend-dispatcher-58 INFO  - JES batch polling interval is 33333 milliseconds
2019-04-27 21:24:27,503 cromwell-system-akka.dispatchers.backend-dispatcher-58 ERROR - The JES API worker actor managed to unexpectedly terminate whilst doing absolutely nothing (Polling stopped itself unexpectedly). This is probably a programming error. Making a new one...
java.lang.RuntimeException: Polling stopped itself unexpectedly
    at cromwell.backend.google.pipelines.common.api.PipelinesApiRequestManager$$anonfun$1.$anonfun$applyOrElse$1(PipelinesApiRequestManager.scala:114)
    at cromwell.backend.google.pipelines.common.api.PipelinesApiRequestManager.onFailure(PipelinesApiRequestManager.scala:199)
    at cromwell.backend.google.pipelines.common.api.PipelinesApiRequestManager$$anonfun$1.applyOrElse(PipelinesApiRequestManager.scala:114)
    at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
    at akka.actor.Actor.aroundReceive(Actor.scala:517)
    at akka.actor.Actor.aroundReceive$(Actor.scala:515)
    at cromwell.backend.google.pipelines.common.api.PipelinesApiRequestManager.akka$actor$Timers$$super$aroundReceive(PipelinesApiRequestManager.scala:32)
    at akka.actor.Timers.aroundReceive(Timers.scala:55)
    at akka.actor.Timers.aroundReceive$(Timers.scala:40)
    at cromwell.backend.google.pipelines.common.api.PipelinesApiRequestManager.aroundReceive(PipelinesApiRequestManager.scala:32)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:588)
    at akka.actor.dungeon.DeathWatch.receivedTerminated(DeathWatch.scala:67)
    at akka.actor.dungeon.DeathWatch.receivedTerminated$(DeathWatch.scala:64)
    at akka.actor.ActorCell.receivedTerminated(ActorCell.scala:431)
    at akka.actor.ActorCell.autoReceiveMessage(ActorCell.scala:573)
    at akka.actor.ActorCell.invoke(ActorCell.scala:556)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
    at akka.dispatch.Mailbox.run(Mailbox.scala:225)
    at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
    at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2019-04-27 21:24:27,508 cromwell-system-akka.dispatchers.backend-dispatcher-58 ERROR - The JES API worker actor Actor[akka://cromwell-system/user/cromwell-service/JES-Singleton/PAPIQueryManager/PAPIQueryWorker-5191fed5-5c77-4fde-9745-04046c86bba8#384380578] unexpectedly terminated while conducting 100 polls. Making a new one...
java.net.SocketTimeoutException: Read timed out
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
    at java.net.SocketInputStream.read(SocketInputStream.java:171)
    at java.net.SocketInputStream.read(SocketInputStream.java:141)
    at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
    at sun.security.ssl.InputRecord.read(InputRecord.java:503)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975)
    at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:933)
    at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
    at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735)
    at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
    at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:347)
    at com.google.api.client.http.javanet.NetHttpResponse.<init>(NetHttpResponse.java:37)
    at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:143)
    at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84)
    at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1040)
    at com.google.api.client.googleapis.batch.BatchRequest.execute(BatchRequest.java:233)
    at cromwell.backend.google.pipelines.common.api.PipelinesApiRequestWorker.runBatch(PipelinesApiRequestWorker.scala:56)
    at cromwell.backend.google.pipelines.common.api.PipelinesApiRequestWorker.cromwell$backend$google$pipelines$common$api$PipelinesApiRequestWorker$$handleBatch(PipelinesApiRequestWorker.scala:50)
    at cromwell.backend.google.pipelines.common.api.PipelinesApiRequestWorker$$anonfun$receive$1.applyOrElse(PipelinesApiRequestWorker.scala:35)
    at akka.actor.Actor.aroundReceive(Actor.scala:517)
    at akka.actor.Actor.aroundReceive$(Actor.scala:515)
    at cromwell.backend.google.pipelines.common.api.PipelinesApiRequestWorker.aroundReceive(PipelinesApiRequestWorker.scala:19)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:588)
    at akka.actor.ActorCell.invoke(ActorCell.scala:557)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
    at akka.dispatch.Mailbox.run(Mailbox.scala:225)
    at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
    at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

The vast majority (?all) jobs ultimately completed successfully, but this was reported as a failed run because of this.

Is it possible to make configuration options for the wdl-runner's instance? (For example, it looks like 4G of RAM is hard coded in the java call.)

carbocation commented 5 years ago

Actually, in my case, this was solved by passing a larger machine type to the gcloud pipelines run command. (And I was wrong about the hardcoded RAM limit - that's just java heap settings, not total memory use.)

geoffjentry commented 5 years ago

Hi @carbocation - what you're running into is that wdl_runner is using an in memory DB instead of a MySQL db, the latter is far more efficient in both cpu and memory. In general it's worth thinking of wdl_runner as a convenient tool for tutorials and workshops but if you're looking to do real work you should consider running Cromwell directly yourself as you'll have a lot more configurability

carbocation commented 5 years ago

Thanks @geoffjentry . I actually identified the wrong error message. It looks like Akka actors dying is not much of a problem? (I usually write go, forgive my lack of familiarity.) The issue that caused my jobs to ultimately fail actually seems to be heap space, which probably gets to your point about standing up a real database. However, I think it will be more straightforward to fork and recompile Docker than to deal with management of a MySQL database. (Specifically, once you build it, you're stuck supporting it, and supporting a SQL server is something I'm not trying to do as a postdoc.) Though perhaps I'm misunderstanding.

For posterity, the error message that actually seems to have killed the pipeline is below; hence my plan to fork and increase the heap as a first pass:

2019-04-27 23:35:53,486 cromwell-system-akka.dispatchers.service-dispatcher-164 ERROR - WriteMetadataActor Failed to properly process data
java.sql.SQLException: java.lang.OutOfMemoryError: Java heap space
    at org.hsqldb.jdbc.JDBCUtil.sqlException(Unknown Source)
    at org.hsqldb.jdbc.JDBCUtil.sqlException(Unknown Source)
    at org.hsqldb.jdbc.JDBCPreparedStatement.addBatch(Unknown Source)
    at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.addBatch(HikariProxyPreparedStatement.java)
    at slick.jdbc.JdbcActionComponent$InsertActionComposerImpl$MultiInsertAction.$anonfun$run$15(JdbcActionComponent.scala:530)
    at slick.jdbc.JdbcActionComponent$InsertActionComposerImpl$MultiInsertAction.$anonfun$run$15$adapted(JdbcActionComponent.scala:528)
    at scala.collection.Iterator.foreach(Iterator.scala:944)
    at scala.collection.Iterator.foreach$(Iterator.scala:944)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1432)
    at scala.collection.IterableLike.foreach(IterableLike.scala:71)
    at scala.collection.IterableLike.foreach$(IterableLike.scala:70)
    at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
    at slick.jdbc.JdbcActionComponent$InsertActionComposerImpl$MultiInsertAction.$anonfun$run$14(JdbcActionComponent.scala:528)
    at slick.jdbc.JdbcBackend$SessionDef.withPreparedStatement(JdbcBackend.scala:386)
    at slick.jdbc.JdbcBackend$SessionDef.withPreparedStatement$(JdbcBackend.scala:381)
    at slick.jdbc.JdbcBackend$BaseSession.withPreparedStatement(JdbcBackend.scala:448)
    at slick.jdbc.JdbcActionComponent$InsertActionComposerImpl.preparedInsert(JdbcActionComponent.scala:501)
    at slick.jdbc.JdbcActionComponent$InsertActionComposerImpl$MultiInsertAction.run(JdbcActionComponent.scala:526)
    at slick.jdbc.JdbcActionComponent$SimpleJdbcProfileAction.run(JdbcActionComponent.scala:30)
    at slick.jdbc.JdbcActionComponent$SimpleJdbcProfileAction.run(JdbcActionComponent.scala:27)
    at slick.dbio.DBIOAction$$anon$4.$anonfun$run$3(DBIOAction.scala:239)
    at scala.collection.Iterator.foreach(Iterator.scala:944)
    at scala.collection.Iterator.foreach$(Iterator.scala:944)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1432)
    at scala.collection.IterableLike.foreach(IterableLike.scala:71)
    at scala.collection.IterableLike.foreach$(IterableLike.scala:70)
    at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
    at slick.dbio.DBIOAction$$anon$4.run(DBIOAction.scala:239)
    at slick.dbio.DBIOAction$$anon$4.run(DBIOAction.scala:237)
    at slick.dbio.SynchronousDatabaseAction$$anon$7.run(DBIOAction.scala:486)
    at slick.basic.BasicBackend$DatabaseDef$$anon$2.liftedTree1$1(BasicBackend.scala:275)
    at slick.basic.BasicBackend$DatabaseDef$$anon$2.run(BasicBackend.scala:275)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.hsqldb.HsqlException: java.lang.OutOfMemoryError: Java heap space
    at org.hsqldb.error.Error.error(Unknown Source)
    at org.hsqldb.SessionData.allocateLobForResult(Unknown Source)
    at org.hsqldb.Session.allocateResultLob(Unknown Source)
    at org.hsqldb.jdbc.JDBCPreparedStatement.performPreExecute(Unknown Source)
    ... 33 common frames omitted
Caused by: java.lang.OutOfMemoryError: Java heap space
    at org.hsqldb.persist.LobStoreMem.setBlockBytes(Unknown Source)
    at org.hsqldb.persist.LobManager.setBytesISNormal(Unknown Source)
    at org.hsqldb.persist.LobManager.setBytesIS(Unknown Source)
    at org.hsqldb.persist.LobManager.setCharsForNewClob(Unknown Source)
    at org.hsqldb.SessionData.allocateLobForResult(Unknown Source)
    at org.hsqldb.Session.allocateResultLob(Unknown Source)
    at org.hsqldb.jdbc.JDBCPreparedStatement.performPreExecute(Unknown Source)
    at org.hsqldb.jdbc.JDBCPreparedStatement.addBatch(Unknown Source)
    at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.addBatch(HikariProxyPreparedStatement.java)
    at slick.jdbc.JdbcActionComponent$InsertActionComposerImpl$MultiInsertAction.$anonfun$run$15(JdbcActionComponent.scala:530)
    at slick.jdbc.JdbcActionComponent$InsertActionComposerImpl$MultiInsertAction.$anonfun$run$15$adapted(JdbcActionComponent.scala:528)
    at slick.jdbc.JdbcActionComponent$InsertActionComposerImpl$MultiInsertAction$$Lambda$2715/1435587177.apply(Unknown Source)
    at scala.collection.Iterator.foreach(Iterator.scala:944)
    at scala.collection.Iterator.foreach$(Iterator.scala:944)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1432)
    at scala.collection.IterableLike.foreach(IterableLike.scala:71)
    at scala.collection.IterableLike.foreach$(IterableLike.scala:70)
    at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
    at slick.jdbc.JdbcActionComponent$InsertActionComposerImpl$MultiInsertAction.$anonfun$run$14(JdbcActionComponent.scala:528)
    at slick.jdbc.JdbcActionComponent$InsertActionComposerImpl$MultiInsertAction$$Lambda$2713/467029792.apply(Unknown Source)
    at slick.jdbc.JdbcBackend$SessionDef.withPreparedStatement(JdbcBackend.scala:386)
    at slick.jdbc.JdbcBackend$SessionDef.withPreparedStatement$(JdbcBackend.scala:381)
    at slick.jdbc.JdbcBackend$BaseSession.withPreparedStatement(JdbcBackend.scala:448)
    at slick.jdbc.JdbcActionComponent$InsertActionComposerImpl.preparedInsert(JdbcActionComponent.scala:501)
    at slick.jdbc.JdbcActionComponent$InsertActionComposerImpl$MultiInsertAction.run(JdbcActionComponent.scala:526)
    at slick.jdbc.JdbcActionComponent$SimpleJdbcProfileAction.run(JdbcActionComponent.scala:30)
    at slick.jdbc.JdbcActionComponent$SimpleJdbcProfileAction.run(JdbcActionComponent.scala:27)
    at slick.dbio.DBIOAction$$anon$4.$anonfun$run$3(DBIOAction.scala:239)
    at slick.dbio.DBIOAction$$anon$4$$Lambda$2710/1874077620.apply(Unknown Source)
    at scala.collection.Iterator.foreach(Iterator.scala:944)
    at scala.collection.Iterator.foreach$(Iterator.scala:944)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1432)
2019-04-27 23:35:57,378 cromwell-system-akka.dispatchers.service-dispatcher-100 ERROR - WriteMetadataActor Failed to properly process data
java.sql.SQLException: java.lang.OutOfMemoryError: Java heap space
    at org.hsqldb.jdbc.JDBCUtil.sqlException(Unknown Source)
    at org.hsqldb.jdbc.JDBCUtil.sqlException(Unknown Source)
    at org.hsqldb.jdbc.JDBCPreparedStatement.addBatch(Unknown Source)
    at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.addBatch(HikariProxyPreparedStatement.java)
    at slick.jdbc.JdbcActionComponent$InsertActionComposerImpl$MultiInsertAction.$anonfun$run$15(JdbcActionComponent.scala:530)
    at slick.jdbc.JdbcActionComponent$InsertActionComposerImpl$MultiInsertAction.$anonfun$run$15$adapted(JdbcActionComponent.scala:528)
    at scala.collection.Iterator.foreach(Iterator.scala:944)
    at scala.collection.Iterator.foreach$(Iterator.scala:944)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1432)
    at scala.collection.IterableLike.foreach(IterableLike.scala:71)
    at scala.collection.IterableLike.foreach$(IterableLike.scala:70)
    at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
    at slick.jdbc.JdbcActionComponent$InsertActionComposerImpl$MultiInsertAction.$anonfun$run$14(JdbcActionComponent.scala:528)
    at slick.jdbc.JdbcBackend$SessionDef.withPreparedStatement(JdbcBackend.scala:386)
    at slick.jdbc.JdbcBackend$SessionDef.withPreparedStatement$(JdbcBackend.scala:381)
    at slick.jdbc.JdbcBackend$BaseSession.withPreparedStatement(JdbcBackend.scala:448)
    at slick.jdbc.JdbcActionComponent$InsertActionComposerImpl.preparedInsert(JdbcActionComponent.scala:501)
    at slick.jdbc.JdbcActionComponent$InsertActionComposerImpl$MultiInsertAction.run(JdbcActionComponent.scala:526)
    at slick.jdbc.JdbcActionComponent$SimpleJdbcProfileAction.run(JdbcActionComponent.scala:30)
    at slick.jdbc.JdbcActionComponent$SimpleJdbcProfileAction.run(JdbcActionComponent.scala:27)
    at slick.dbio.DBIOAction$$anon$4.$anonfun$run$3(DBIOAction.scala:239)
    at scala.collection.Iterator.foreach(Iterator.scala:944)
    at scala.collection.Iterator.foreach$(Iterator.scala:944)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1432)
    at scala.collection.IterableLike.foreach(IterableLike.scala:71)
    at scala.collection.IterableLike.foreach$(IterableLike.scala:70)
    at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
    at slick.dbio.DBIOAction$$anon$4.run(DBIOAction.scala:239)
    at slick.dbio.DBIOAction$$anon$4.run(DBIOAction.scala:237)
    at slick.dbio.SynchronousDatabaseAction$$anon$7.run(DBIOAction.scala:486)
    at slick.basic.BasicBackend$DatabaseDef$$anon$2.liftedTree1$1(BasicBackend.scala:275)
    at slick.basic.BasicBackend$DatabaseDef$$anon$2.run(BasicBackend.scala:275)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.hsqldb.HsqlException: java.lang.OutOfMemoryError: Java heap space
    at org.hsqldb.error.Error.error(Unknown Source)
    at org.hsqldb.SessionData.allocateLobForResult(Unknown Source)
    at org.hsqldb.Session.allocateResultLob(Unknown Source)
    at org.hsqldb.jdbc.JDBCPreparedStatement.performPreExecute(Unknown Source)
    ... 33 common frames omitted
Caused by: java.lang.OutOfMemoryError: Java heap space
    at org.hsqldb.persist.LobStoreMem.setBlockBytes(Unknown Source)
    at org.hsqldb.persist.LobManager.setBytesISNormal(Unknown Source)
    at org.hsqldb.persist.LobManager.setBytesIS(Unknown Source)
    at org.hsqldb.persist.LobManager.setCharsForNewClob(Unknown Source)
    at org.hsqldb.SessionData.allocateLobForResult(Unknown Source)
    at org.hsqldb.Session.allocateResultLob(Unknown Source)
    at org.hsqldb.jdbc.JDBCPreparedStatement.performPreExecute(Unknown Source)
    at org.hsqldb.jdbc.JDBCPreparedStatement.addBatch(Unknown Source)
    at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.addBatch(HikariProxyPreparedStatement.java)
    at slick.jdbc.JdbcActionComponent$InsertActionComposerImpl$MultiInsertAction.$anonfun$run$15(JdbcActionComponent.scala:530)
    at slick.jdbc.JdbcActionComponent$InsertActionComposerImpl$MultiInsertAction.$anonfun$run$15$adapted(JdbcActionComponent.scala:528)
    at slick.jdbc.JdbcActionComponent$InsertActionComposerImpl$MultiInsertAction$$Lambda$2715/1435587177.apply(Unknown Source)
    at scala.collection.Iterator.foreach(Iterator.scala:944)
    at scala.collection.Iterator.foreach$(Iterator.scala:944)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1432)
    at scala.collection.IterableLike.foreach(IterableLike.scala:71)
    at scala.collection.IterableLike.foreach$(IterableLike.scala:70)
    at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
    at slick.jdbc.JdbcActionComponent$InsertActionComposerImpl$MultiInsertAction.$anonfun$run$14(JdbcActionComponent.scala:528)
    at slick.jdbc.JdbcActionComponent$InsertActionComposerImpl$MultiInsertAction$$Lambda$2713/467029792.apply(Unknown Source)
    at slick.jdbc.JdbcBackend$SessionDef.withPreparedStatement(JdbcBackend.scala:386)
    at slick.jdbc.JdbcBackend$SessionDef.withPreparedStatement$(JdbcBackend.scala:381)
    at slick.jdbc.JdbcBackend$BaseSession.withPreparedStatement(JdbcBackend.scala:448)
    at slick.jdbc.JdbcActionComponent$InsertActionComposerImpl.preparedInsert(JdbcActionComponent.scala:501)
    at slick.jdbc.JdbcActionComponent$InsertActionComposerImpl$MultiInsertAction.run(JdbcActionComponent.scala:526)
    at slick.jdbc.JdbcActionComponent$SimpleJdbcProfileAction.run(JdbcActionComponent.scala:30)
    at slick.jdbc.JdbcActionComponent$SimpleJdbcProfileAction.run(JdbcActionComponent.scala:27)
    at slick.dbio.DBIOAction$$anon$4.$anonfun$run$3(DBIOAction.scala:239)
    at slick.dbio.DBIOAction$$anon$4$$Lambda$2710/1874077620.apply(Unknown Source)
    at scala.collection.Iterator.foreach(Iterator.scala:944)
    at scala.collection.Iterator.foreach$(Iterator.scala:944)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1432)
Uncaught error from thread [cromwell-system-akka.dispatchers.engine-dispatcher-90]: Uncaught error from thread [cromwell-system-akka.dispatchers.backend-dispatcher-118]: GC overhead limit exceededUncaught error from thread [cromwell-system-akka.dispatchers.backend-dispatcher-190]: GC overhead limit exceeded, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[Uncaught error from thread [cromwell-system-akka.dispatchers.backend-dispatcher-122]: GC overhead limit exceededGC overhead limit exceeded, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[cromwell-systemUncaught error from thread [cromwell-system-akka.dispatchers.backend-dispatcher-42]: GC overhead limit exceeded, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[Uncaught error from thread [cromwell-system-akka.dispatchers.backend-dispatcher-187]: GC overhead limit exceeded, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[cromwell-system]
Uncaught error from thread [cromwell-system-akka.dispatchers.backend-dispatcher-41]: GC overhead limit exceeded, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[cromwell-system]
Uncaught error from thread [cromwell-system-akka.dispatchers.backend-dispatcher-188]: GC overhead limit exceeded, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[cromwell-system]
2019-04-27 23:36:34,071 cromwell-system-scheduler-1 ERROR - exception on LARS’ timer thread
java.lang.OutOfMemoryError: GC overhead limit exceeded
Uncaught error from thread [cromwell-system-akka.dispatchers.backend-dispatcher-115]: GC overhead limit exceeded, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[Uncaught error from thread [Uncaught error from thread [cromwell-system-akka.dispatchers.backend-dispatcher-177]: GC overhead limit exceeded, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[cromwell-system]
java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
Uncaught error from thread [cromwell-system-akka.dispatchers.backend-dispatcher-175]: Uncaught error from thread [cromwell-system-akka.dispatchers.backend-dispatcher-176]: GC overhead limit exceeded, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled forUncaught error from thread [cromwell-system-akka.dispatchers.backend-dispatcher-117]: GC overhead limit exceeded, Uncaught error from thread [shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[cromwell-system]
Uncaught error from thread [cromwell-system-akka.dispatchers.backend-dispatcher-191]: GC overhead limit exceededUncaught error from thread [cromwell-system-akka.dispatchers.service-dispatcher-145]: GC overhead limit exceeded, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[cromwell-system]
Uncaught error from thread [cromwell-system-akka.dispatchers.backend-dispatcher-127]: GC overhead limit exceeded, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[cromwell-system]
2019-04-27 23:36:53,346 cromwell-system-scheduler-1 INFO  - starting new LARS thread
Uncaught error from thread [cromwell-system-akka.dispatchers.engine-dispatcher-93]
Uncaught error from thread [java.lang.OutOfMemoryError: GC overhead limit exceeded
, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[Uncaught error from thread [cromwell-system-akka.dispatchers.service-dispatcher-164]: GC overhead limit exceeded, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[cromwell-system]
java.lang.OutOfMemoryError: GC overhead limit exceeded