Out of memory GC overhead exceeded

[feiranwang]

This is weird since I set on madmax -Xmx=128g in sbt_opts and sampler.java_args. I have 1 million variables and 3 million evidence...

BTW, What does [default-dispatcher-54][taskManager][TaskManager] INFO Memory usage: 526/983MB (max: 983MB) stand for?

Uncaught error from thread [deepdive-scheduler-1Uncaught error from thread [deepdive-akka.actor.default-dispatcher-4] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[deepdivedeepdive] ] java.lang.OutOfMemoryError: GC overhead limit exceeded at scala.collection.mutable.ListBuffer.$plus$eq(ListBuffer.scala:168) at scala.collection.mutable.ListBuffer.$plus$eq(ListBuffer.scala:45) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.immutable.List.foreach(List.scala:318) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at anorm.Row$class.asMap(Anorm.scala:255) at anorm.SqlRow.asMap$lzycompute(Anorm.scala:293) at anorm.SqlRow.asMap(Anorm.scala:293) at org.deepdive.extraction.datastore.PostgresExtractionDataStoreComponent$PostgresExtractionDataStore$$anonfun$queryAsMap$1$$anonfun$2.apply(PostgresExtractionDataStore.scala:42) at org.deepdive.extraction.datastore.PostgresExtractionDataStoreComponent$PostgresExtractionDataStore$$anonfun$queryAsMap$1$$anonfun$2.apply(PostgresExtractionDataStore.scala:41) at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at scala.collection.Iterator$GroupedIterator.takeDestructively(Iterator.scala:914) at scala.collection.Iterator$GroupedIterator.go(Iterator.scala:929) at scala.collection.Iterator$GroupedIterator.fill(Iterator.scala:969) at scala.collection.Iterator$GroupedIterator.hasNext(Iterator.scala:972) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) at org.deepdive.inference.FactorGraphBuilder$$anonfun$2.apply(FactorGraphBuilder.scala:77) at org.deepdive.inference.FactorGraphBuilder$$anonfun$2.apply(FactorGraphBuilder.scala:69) at org.deepdive.extraction.datastore.PostgresExtractionDataStoreComponent$PostgresExtractionDataStore$$anonfun$queryAsMap$1.apply(PostgresExtractionDataStore.scala:48) at org.deepdive.extraction.datastore.PostgresExtractionDataStoreComponent$PostgresExtractionDataStore$$anonfun$queryAsMap$1.apply(PostgresExtractionDataStore.scala:34) at scalikejdbc.LoanPattern$.using(LoanPattern.scala:29) at scalikejdbc.package$.using(package.scala:76) at org.deepdive.datastore.JdbcDataStore$class.withConnection(JdbcDataStore.scala:15) at org.deepdive.datastore.PostgresDataStore$.withConnection(PostgresDataStore.scala:9) at org.deepdive.extraction.datastore.PostgresExtractionDataStoreComponent$PostgresExtractionDataStore.queryAsMap(PostgresExtractionDataStore.scala:34) at org.deepdive.inference.FactorGraphBuilder$class.addFactorsAndVariables(FactorGraphBuilder.scala:69) at org.deepdive.inference.FactorGraphBuilder$PostgresFactorGraphBuilder.addFactorsAndVariables(FactorGraphBuilder.scala:18) at org.deepdive.inference.FactorGraphBuilder$$anonfun$receive$1$$anonfun$1.apply$mcV$sp(FactorGraphBuilder.scala:57) at org.deepdive.inference.FactorGraphBuilder$$anonfun$receive$1$$anonfun$1.apply(FactorGraphBuilder.scala:57) java.lang.OutOfMemoryError: GC overhead limit exceeded at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:408) at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:374) at java.lang.Thread.run(Unknown Source) 04:39:11.470 [deepdive-scheduler-1][ActorSystem(deepdive)][ActorSystemImpl] ERROR exception on LARS’ timer thread java.lang.OutOfMemoryError: GC overhead limit exceeded at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:408) ~[sampler-assembly-0.1.jar:0.1] at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:374) ~[sampler-assembly-0.1.jar:0.1] at java.lang.Thread.run(Unknown Source) ~[na:1.7.0_15] 04:39:11.472 [deepdive-scheduler-1][ActorSystem(deepdive)][ActorSystemImpl] INFO starting new LARS thread 04:39:11.472 [default-dispatcher-29][taskManager][TaskManager] INFO Memory usage: 656/910MB (max: 910MB) 04:39:11.473 [deepdive-scheduler-1][ActorSystem(deepdive)][ActorSystemImpl] ERROR Uncaught error from thread [deepdive-scheduler-1] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled java.lang.OutOfMemoryError: GC overhead limit exceeded at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:408) ~[sampler-assembly-0.1.jar:0.1] at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:374) ~[sampler-assembly-0.1.jar:0.1] at java.lang.Thread.run(Unknown Source) ~[na:1.7.0_15] 04:39:11.476 [default-dispatcher-4][ActorSystem(deepdive)][ActorSystemImpl] ERROR Uncaught error from thread [deepdive-akka.actor.default-dispatcher-4] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled java.lang.OutOfMemoryError: GC overhead limit exceeded at scala.collection.mutable.ListBuffer.$plus$eq(ListBuffer.scala:168) ~[scala-library.jar:0.13.1] at scala.collection.mutable.ListBuffer.$plus$eq(ListBuffer.scala:45) ~[scala-library.jar:0.13.1] at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) ~[scala-library.jar:0.13.1] at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) ~[scala-library.jar:0.13.1] at scala.collection.immutable.List.foreach(List.scala:318) ~[scala-library.jar:0.13.1] at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) ~[scala-library.jar:0.13.1] at scala.collection.AbstractTraversable.map(Traversable.scala:105) ~[scala-library.jar:0.13.1] at anorm.Row$class.asMap(Anorm.scala:255) ~[anorm_2.10-2.1.5.jar:2.1.5] at anorm.SqlRow.asMap$lzycompute(Anorm.scala:293) ~[anorm_2.10-2.1.5.jar:2.1.5] at anorm.SqlRow.asMap(Anorm.scala:293) ~[anorm_2.10-2.1.5.jar:2.1.5] at org.deepdive.extraction.datastore.PostgresExtractionDataStoreComponent$PostgresExtractionDataStore$$anonfun$queryAsMap$1$$anonfun$2.apply(PostgresExtractionDataStore.scala:42) ~[classes/:na] at org.deepdive.extraction.datastore.PostgresExtractionDataStoreComponent$PostgresExtractionDataStore$$anonfun$queryAsMap$1$$anonfun$2.apply(PostgresExtractionDataStore.scala:41) ~[classes/:na] at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) ~[scala-library.jar:0.13.1] at scala.collection.Iterator$GroupedIterator.takeDestructively(Iterator.scala:914) ~[scala-library.jar:0.13.1] at scala.collection.Iterator$GroupedIterator.go(Iterator.scala:929) ~[scala-library.jar:0.13.1] at scala.collection.Iterator$GroupedIterator.fill(Iterator.scala:969) ~[scala-library.jar:0.13.1] at scala.collection.Iterator$GroupedIterator.hasNext(Iterator.scala:972) ~[scala-library.jar:0.13.1] at scala.collection.Iterator$class.foreach(Iterator.scala:727) ~[scala-library.jar:0.13.1] at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) ~[scala-library.jar:0.13.1] at org.deepdive.inference.FactorGraphBuilder$$anonfun$2.apply(FactorGraphBuilder.scala:77) ~[classes/:na] at org.deepdive.inference.FactorGraphBuilder$$anonfun$2.apply(FactorGraphBuilder.scala:69) ~[classes/:na] at org.deepdive.extraction.datastore.PostgresExtractionDataStoreComponent$PostgresExtractionDataStore$$anonfun$queryAsMap$1.apply(PostgresExtractionDataStore.scala:48) ~[classes/:na] at org.deepdive.extraction.datastore.PostgresExtractionDataStoreComponent$PostgresExtractionDataStore$$anonfun$queryAsMap$1.apply(PostgresExtractionDataStore.scala:34) ~[classes/:na] at scalikejdbc.LoanPattern$.using(LoanPattern.scala:29) ~[scalikejdbc_2.10-1.7.4.jar:1.7.4] at scalikejdbc.package$.using(package.scala:76) ~[scalikejdbc_2.10-1.7.4.jar:1.7.4] at org.deepdive.datastore.JdbcDataStore$class.withConnection(JdbcDataStore.scala:15) ~[classes/:na] at org.deepdive.datastore.PostgresDataStore$.withConnection(PostgresDataStore.scala:9) ~[classes/:na] at org.deepdive.extraction.datastore.PostgresExtractionDataStoreComponent$PostgresExtractionDataStore.queryAsMap(PostgresExtractionDataStore.scala:34) ~[classes/:na] at org.deepdive.inference.FactorGraphBuilder$class.addFactorsAndVariables(FactorGraphBuilder.scala:69) ~[classes/:na] at org.deepdive.inference.FactorGraphBuilder$PostgresFactorGraphBuilder.addFactorsAndVariables(FactorGraphBuilder.scala:18) ~[classes/:na] at org.deepdive.inference.FactorGraphBuilder$$anonfun$receive$1$$anonfun$1.apply$mcV$sp(FactorGraphBuilder.scala:57) ~[classes/:na] at org.deepdive.inference.FactorGraphBuilder$$anonfun$receive$1$$anonfun$1.apply(FactorGraphBuilder.scala:57) ~[classes/:na]

[dennybritz]

Your SBT_OPTS are not working, you can see that the DeepDive process only gets 1GB of memory:

[default-dispatcher-54][taskManager][TaskManager] INFO Memory usage: 526/983MB (max: 983MB)

[feiranwang]

Yes SBT_OPTS are not working... is there other ways to change the heap size? I've tried adding parameters directly in run.sh and environment variable SBT_OPTS... but didn't work

[feiranwang]

I modified sbt/sbt-launch-lib.bash and finally got it right...