Out of memory error during sbt assembly

[bgyori]

As a follow-up to #665 and #667 having been merged, I tried to do sbt -J-Xmx32G assembly on the latest master and keep getting:

...
[info] Including from cache: scala-reflect-2.11.11.jar
[info] Including from cache: bioresources-1.1.30.jar
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError
    at java.util.concurrent.FutureTask.report(FutureTask.java:122)
...

Note that I'm using the -J-Xmx32G setting and my computer actually has 20+GB free memory. I also tried to remove my whole .ivy cache but that didn't help either.

[kwalcock]

I don't think that the size of the ivy cache should be related. The -J-Xmx32G may be telling sbt how much memory to give java when sbt arranges for a java program to run, but it doesn't seem like it would work to increase the amount of memory that sbt itself is getting. Would just -Xmx work? I'm looking at sbt.bat, but you have something else, and it's quite complicated. Is it (or something similar) worth a quick try?

[MihaiSurdeanu]

For sbt config, I believe you have to edit this file instead: https://github.com/clulab/reach/blob/master/.sbtopts

[bgyori]

Thanks! Based on your comments I tried sbt -Xmx16g assembly but for that, I get [error] Not a valid command: Xmx16g, I also looked at .sbtopts and changed the first line from -J-Xmx13G to -J-Xmx32G but got the same out of memory error.

[kwalcock]

I found the sbt file and it's informative reading. You might try export JAVA_OPTS=-Xmx12G and that might be used when starting sbt. Or the switch can be placed in .jvmopts.

[bgyori]

Thanks @kwalcock, I tried both of those settings but it didn't help. I also tried to do sbt assembly on a different machine, also with 32G RAM, and got the same out of memory error. Does it work for either of you? This has never been a problem so I suspect that it's due to one of the two PRs I mentioned above, though I'm not sure how.

[MihaiSurdeanu]

Well, I can at least confirm I can replicate this issue. I'll look into it. I'd appreciate help from @kwalcock if he has any cycles available :)

[MihaiSurdeanu]

I wonder if this is because now processors-models is 2.2GB...

[kwalcock]

I'll be looking at this tomorrow when I check on getting Scala 2.11/12 support added.

[MihaiSurdeanu]

A couple of updates:

• sbt is indeed using the settings from .sbtopts (I set them incorrectly and it complained).
• I tried setting -Xmx200G and it still ran out of memory. So I suspect it's a different JVM memory zone that needs to be configured, not the main RAM.

[kwalcock]

There is a bug/feature in older versions of the assembly plugin that make it unable to work with some large files. It was fixed here (https://github.com/sbt/sbt-assembly/pull/312) and released here (https://github.com/sbt/sbt-assembly/releases/tag/v0.14.7). The current version is addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.10") and when I use it, there are no complaints about memory. There is a complaint about something else, a merge conflict. I'm adding code to deal with both problems in a PR. I'm not sure it's the correct solution, but that can be tested afterwards. Assembly completed now after 12 minutes without complaint. Whether the program runs correctly or not I will leave to others for now.

[MihaiSurdeanu]

@bgyori: can you please run a quick test?

[bgyori]

Fixed in #669