search

mikera / clisk

The Clojure Image Synthesis Kit
281 stars 13 forks source link

MemoryError in Metaspace #18

Open kephale opened 7 years ago

kephale commented 7 years ago

I'm getting a Caused by: java.lang.OutOfMemoryError: Metaspace from sessions where I create too many clisk.live trees (4k trees of depth <10).

Relevant lines from stack trace (using the 0.11.0 release) at clisk.util$expression_info_internal.invoke(util.clj:32) at clisk.node$node_info.invoke(node.clj:435) at clisk.node$validate.invoke(node.clj:451) at clisk.core$image.doInvoke(core.clj:73)

Edit: I know the simple workaround is increasing metaspace, but I'm wondering if there is an alternative. Ideally this will run for ~100k-1m trees (or more).

mikera commented 7 years ago

Might be dependent on Clojure / Java version? What are you using?

Also it may be possible to strip out the validate call altogether for runtime use

kephale commented 7 years ago

Clojure 1.7 (I'll try 1.8 on the next run) Oracle Java 1.8.0_112-b16

Aha, interesting, I'll try removing the validate calls as well!

Thank you!

mikera commented 7 years ago

Oh one other thing to think about - be careful not to hold onto to many references to the compiled nodes, they basically all require generation of custom classes which might be what is exhausting your metaspace. A few should be OK but this was never designed to handle hundred of thousands of compiled classes at once.

kephale commented 7 years ago

For the record, I believe using an alternative version of image without validate solved it!

mikera commented 7 years ago

Cool thanks for confirming it worked!

mikera commented 7 years ago

Leaving open as a reminder in case we want to optimise this in the future