search

OHDSI / WhiteRabbit

WhiteRabbit is a small application that can be used to analyse the structure and contents of a database as preparation for designing an ETL. It comes with RabbitInAHat, an application for interactive design of an ETL to the OMOP Common Data Model with the help of the the scan report generated by White Rabbit.
http://ohdsi.github.io/WhiteRabbit
Apache License 2.0
174 stars 85 forks source link

Out of memory when saving large mapping object #290

Open MaximMoinat opened 3 years ago

MaximMoinat commented 3 years ago

And som other warnings about illegal operations:

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.cedarsoftware.util.io.MetaUtils (file:/Users/maxim/Develop/OHDSI/WhiteRabbit/release_zips/WhiteRabbit_v0.10.3/repo/com/cedarsoftware/json-io/3.0.1/json-io-3.0.1.jar) to field java.lang.Boolean.value
WARNING: Please consider reporting this to the maintainers of com.cedarsoftware.util.io.MetaUtils
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: Java heap space
MaximMoinat commented 3 years ago

A quick solution is to increase the amount of memory assigned when running the jar: In a text editor, open rabbitInAHat.sh (Mac) or rabbitInAHat.bat (Windows) and change the -Xmx option to a larger value. By default this is 1200m, and for me the issue was fixed when increasing this to -Xmx2400m.

exec "$JAVACMD" $JAVA_OPTS -Xmx2400m \
  -classpath "$CLASSPATH" \
  -Dapp.name="rabbitInAHat" \
  -Dapp.pid="$$" \
  -Dapp.repo="$REPO" \
  -Dapp.home="$BASEDIR" \
  -Dbasedir="$BASEDIR" \
  org.ohdsi.rabbitInAHat.RabbitInAHatMain \
  "$@"
blootsvoets commented 3 years ago

The warning is unrelated. Java 11 only warns on certain reflective operations (e.g., changing a private method to public at runtime). In a future Java version, it might prohibit those operations. Sometimes a later version of the library (in this case, json-io) will have fixed the access.

blootsvoets commented 3 years ago

Can you pinpoint in code where the large object is written? It may help to write the object in streaming style rather than collecting the whole dataset in a single object and then write it to disk.

MaximMoinat commented 3 years ago

@blootsvoets The out of memory error is thrown upon creating a string object (json representation of the model). https://github.com/OHDSI/WhiteRabbit/blob/172f8c30c9b252cdf063ef38289156e709aaa3d9/rabbitinahat/src/main/java/org/ohdsi/rabbitInAHat/dataModel/ETL.java#L137-L139