search

OHDSI / StudyProtocols

Repository of OHDSI Collaborative Research Protocols
37 stars 41 forks source link

Heap Space Error on CSV Export #1

Closed jduke99 closed 8 years ago

jduke99 commented 9 years ago

Running Oracle. First two exports work fine:

extractAndWriteToFile(conn, "person_count", resultsSchema, sourceName, "T2DM", dbms) extractAndWriteToFile(conn, "person_count_year", resultsSchema, sourceName, "T2DM", dbms)

Last two exports throw an error:

extractAndWriteToFile(conn, "seq_count", resultsSchema, sourceName, "T2DM", dbms) Error executing SQL: Error in .jcall(rp, "I", "fetch", stride): java.lang.OutOfMemoryError: Java heap space Error in value[3L] : no loop for break/next, jumping to top level

extractAndWriteToFile(conn, "seq_count_year", resultsSchema, sourceName, "T2DM", dbms) Error executing SQL: Error in .jcall(rp, "I", "fetch", stride): java.lang.OutOfMemoryError: Java heap space

Any ideas?

schuemie commented 9 years ago

Sorry about that! I had not figured the exported tables would be so big they would not fit in Java's memory.

Please see the java_heap branch for a possible solution: using dbGetQuery.ffdf() instead of querySql() (both in the DatabaseConnector package). This new function will load the data into an ffdf object, our current preferred solution for storing (very) large data objects.