datacite / mds

The DataCite Metadata Store (MDS)
https://mds.datacite.org
Apache License 2.0
25 stars 16 forks source link

GET /doi: out of memory for large datacentres #306

Closed koelnconcert closed 6 years ago

koelnconcert commented 9 years ago

Looks like the result set is not streamed but buffered completely in MDS.

koelnconcert commented 9 years ago

Remark: Even when streaming results via while (result.next()), the mysql jdbc driver might still buffer everything in heap beforehand. Activate real streaming via:

// Enable mysql streaming of results: http://dev.mysql.com/doc/connector-j/en/connector-j-reference-implementation-notes.html
Statement stat = conn.createStatement(ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY); stat.setFetchSize(Integer.MIN_VALUE);