hbz / lobid

Linking Open Bibliographic Data
https://lobid.org/
Eclipse Public License 2.0
16 stars 4 forks source link

Add fast download using elasticsearch's scan/scroll #175

Closed dr0i closed 9 years ago

dr0i commented 9 years ago

See #176 .

Makes use of elasticsearch scroll and scan. Fits into the API, so content negotiation is possible. Since there are some memory issues in play 2.3, e.g https://stackoverflow.com/questions/28866981/how-to-chunk-responses-in-play-without-running-out-of-memory, some workarounds are built into the code:

It is my experience that Promise is leaking (see hbz/lobid#188) though the memory consumption constraints mentioned above should give us enough time to react if the play instance is going to break not having enough RAM.

acka47 commented 9 years ago

@dr0i What about this open pull request, did you forget it?