Doing an export from the YourNextMP instance takes a very long time and uses up quite a bit of memory. This sometime results in 504 errors when it takes so long the request times out.
Ideally there should be some way to make this use node's concept of streams and pipes to shift the data straight out of mongo, through the transform module, through gzip and the onto the response. Failing that we might have to shift this off to a background job and then make people poll for success like we do for the import.
Doing an export from the YourNextMP instance takes a very long time and uses up quite a bit of memory. This sometime results in 504 errors when it takes so long the request times out.
Ideally there should be some way to make this use node's concept of streams and pipes to shift the data straight out of mongo, through the transform module, through gzip and the onto the response. Failing that we might have to shift this off to a background job and then make people poll for success like we do for the import.