aodn-archive / DELETE_ME_go-go-duck

NetCDF aggregation service
1 stars 0 forks source link

OutOfMemoryException when downloading large file #67

Closed dnahodil closed 10 years ago

dnahodil commented 10 years ago

Steps to reproduce Create an aggregation which will result in a large file (problem spotted on a 1.9GB file) Try to download the file using the GoGoDuck webapp

What should happen File downloads

What does happen GoGoDuck webapp throws OutOfMemoryException (seen in log) Download doesn't work

dnahodil commented 10 years ago

Problem is at https://github.com/aodn/go-go-duck/blob/master/grails-app/controllers/au/org/emii/gogoduck/aggr/AggrController.groovy#L22

Looks like the whole file is loaded into memory before it is written out. Streaming the file would probably fix this.

danfruehauf commented 10 years ago

Since the file is already on disk, there should be an easy way to just serve it, no?

dnahodil commented 10 years ago

I'm not sure what you mean by "an easy way". Streaming it won't be hard.

danfruehauf commented 10 years ago

I thought grails might have a way to serve a file statically. Never mind.

danfruehauf commented 10 years ago

If you stream it, will the user know the size of the file once the download starts?

dnahodil commented 10 years ago

There probably isn't a much more simple way than what we're doing. We have a level of indirection which means we need to look up the Job ID to find where the file is on the filesystem.

We can check the size of the file on disk and should be able to set the Content-Length header so their browser knows how big the file will be.

danfruehauf commented 10 years ago

Fair enough. I'm just trying to understand grails capabilities. I was wondering whether there is a simple serveFile(pathToFile) method.

dnahodil commented 10 years ago

Turns out there is a method we could have used but it wasn't introduced until Grails 2 (we're still on 1.3.7). render(file: 'path', fileName: 'file.txt').

danfruehauf commented 10 years ago

Aha! This is what I was after! :)