Reduce RAM needs for large jobs

[ndushay]

Tommy says "when the downloader validates the checksum, it has to load the entire file into memory. I think it’s not good about freeing up that memory after use - and then it craps out"

Nicholas says: "warcs are typically capped at 1 gb, but sometimes the contents (e.g., video files) can't be truncated. looks like the last one it downloaded was ~2.2 gb"

So the task is to determine:

• [ ] where is the memory going? (confirm above)
• [ ] how can we free it up promptly? (force garbage collect? close files? use different config opts on the java process? etc.)

[thibgc]

About to make a (successful, I hope, this time) pull request that fixes this.

[thibgc]

Pull request #122