reduce blocking in twisted server

[danaronson]

Currently each http requests blocks the server. Since twisted is single threaded unblock by using deferred(s). The basic blocking reasons that both need to be fixed is: Reading from many files on disk to get the json blobs (fix by reading a few files at a time and then deferring until later

To fix this:

• [ ] don't require file system reads to get the dates of the data files (just some cleanup)

• [ ] read a set of files at a time, and then pause (maybe for 10 or 100 ms to allow other work to happen)

[mitra42]

I think this is what you just pushed Dan ?