Currently each http requests blocks the server. Since twisted is single threaded unblock by using deferred(s).
The basic blocking reasons that both need to be fixed is: Reading from many files on disk to get the json blobs (fix by reading a few files at a time and then deferring until later
To fix this:
[ ] don't require file system reads to get the dates of the data files (just some cleanup)
[ ] read a set of files at a time, and then pause (maybe for 10 or 100 ms to allow other work to happen)
Currently each http requests blocks the server. Since twisted is single threaded unblock by using deferred(s). The basic blocking reasons that both need to be fixed is: Reading from many files on disk to get the json blobs (fix by reading a few files at a time and then deferring until later
To fix this:
[ ] don't require file system reads to get the dates of the data files (just some cleanup)
[ ] read a set of files at a time, and then pause (maybe for 10 or 100 ms to allow other work to happen)