Trying out running the datagetter in multi threaded mode.
I've created a branch that spawns off the http get and flatten tool convert parts into their own threads. https://github.com/ThreeSixtyGiving/datagetter/compare/mw/parallel In theory this should mean that we don't have to wait on slow webservers in a serial/sequential way and the flatten process won't block the next download from happening. For example we can do 4 http requests and 4 converts in 4 threads and combine the results afterwards.
It seems to be faster but a bit of time spent getting and comparing the results would be good to verify it. This isn't currently a priority so will come back to it when we have time and a more stable environment to run it on.
Trying out running the datagetter in multi threaded mode.
I've created a branch that spawns off the http get and flatten tool convert parts into their own threads. https://github.com/ThreeSixtyGiving/datagetter/compare/mw/parallel In theory this should mean that we don't have to wait on slow webservers in a serial/sequential way and the flatten process won't block the next download from happening. For example we can do 4 http requests and 4 converts in 4 threads and combine the results afterwards.
It seems to be faster but a bit of time spent getting and comparing the results would be good to verify it. This isn't currently a priority so will come back to it when we have time and a more stable environment to run it on.