download all files at the same time

[jafeha]

downloading takes a lot of time in parabird, depending on the internet connection. but if we download all files at the same time, we might get better bandwidth usage. if found this article here: http://stackoverflow.com/questions/2957116/make-2-functions-run-at-the-same-time

its suggesting the following code:

import threading
from threading import Thread

def func1():
    print 'Working'

def func2():
    print 'Working'

if __name__ == '__main__':
    Thread(target = func1).start()
    Thread(target = func2).start()

@jojoo- will that code work for us or will it just nuke our systems to the ground?

[jojoo-]

i worked with multiprocessing rather than Threads. afaik multiprocessing has the higher level interface.

that works in a cgi i wrote:

from multiprocessing import Pool
(...)
pool = Pool(processes=25)
pool.map(function,generator)

that replaced a

for i in generator:
    function(i)

What i read in http://stackoverflow.com/questions/3044580/multiprocessing-vs-threading-python and http://stackoverflow.com/questions/1226584/multiprocess-or-threading-in-python tells me, that we should rather go with threads, but with a queue function

the code above is not suitable for us, because we need to parralelize the same function with different parameters. i'll look into that - after i got caching to work

[jafeha]

i'll move this to the next milestone, because it is no blocker for releasing a first version of pb.