Multiprocessing

[brokkr]

Set up a socket for receiving feed updates and fire off one process for each subscription. The feed processes report back to the socket. On that socket runs a single, serial downloader that processes the updates (little Wanted+Unwanted+Lacking etc. packages). The processing includes deletes, downloads, and reports to user. The downloader/main process simply deals with the updates in the order they appear on the socket, i.e. more responsive servers will get first in line.

The proposed distinction between multiple update processes and main process is identifiable in the current code as that between 'plans' and 'execution'.

Since the downloading will still be serial, multiprocessing won't accomplish much in terms of speed gains but it should minimize 'lag' and waiting. We stay away from parallel downloads partly because each dl would steal bandwidth from the others, partly because most updates won't see multiple downloads if your average user subscribes to say 10-20 podcasts and update once an hour (assuming). Finally and most importantly, total multiprocessing invites far more chaos when things go wrong and would require a greater ui rethink.

[brokkr]

Since each worker would only be running a feedparser instance and doing nothing too cpu intensive, multithreading should be fully sufficient. And instead of implementing our own network sockets and protocols, maybe we just use Queue, yes?

• Each thread is a subclass of Thread with an init that is handed q, a Queue instance, as well as conf and a sub.
• It has a run method that make a Feed, Combo and Wanted instance. (stops around line 70)
• It dumps it's results into q and runs q.task_done()
• The threads are spawned in poca around line 25

Outcome returns obviously need to be modified to accomodate new structure.

[brokkr]

Q: When we're running down the queue how do we know when we're done?

In other words: If the queue is empty is it because a thread hasn't returned yet or because all threads have ended and we're through.

We know that all threads have ended when the

for t in threads:
    t.join()

loop is over. But we don't want to wait for that before starting to process threads. What we need is to check if there are 'living threads' still each time we end one subscription.

EDIT: No, we don't.

• We know how many threasdd we started (len(threads))
• We know how many are still in q (q.unfinished_tasks)
• We can start a counter on finished_tasks

We don't even need to lock anything down as these are all in the main process:

q.unfinished_tasks + finished_tasks == len(threads)

[brokkr]

Output:

• We want to do the same output functions whether we're running a single dl thread or multiple concurrent threads. This means we default to something quite basic (plans). If the user wants more detail, there is the -v flag. We explicitly warn against using the flag with concurrent threads because it fs st up. This way we don't have to detect anything or write a lot of ifs or multiple loggers into the code.
• What about errors on single entries? In serial it's best to drop them into the natural stream. This becomes impossible in concurrent mode unless we can find a way to postpone 'plans' output in concurrent thread mode.
• Maybe... if instead of using plans, we just summarize the summary? Count len(dl_ed) etc.?
• Logging: The only challenge here is locking the file so that multiple subupgrades do not attempt writing at the same time.