Worker processes connect to store instead of broker

[turicas]

Pros

• We won't need to pickle and unpickle objects to pass from worker process to broker process (so we can pass to store unpicklable objects and it'll be)

  Cons

• If we use a database in the store, it'll be flooded by lots of collections (one per worker instead of one per broker)
• Needs to take care about the wrapper code so the worker can't access the store code/objects directly

[israelst]

Is there any test about this connection flood?

what is demanding this change?

[turicas]

Currently, no tests (just thoughts). We'll add corpus workers on PyPLN (currently we only have workers for documents) and the amount of data passed from the broker to the worker process will increase in one or two power factors (maybe more, in some cases). So, in this case, maybe the best thing to do is to do not pickle and unpickle this huge amount of data between the process (it can slow down the execution of the job -- we don't want to have this process of 'communicating' between broker and worker to be the bottleneck of the system). Obviously, we need some benchmarks to check if it is really necessary -- my opinion is that it will be necessary and the best solution will be leave the user to choose between using workers or broker to communicate with the store, so probably it'll be a option passed to broker.

[israelst]

I get it