A non master-slave partition mode in process function

[wjlei1990]

Hi Lion,

In ASDFDataSet.process(), you used the master-slave working mode. I think you introduce it mainly because of balancing the workload.

I had a discussion with @mpbl about it and we think we should also have another job partition way(the tradition and simple way), which is partition the job prior to job running. For example, if we get 1000 stations and 10 cores, every processors would take 100 and they know which 100 they would take. The advantage of doing that is:

• code is much easier, so more maintainable, less bugs.
• scalable. If in the future we have 100000 stations, and we want to have 1000 cores. The master-slave may suffer from communications between cores.

The drawback:

• we can't ensure workload balance. Under some extreme conditions, it will perform very poor. However, if we shuffle the job queues, it won't be to bad.

What do you think? Worth adding such a method?

[krischer]

No activity in a while. Closing for now.