microsoft / Qcodes

Modular data acquisition framework
http://microsoft.github.io/Qcodes/
MIT License
336 stars 315 forks source link

Concurrency and parallelism. #272

Closed giulioungaretti closed 8 years ago

giulioungaretti commented 8 years ago

The architecture implemented at the moment is showing some issues that are non trivial to fix:

Need to:

And then carefully design how we want to do concurrency and parallelism:

Notes:

Rubenknex commented 8 years ago

I don't know much about parallelism in python but the pathos (https://github.com/uqfoundation/pathos) and corresponding dill (https://github.com/uqfoundation/dill) projects might be interesting to look at? It's a kind of fork from the python multiprocessing library and fixes a lot of pickling issues among others.

akhmerov commented 8 years ago

Can you please explain what you mean by pickling issues? What is actually pickled currently?

giulioungaretti commented 8 years ago

@akhmerov multiprocessing, in use now, uses Pickle to pass objects to the child processes. The list of pickable objects is rather small, and numpy f.ex raccomends not to pickle.

The entire server architecture is build to avoid pickling, which seems a bit brittle because of the need of proxying objects.

akhmerov commented 8 years ago

Sure, but what objects are passed around? The instruments? I imagine a much more sustainable approach would be to instead create all these objects in other processes/workers.

giulioungaretti commented 8 years ago

Closing this down as it will be take care of. See #314 and #315 .