Concurrency and parallelism.

microsoft / Qcodes

Modular data acquisition framework

http://microsoft.github.io/Qcodes/

MIT License

336 stars 315 forks source link

Concurrency and parallelism. #272

Closed giulioungaretti closed 8 years ago

giulioungaretti commented 8 years ago

The architecture implemented at the moment is showing some issues that are non trivial to fix:

pickling,
monitoring,
performance
in general seems to hard to grasp for the users
intermittent test failures due to syncing issues

Need to:

[ ] "Foreground" fully functional with live plot
[ ] No multiprocessing as default, keep in the code as "experimental"

And then carefully design how we want to do concurrency and parallelism:

python or not python ?
- if not python ØMQ, erlang, f# backends
servers running on fridges and instruments

Notes:

@akhmerov and @jbweston from kwant <3 concurrent.futures.Executor

Rubenknex commented 8 years ago

I don't know much about parallelism in python but the pathos (https://github.com/uqfoundation/pathos) and corresponding dill (https://github.com/uqfoundation/dill) projects might be interesting to look at? It's a kind of fork from the python multiprocessing library and fixes a lot of pickling issues among others.

akhmerov commented 8 years ago

Can you please explain what you mean by pickling issues? What is actually pickled currently?

giulioungaretti commented 8 years ago

@akhmerov multiprocessing, in use now, uses Pickle to pass objects to the child processes. The list of pickable objects is rather small, and numpy f.ex raccomends not to pickle.

The entire server architecture is build to avoid pickling, which seems a bit brittle because of the need of proxying objects.

akhmerov commented 8 years ago

Sure, but what objects are passed around? The instruments? I imagine a much more sustainable approach would be to instead create all these objects in other processes/workers.

giulioungaretti commented 8 years ago

Closing this down as it will be take care of. See #314 and #315 .