search

ipython / ipyparallel

IPython Parallel: Interactive Parallel Computing in Python
https://ipyparallel.readthedocs.io/
Other
2.59k stars 1.01k forks source link

Question: production use #896

Open dineshbvadhia opened 1 month ago

dineshbvadhia commented 1 month ago

Hi. I have previously used ipyparallel successfully on a multicore laptop and then ran the same code on a 1024 node HPC. The next stage is to run the code on multicore server machines. In this case, the software would be packaged to be distributed to different organizations and run as an interactive production system. The question is whether ipyparallel is suitable for unmanaged production environments?

minrk commented 3 weeks ago

I'm not sure I can really answer that. It's open source so comes entirely without warrantee or support, etc. and is not a super active project, but it is used in production by a number of folks.

Depending on what you are using it for, dask distributed is often a more mature and sophisticated tool.to choose. These days, IPython Parallel is most appropriate for MPI-style tasks (especially actual literal MPI).

dineshbvadhia commented 3 weeks ago

@minrk Don't worry, wasn't looking for a warrantee :). Good to know some folks are using ipyparallel in production.

What I like about ipyparallel is the 'real' cluster computing with stay-alive workers which can be created on each node (core). In my case, each worker runs an http-server acting on incoming requests from the master. MPI is a nice to have but as long as an equivalent gather-communication facility is available then good. Is this possible with dask-distributed?

minrk commented 3 weeks ago

Yes, absolutely. Dask distributed can have long-running workers on many nodes. It also has a much more sophisticated understanding of workers and threads (e.g. you can have one worker on a machine with 8 threads and it will use all resources appropriately).

dineshbvadhia commented 3 weeks ago

@minrk Dask Distributed sounds promising and will look into it. Will be sad to leave ipyparallel though as it has served me incredibly well :(.

On a related subject: Are you aware of similar cluster computing functionality written in rust preferably available through python?

minrk commented 3 weeks ago

Don't get me wrong, you're welcome to use IPython Parallel if you're happy with it and it's been working well for you. These days, dask is just a more mature tool used very widely in production.

I'm not up on the rust ecosystem, so I'm not sure on that one.