Embarrassingly parallel algorithms are those that can be easily and efficiently divided into smaller, independent tasks that can be executed concurrently across multiple processing units, such as CPUs, GPUs, or nodes, without requiring significant communication or synchronisation between them. This type of parallelism is often referred to as “embarrassingly parallel” because it’s so straightforward and easy to implement. Refer this to see how to implement embarrassingly parallel algorithms using joblib.
Characteristics of Embarrassingly Parallel Problems(Algorithms):
Independent tasks: Each task can be executed independently, without relying on the output of other tasks.
No communication: Tasks do not require frequent or significant communication with each other.
No data dependencies: Tasks do not rely on shared data or intermediate results from other tasks.
Easy to divide: The problem can be easily split into smaller, parallelizable tasks.
Adding a new algorithm to nx-parallel
Step 1 : Find a NetworkX algorithm that is embarrassingly parallel
Embarrassingly parallel algorithms
Embarrassingly parallel algorithms are those that can be easily and efficiently divided into smaller, independent tasks that can be executed concurrently across multiple processing units, such as CPUs, GPUs, or nodes, without requiring significant communication or synchronisation between them. This type of parallelism is often referred to as “embarrassingly parallel” because it’s so straightforward and easy to implement. Refer this to see how to implement embarrassingly parallel algorithms using joblib.
Characteristics of Embarrassingly Parallel Problems(Algorithms):
Adding a new algorithm to nx-parallel
Step 1 : Find a NetworkX algorithm that is embarrassingly parallel
_dispatchabledecorator(in the networkx's codebase) on the algorithm you have selected to parallelise.Step 2 : Complete the checklist in Contributor's guide
This checklist in Contributor's guide for adding a new algorithm to the package.
Some references:
It would be better to look at currently existing algorithms than previous PRs that add parallel algorithms(like https://github.com/networkx/nx-parallel/pull/60 , https://github.com/networkx/nx-parallel/pull/37 , https://github.com/networkx/nx-parallel/pull/45 and many more) because we have changed a few things.
Feel free to ask questions if you get stuck anywhere or if anything is not clear or if you want something to be explained in a bit more detail here.
Thank you :)