Closed dPys closed 2 months ago
Note, that this now more closely follows the proposed @nx._dispatch structure:
In [1]: import networkx as nx; import nx_parallel
In [2]: G = nx.barabasi_albert_graph(100, 3)
In [3]: H = nx_parallel.ParallelGraph(G)
In [4]: nx.betweenness_centrality(H)
Out[4]:
{0: 0.25151869905207225,
1: 0.049610587404427434,
2: 0.12300459374399848,
3: 0.08304068354565398,
4: 0.13909529780368793,
5: 0.033370757979639815,
6: 0.025224691529040805,
7: 0.010337586781618939,
8: 0.013377277246126217,
9: 0.05723111951263706,
10: 0.1619649220082718,
...
91: 0.000662037520417236,
92: 0.0004985754985754986,
93: 0.0006179450357121424,
94: 0.0005769500203637495,
95: 0.0020165815010068505,
96: 0.0016015554110792203,
97: 0.0015408701683211486,
98: 0.0019846075040880237,
99: 0.0016253091741785242}
I'm so glad to see the state of nx-parallel improve to what it is now. It's really starting to come together @Schefflera-Arboricola ! Even though this PR is being closed due to the merge conflicts, would still be nice to be listed as a contributor?
By the way-- are you all still meeting regularly (Wednesdays if I recall? I believe I have the link still...) If so, would love to start sitting in on those discussions? This repo seems ripe for a code sprint and perhaps even a conference submission?
Thank you!
And thank you for your contributions. I apologize we don't have you as a contributor on this repository. I should have imported commits of this PR into my PR when I was borrowing the create_iterables
function from this PR. At that time, I didn’t even know it was possible to add someone else's commits to my PRs and I very recently learned about it. Please let me know how we can credit you for your contributions.
And yes we have weekly meetings, here’s the calendar link: https://scientific-python.org/calendars/networkx.ics. We’d be delighted to have you join! Also, feel free to lead any sprints on nx-parallel at any upcoming conferences.
Also, if you could give any feedback as a user of nx-parallel then that will be great too :)
Thank you and hope to see you around!
You're very welcome! No worries about the commits—happy to have contributed. I joined last week's meeting and should also be there this afternoon. Thanks for the invite and offer to lead sprints on nx-parallel at any upcoming conferences. I don't have any conferences that I'm planning to attend in the next few months, but will consider sprinting if/when I do. And, of course, I'd be glad to provide feedback on nx-parallel as I continue tinkering.
This PR includes the following changes:
Backends
class to a new moduleexternal
for handling different parallelization backends ( "multiprocessing", "dask", "ray", "loky", "threading", and "ipyparallel") through the joblib API.optional_package
function to thenx_parallel/misc.py
file to accommodate for the possibility of multiple backend dependencies.partition
module, comprising generalNxMap
andNxReduce
classes that can be used to more easily and consistently chunk, map, and reduce parallelizable components of most nx algorithms (as of now, "nodes", "edges", "isolates", and "neighborhoods" are supported).algorithms
module as proposed by @20kavishs, but with the help for thepartition
classes, this greatly simplifies the implementation for each parallel algorithm variant.dependencies
section in thepyproject.toml
file to impose some minimal version requirement for NetworkX and joblib.