networkx / nx-parallel

A networkx backend that uses joblib to run graph algorithms in parallel.
BSD 3-Clause "New" or "Revised" License
34 stars 21 forks source link

ENH: Adding and documenting configs in nx-parallel #75

Closed Schefflera-Arboricola closed 2 months ago

Schefflera-Arboricola commented 3 months ago

This PR makes nx-parallel compatible with networkx's config system and documents how to configure nx-parallel using joblib.parallel_config.

In PR https://github.com/networkx/nx-parallel/pull/68, I attempted to create a unified configuration system in nx-parallel by wrapping the joblib.Parallel()(inside nx-parallel algorithms) within a with joblib.parallel_config(configs) context, here configs are extracted from nx.config.backends.parallel. This approach made NetworkX’s config closely mirror joblib’s, giving the appearance of synchronization between the two systems. However, in the last meeting, Dan mentioned that this approach complicates things.

In this PR, I tried to simplify things by clearly documenting both configuration methods for nx-parallel and also added implementation to make nx-parallel compatible with NetworkX’s config. The _set_nx_config decorator wrap the nx-parallel function call within a joblib.parallel_config context only when NetworkX configs are enabled. If not, then configs set via joblib take precedence. (to make nx-parallel compatible with joblib.parallel_config I just had to remove n_jobs from the internal joblib.Parallel)

In the last meeting, Dan also questioned why we need to make nx-parallel compatible with networkx’s config and why can’t we simply use joblib to configure nx-parallel. And it is necessary to keep nx-parallel compatible with other networkx backends and backend functionalities. Like we would need this compatibility if we would want to change backend_priority and some backend configs in a context like this:

with nx.config(backend_priority=[‘cugraph`, ‘parallel’], backends[“parallel”][“n_jobs”] = 5):
    nx.square_clustering(G)

refer Config.md file in this PR for more.

In the issue https://github.com/networkx/nx-parallel/issues/76, I have clearly outlined the challenges in creating a unified configuration system in nx-parallel.

ref. PR https://github.com/networkx/nx-parallel/pull/68 for more context

Thank you :)

Schefflera-Arboricola commented 3 months ago

Notes for reviewers(@dschult):