This PR makes nx-parallel compatible with networkx's config system and documents how to configure nx-parallel using joblib.parallel_config.
In PR https://github.com/networkx/nx-parallel/pull/68, I attempted to create a unified configuration system in nx-parallel by wrapping the joblib.Parallel()(inside nx-parallel algorithms) within a with joblib.parallel_config(configs) context, here configs are extracted from nx.config.backends.parallel. This approach made NetworkX’s config closely mirror joblib’s, giving the appearance of synchronization between the two systems. However, in the last meeting, Dan mentioned that this approach complicates things.
In this PR, I tried to simplify things by clearly documenting both configuration methods for nx-parallel and also added implementation to make nx-parallel compatible with NetworkX’s config. The _set_nx_config decorator wrap the nx-parallel function call within a joblib.parallel_config context only when NetworkX configs are enabled. If not, then configs set via joblib take precedence. (to make nx-parallel compatible with joblib.parallel_config I just had to remove n_jobs from the internal joblib.Parallel)
In the last meeting, Dan also questioned why we need to make nx-parallel compatible with networkx’s config and why can’t we simply use joblib to configure nx-parallel. And it is necessary to keep nx-parallel compatible with other networkx backends and backend functionalities. Like we would need this compatibility if we would want to change backend_priority and some backend configs in a context like this:
with nx.config(backend_priority=[‘cugraph`, ‘parallel’], backends[“parallel”][“n_jobs”] = 5):
nx.square_clustering(G)
TODOs(not done yet to avoid over-crowding this PR; for ease of review; only done for square_clustering right now):
add the _set_nx_config decorator to all algorithms
rename cpu_count() to get_n_jobs()
rename total_cores to n_jobs
Please LMK if it would be possible to get this PR also reviewed and merged before GSoC ends(on August 26th) based on your schedule, so that I can accordingly mention it in my GSoC final report. I'm open to scheduling a meeting to discuss this PR if that would help you review this better and faster.
This PR makes nx-parallel compatible with networkx's config system and documents how to configure nx-parallel using
joblib.parallel_config
.In PR https://github.com/networkx/nx-parallel/pull/68, I attempted to create a unified configuration system in
nx-parallel
by wrapping thejoblib.Parallel()
(inside nx-parallel algorithms) within awith joblib.parallel_config(configs)
context, hereconfigs
are extracted fromnx.config.backends.parallel
. This approach made NetworkX’s config closely mirror joblib’s, giving the appearance of synchronization between the two systems. However, in the last meeting, Dan mentioned that this approach complicates things.In this PR, I tried to simplify things by clearly documenting both configuration methods for
nx-parallel
and also added implementation to make nx-parallel compatible with NetworkX’s config. The_set_nx_config
decorator wrap the nx-parallel function call within ajoblib.parallel_config
context only when NetworkX configs are enabled. If not, then configs set via joblib take precedence. (to make nx-parallel compatible withjoblib.parallel_config
I just had to removen_jobs
from the internaljoblib.Parallel
)In the last meeting, Dan also questioned why we need to make nx-parallel compatible with networkx’s config and why can’t we simply use joblib to configure nx-parallel. And it is necessary to keep nx-parallel compatible with other networkx backends and backend functionalities. Like we would need this compatibility if we would want to change
backend_priority
and some backend configs in a context like this:refer
Config.md
file in this PR for more.In the issue https://github.com/networkx/nx-parallel/issues/76, I have clearly outlined the challenges in creating a unified configuration system in
nx-parallel
.ref. PR https://github.com/networkx/nx-parallel/pull/68 for more context
Thank you :)