Open Schefflera-Arboricola opened 8 months ago
Thanks for this summary. It has been a while since I looked at that PR. I think it is worth thinking about options for commonly used code to make it easier to write new functions -- and also easier to read once you know the commonly used parts. I'm sure that we don't want to do all the suggestions in that PR and I'm sure we probably should be doing some or even many of them. Using functions instead of classes is also a fine way to approach it. Classes are often better when there is data that is very close to many functions -- the class avoids having to pass the data around a lot. But functions are often better as building blocks that we can put together to do complicated things where each part is simple.
We should probably try several approaches to identify which is the best -- and we should know that our personal preference will color the results. There probably isn't a "right way". But I'm sure tere are non-easy ways. :) Overall your suggestions seem reasonable.
Some Side Notes:
Here, Map
and Reduce
come from MapReduce architecture(ref. 1, 2). It is a distributed programming model. It breaks down tasks into map and reduce phases, where the map
tasks process input data and produce intermediate results, which are then aggregated and processed by reduce
tasks to generate the final output.
As of now, there are no distributed graph algorithms in the repo. All are parallel. Because for joblib.Parallel
the sub-tasks run concurrently, they are within the same computing environment, making it a parallel computing strategy rather than a distributed one(ref. 3)(but I'm not 100% sure about this!)
I took a look at PR #7 and wanted to share some thoughts for consideration:
Code Organization in
external.py
andmisc.py
: I suggest keeping most of theBackend
class (inexternal.py
) and moving theoptional_package
function (inmisc.py
) toexternal.py
. My reasoning is that, we probably want to have networkx and joblib as our main dependencies and we would probably be just using other packages for joblib's backends, so it seems logical to have theoptional_package
function with theBackend
class. Let me know if you see any potential drawbacks to this approach.Exclusion of
NxReduce
inpartition.py
: I propose excluding theNxReduce
class frompartition.py
. Since most of its functions are one-liners, it might be more intuitive for new contributors to write these directly instead of using theNxReduce
class. Additionally, this adjustment could enhance code readability, especially for those new to the library. I couldn't identify any scenarios where we'd preferNxReduce
over a simplereduce(lambda ...)
. Please share your insights on this.Refactoring the
NxMap
class : For theNxMap
class, I think we would have sufficient instances where it would be useful. But, maybe it would be better to have 2 independent functions(excluding__call__
, bcoz one-line function) instead of oneNxMap
class, like if we want to have theget_chunks
parameter(in PR #29) that lets the user define their ownnode_chunks
then we would not be requiringchunks
, but we might requirecreate_iterables
function independently(depending on the algo). Please let me know your thoughts on this.Thank you :)