jakipatryk / spark-persistent-homology

(WIP, not fast enough for any production usage yet) Library for persistent homology computations in Apache Spark.
Apache License 2.0
0 stars 0 forks source link

Dynamically adjust number and usage of partitions during matrix reduction #19

Open jakipatryk opened 1 year ago

jakipatryk commented 1 year ago

As matrix reductions processes, in later stages most of partitions are "idle" (not performing any reduction). It makes sense to investigate if it is possible to improve that behaviour.

Note: this might not be achievable, but it is worth checking.