search

go-sif / sif

Sif is a framework for fast, predictable, general-purpose distributed computing in the map/reduce paradigm.
Apache License 2.0
32 stars 3 forks source link

Feature/partition index abstraction #25

Closed Ghnuberath closed 3 years ago

Ghnuberath commented 3 years ago

This PR includes the creation of a new internal interface PartitionIndex, as well as the extraction of the preexisting tree-based implementation to its own internal package. Abstracting the mechanism by which Partitions are indexed is the first step towards including multiple index varieties which are tuned for purposes other than reduction, such as sorting (which Sif does not currently support).

This additionally opens up the codebase to experimentation with alternative implementations of reduction, which could be more performant under specific circumstances (or in general).

codecov-io commented 3 years ago

Codecov Report

Merging #25 (c6ece67) into main (5c468fa) will decrease coverage by 0.40%. The diff coverage is 64.28%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main      #25      +/-   ##
==========================================
- Coverage   57.54%   57.13%   -0.41%     
==========================================
  Files          80       82       +2     
  Lines        4588     4622      +34     
==========================================
+ Hits         2640     2641       +1     
- Misses       1606     1636      +30     
- Partials      342      345       +3
Impacted Files Coverage Δ
testing/test_runner.go 58.69% <ø> (-0.88%) :arrow_down:
internal/dataframe/empty_partition_iterator.go 27.27% <27.27%> (ø)
internal/dataframe/empty_serialized_iterator.go 46.15% <46.15%> (ø)
cluster/coordinator.go 68.18% <50.00%> (-0.77%) :arrow_down:
internal/pindex/tree/partition_tree_iterator.go 46.15% <50.00%> (ø)
.../pindex/tree/partition_tree_serialized_iterator.go 54.34% <50.00%> (ø)
internal/pindex/tree/partition_tree.go 81.12% <80.00%> (ø)
internal/dataframe/plan_executor.go 77.09% <82.35%> (+0.62%) :arrow_up:
internal/util/util.go 85.71% <100.00%> (-7.15%) :arrow_down:
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 20e1b8f...c6ece67. Read the comment docs.