search

go-sif / sif

Sif is a framework for fast, predictable, general-purpose distributed computing in the map/reduce paradigm.
Apache License 2.0
32 stars 3 forks source link

Refactor: schema clarification #12

Closed Ghnuberath closed 4 years ago

Ghnuberath commented 4 years ago

Sif does not actually remove data when the client uses RemoveColumn(), but rather waits until the client explicitly calls Repack() to actually modify the underlying data. As a result, each Sif operation has always had a notion of a "public" schema (which takes into account virtual column removals in order to produce useful error messages for columns which have already been "removed") and a "private" schema (which reflects the true structure of the underlying data).

Handling this across the codebase had gotten quite hacky, so this refactor makes that separation and terminology explicit across the entire codebase.

This refactor exposed several bugs with Repack(), which have been fixed.

codecov-io commented 4 years ago

Codecov Report

Merging #12 into master will increase coverage by 0.76%. The diff coverage is 76.63%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #12      +/-   ##
==========================================
+ Coverage   55.91%   56.67%   +0.76%     
==========================================
  Files          72       72              
  Lines        4661     4783     +122     
==========================================
+ Hits         2606     2711     +105     
- Misses       1767     1781      +14     
- Partials      288      291       +3
Impacted Files Coverage Δ
internal/dataframe/task.go 100.00% <ø> (ø)
internal/partition/partition-collected.go 55.55% <0.00%> (ø)
operations/transform/filter.go 0.00% <0.00%> (ø)
operations/transform/group.go 0.00% <0.00%> (ø)
operations/transform/rename_column.go 0.00% <0.00%> (ø)
operations/transform/repartition_reduce.go 0.00% <0.00%> (ø)
internal/partition/partition-keyable.go 67.64% <33.33%> (ø)
internal/partition/partition-operable.go 53.08% <60.00%> (+1.18%) :arrow_up:
operations/transform/with_column.go 76.47% <73.33%> (-1.31%) :arrow_down:
internal/partition/partition-reduceable.go 65.21% <75.00%> (+3.36%) :arrow_up:
... and 26 more

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update cb52d49...1afe273. Read the comment docs.