search

go-sif / sif

Sif is a framework for fast, predictable, general-purpose distributed computing in the map/reduce paradigm.
Apache License 2.0
32 stars 3 forks source link

Per-column compression and decompression #30

Closed Ghnuberath closed 2 years ago

Ghnuberath commented 2 years ago

Implementing lazy compression and decompression of data at a per-column level. Column data may remain serialized throughout a pipeline run if it is never accessed, saving memory and eliminating unnecessary compute devoted to serialization/deserialization of unused data.

PartitionCompressor interface and associated LZ4 implementation have been deprecated and removed in favour of a general Compressor interface which is used at the column-data-level.

codecov-commenter commented 2 years ago

Codecov Report

Merging #30 (6c0e46f) into main (fa8e10e) will increase coverage by 0.27%. The diff coverage is 66.24%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main      #30      +/-   ##
==========================================
+ Coverage   61.30%   61.58%   +0.27%     
==========================================
  Files          77       77              
  Lines        5107     5086      -21     
==========================================
+ Hits         3131     3132       +1     
+ Misses       1631     1604      -27     
- Partials      345      350       +5
Impacted Files Coverage Δ
internal/partition/partition-reduceable.go 57.24% <ø> (-0.26%) :arrow_down:
internal/pcache/cache.go 68.90% <39.13%> (-0.20%) :arrow_down:
internal/partition/partition.go 66.23% <54.54%> (-4.19%) :arrow_down:
internal/pindex/tree/partition_tree.go 72.67% <60.00%> (ø)
.../pindex/tree/partition_tree_serialized_iterator.go 47.91% <63.63%> (-7.83%) :arrow_down:
internal/partition/partition_serializable.go 65.38% <65.38%> (ø)
internal/compress/lz4.go 78.04% <78.04%> (ø)
cluster/coordinator.go 67.20% <100.00%> (ø)
cluster/worker.go 65.71% <100.00%> (ø)
internal/dataframe/plan_executor.go 70.81% <100.00%> (ø)
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 660a1a4...6c0e46f. Read the comment docs.