JuliaData / TableOperations.jl

Common table operations on Tables.jl interface implementations
Other
47 stars 9 forks source link

Start work on TableOperations.joinpartitions #10

Closed quinnj closed 4 years ago

quinnj commented 4 years ago

With the new Tables.partitions functionality in the Tables.jl package, it can be helpful for non-partitioned in-memory table structures to have a way to treat the separate partitions as a single, long partition. The SentinelArrays.jl package includes a ChainedVector type that allows lazily "chaining" AbstractArrays together, treated as a single, long vector. TableOperations.joinpartitions(x) takes any partitioned input x, and appends the columns together via ChainedVector, producing a JoinedPartitions object which itself, satisfies the Tables.columns interface, i.e. you could do df = DataFrame(TableOperations.joinpartitions(x)) to concatenate all partitions of the input and operate on them as a whole in a DataFrame.

Still need to add tests and docs.

codecov[bot] commented 4 years ago

Codecov Report

Merging #10 into master will decrease coverage by 0.26%. The diff coverage is 90.47%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #10      +/-   ##
==========================================
- Coverage   94.26%   94.00%   -0.27%     
==========================================
  Files           1        1              
  Lines         122      150      +28     
==========================================
+ Hits          115      141      +26     
- Misses          7        9       +2     
Impacted Files Coverage Δ
src/TableOperations.jl 94.00% <90.47%> (-0.27%) :arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 36736df...8979b16. Read the comment docs.