velvia / compressed-vec

SIMD Floating point and integer compressed vector library
Apache License 2.0
78 stars 8 forks source link
compression rust simd

compressed_vec

crate documentation CircleCI

Floating point and integer compressed vector library, SIMD-enabled for fast processing/iteration over compressed representations.

This is a compressed vec library, rather than a compression library. What does that mean? A compression library takes some uncompressed data and provides essentially compress() and decompress() functions. Typically you have to decompress data to be able to do anything with it, resulting in extra latency and allocations.

On the other hand, this compressed vec library allows you to iterate over and process the compressed representations directly. It is designed to balance fast iteration and SIMD processing/filtering, while compressing vectors to within 2x of the best columnar compression technology such as Apache Parquet, using techniques such as delta and XOR encoding. Applications:

Performance Numbers

Numbers are from my laptop: 2.9 GHz Core i9, 6/12 cores, 12MB L3, AVX2; from running cargo bench vector, which benchmarks a filter-and-count-matches operation directly on encoded/compressed vectors.

Vector type(s) Elements/sec Raw GBs per sec
u32 dense (no sparsity) 1.7 Gelems/s 6.8 GB/s
u32 sparse (99% zeros) 13.9 Gelems/s 55.6 GB/s
Two u32 vectors (sparse + dense)* 1.3-5.2 Gelems/s 5-20 GB/s
u64 vector, dense 955M - 1.1 Gelems/s 7.6 - 9.1 GB/s
f32, XOR encoded, 60% density 985 Melems/s 3.9 GB/s

Creation, Iteration

To create an f32 compressed vector:

    use compressed_vec::VectorF32XorAppender;
    let mut appender = VectorF32XorAppender::try_new(2048).unwrap();
    let encoded_bytes = appender.encode_all(vec![1.0, 1.5, 2.0, 2.5]).unwrap();

The simplest way to iterate on this compressed vector (note this does not allocate at all):

    use compressed_vec::VectorReader;
    let reader = VectorReader::<f32>::try_new(&encoded_bytes[..]).unwrap();
    let sum = reader.iterate().sum::<f32>();   // Yay, no allocations!

Filtering and SIMD Processing

iterate() is the easiest API to go through individual elements of the compressed vector, but it is not the fastest. Fast data processing, such as done in the filter-and-count benchmarks above, are performed using Sinks, which are defined in the sink module. Sinks operate on a SIMD word at a time, and the sink API is designed for inlining.

For example, let's say that we want to add 2.5 to the f32 vector above, and then write out the results to a Vec<f32>. Internally, XOR encoding and decoding is performed (using a sink). The sinks can be stacked during decoding, for an almost entirely SIMD pipeline:

    use compressed_vec::{VectorReader, AddConstSink, VecSink};
    let reader = VectorReader::<f32>::try_new(&encoded_bytes[..]).unwrap();
    let mut vecsink = VecSink::<f32>::new();
    let mut addsink = AddConstSink::new(2.5f32, &mut vecsink);
    reader.decode_to_sink(&mut addsink).unwrap();
    println!("And the transformed vector is: {:?}", vecsink.vec);

Vector Format

Details of the vector format can be found here.

The vector format follows columnar compression techniques used throughout the big data and database world, and roughly follows the Google Procella paper with its custom Artus format:

Collaboration

Please reach out to me to collaborate!