search

bebop / poly

A Go package for engineering organisms.
https://pkg.go.dev/github.com/bebop/poly
MIT License
665 stars 70 forks source link

Add svb compression to slow5 #328

Closed Koeng101 closed 1 year ago

Koeng101 commented 1 year ago

This PR adds a simple way to use StreamVByte compression with slow5. This is the primary way that slow5 files are compressed, so if you are using poly to io between different file stores or pipe between different programs, StreamVByte greatly helps - I've seen reduction in total database size (storing fastq and slow5 reads) from 12Gb to 7.1Gb using just this one weird trick (database engineers hate him)

The code changes have 100% code coverage, and I'm pretty sure I wrote more comments than actual code (not including tests).

Koeng101 commented 1 year ago

@TimothyStiles Any thoughts?

Koeng101 commented 1 year ago

Moving to #341