Providing a version on crates.io?

vigna / sux-rs

Rust implementations of succinct data structures

Apache License 2.0

46 stars 7 forks source link

Providing a version on crates.io? #27

Closed rob-p closed 3 months ago

rob-p commented 12 months ago

Hi all,

This is a really interesting library, that is sure to be useful in many of our projects. Thanks for all of your efforts here (and I'm a huge fan of your work, @vigna)! However, if we'd like to build an application or tool relying on sux-rs and put it on crates.io, we are currently kind of hosed, as there is no version there. There was one ~6 months ago, but it was yanked. I realize that things are still under active development here and improving quickly, but do you think it would be possible to periodically checkpoint those and release them on crates.io?

Thanks! Rob

vigna commented 12 months ago

That should happen very soon. The version on crates.io is just a placeholder. It's just that I don't want to put out APIs and then change them radically.

It's a bit complicated to get that right because Rust was designed without thinking of data structures with implicit representations, and it really gets ugly when you try to use the standard traits. E.g., Index is entirely useless in a succinct or compressed data structure.

Indeed, if you have comments on the traits you see know that would be invaluable input.

vigna commented 12 months ago

In particular, what we're trying to get right is:

IndexedDict, for an indexed dictionary (successor, predecessor, etc.)
BitFieldSlice, for arrays of elements of fixed bit width
Rank / Select
How should we pass inputs that usually comes from buffered files into stuff like VFuncBuilder. Our current idea is to have a Lender returning a Result<T>, so that errors from the underlying I/O can be managed easily, and also one can reuse a buffer for efficiency. This leads however to some complications with type inference and adapters, and the impossibility of using the Iterator/Lender API (you can't use ? with that), so we're trying to understand the best way to use the Rust type system to make things easy for everyone.

rob-p commented 11 months ago

Just pinging @theJasonFan here, in case he has specific input to provide on these traits.

RagnarGrootKoerkamp commented 11 months ago

One possible solution to this may be to publish things as multiple creates with a smaller scope, so that you don't have to commit to a version for everything.

That also has the benefit of reducing dependencies/bloat for users. For example rust-bio does not do this partitioning, and its dependencies are anecdotally always the main thing slowing down my compilation, even if I don't actually use those parts of rust-bio.