ia0 / data-encoding

Efficient and customizable data-encoding functions in Rust
https://data-encoding.rs/
MIT License
177 stars 23 forks source link

SIMD optimization #95

Open thedmm opened 11 months ago

thedmm commented 11 months ago

This crate is currently used by uutils/coreutils for programs like basenc, base32, and base64, and I'm looking for ways to optimize them. I've experimented with an AVX2 implementation of simple Base32 encoding and the results were promising - to this end, I'm looking to write optimized SIMD implementations of encoding and decoding for individual format specifications. Is this something data-encoding can support, given its current API? Would such implementation (and possibly API) changes be welcome? Or should I write my own crate instead?

ia0 commented 11 months ago

Thanks for reaching out!

This crate focuses (in this order) on: correctness, user experience, generality, performance. So SIMD implementations if they only target specific encodings would favor performance over generality, which I'd like to avoid. The way I see there would be a few options:

What do you think? What's the time frame you would like this to be out? And how much effort are you willing to invest?

thedmm commented 11 months ago

Unfortunately, I think the goals I have in mind don't align with the goals for data-encoding.

I love doing these kinds of performance optimizations, so I'm willing to sink a fair chunk of time into this. But given how specific my goals are, I think writing my own crate makes the most sense.

Thanks for creating and maintaining this crate!

ia0 commented 11 months ago

I don't think a proc-macro can deliver the kind of efficiency that I'm hoping to achieve with a manual implementation.

I don't see any theoretical reason it couldn't, but I can see how it seems uncertain (and may need some work to actually get it to work). I'll look into it if I ever get time. I think it's interesting.

I think writing my own crate makes the most sense.

Sounds good! Please send me a link to your crate once it's out. I'm definitely going to take a look at it for both:

ia0 commented 7 months ago

Just an update here. I have a basic design that I think works, but that's too much work for me to finish in the medium term. Maybe I'll pick this back up next year. Keeping open and renaming the issue.