brotli: implement encoder

dsnet / compress

Collection of compression related Go packages.

BSD 3-Clause "New" or "Revised" License

404 stars 25 forks source link

brotli: implement encoder #42

Open jpillora opened 8 years ago

jpillora commented 8 years ago

Please close this if/when brotli encoder is implemented. Nice work btw!

dsnet commented 8 years ago

Thanks for the issue. I'll definitely try to get a working brotli encoder released soon. I'll give fair warning that the first cut might not perform much better than DEFLATE in terms of it's compression ratio. I plan on doing a iterative approach to implementing brotli, where each iteration improves it's compression performance.

jpillora commented 8 years ago

Cool that sounds fine to me +1 On Sat, 6 Feb 2016 at 7:05 AM Joe Tsai notifications@github.com wrote:

Thanks for the issue. I'll definitely try to get a working brotli encoder released soon. I'll give fair warning that the first cut might not perform much better than DEFLATE in terms of it's compression ratio. I plan on doing a iterative approach to implementing brotli, where each iteration improves it's compression performance.

— Reply to this email directly or view it on GitHub https://github.com/dsnet/compress/issues/42#issuecomment-180536182.

dsnet commented 8 years ago

Placing this here for reference: mholt/caddy#525

akyoto commented 6 years ago

Has this been abandoned or is there any chance somebody is still working on a native implementation?

dsnet commented 6 years ago

I wouldn't say "abandoned", but I'm not actively working on it at the present moment. I have a number of other projects currently occupying my time.

akyoto commented 6 years ago

I see, thank you for the reply! Would have loved to see a native implementation in Go but I realize it's a very time-consuming task.

dmitshur commented 6 years ago

@dsnet In https://github.com/mholt/caddy/issues/525#issuecomment-174615921, you said:

I'm actually currently working on finishing my own implementation of a RFC 1951 DEFLATE compressor. The components built to implement DEFLATE will make implementing Brotli significantly easier.

I had a question about that. Given that a pure Go DEFLATE decompressor already exists in Go standard library (compress/flate), what's the reason you're building your own? Is it because you have different objectives compared to the std lib one, or another reason?

Can the components of compress/flate compressor be used to help implement Brotli compressor, or is there a reason why compress/flate isn't helpful for that?

Just looking to understand this better, thanks.

dsnet commented 6 years ago

Obviously, the lack of traffic on this repo implies haven't done much in that regard :disappointed:

The reason are the following:

I already implemented huffman encoding (since it's the same for bzip2 and brotli), so the only other component left for DEFLATE is just LZ77. Implementing LZ77 would allow me to experiment with different algorithms and understand it better.
There are low level functionality that I would want to be able to do on DEFLATE that I don't think they standard library should ever support. See the zlib manual for what features they provide.

dsnet commented 6 years ago

@blitzprog, I appreciate your enthusiasm, but implementation compression algorithms really does require specialized knowledge. You're welcome to do research on how LZ77 works and various approaches to compute it.

faradaytrs commented 5 years ago

Any progress on this?

dmitshur commented 5 years ago

@dsnet FYI, I was made aware (thanks to @mappu) that there is a pure Go implementation of a Brotli encoder that was created using a c2go-based approach by @andybalholm:

https://groups.google.com/forum/#!msg/golang-nuts/J5MwJnmDpOw/yQacKG2pBwAJ

I haven't had a chance to try it yet, but I wanted to share it with you (and others here).

dsnet commented 5 years ago

Thanks for the update @dmitshur! I've been too busy with my actual work to work on this implementation. I'm glad someone else is providing it.