search

purplesyringa / taco

2 stars 0 forks source link

Is this some kind of Reimplementation of Standard ? #1

Closed canewsin closed 2 years ago

canewsin commented 2 years ago

Never heard of taco, are there any existing impls or it's custom compression standard ?

purplesyringa commented 2 years ago

This is a custom implementation. It's undocumented and doesn't even have a matching decompressor. The main difference is that, unlike other compression formats, Taco does not prefer any particular algorithm, and compresses structured data rather than binary files. For instance, the core of bzip2 is the Burrows-Wheeler transform, the core of gzip is LZ, the core of zstandard is entropy encoding. Plus Huffman code, which is usually used in all of these to make compression somewhat more efficient. Although this makes these compression tools universal, they become inefficient in any domain-specific circumstance. For instance, I could write a compressor that easily beats the above for Unicode text in an hour. That's what Taco's about: it detects structure in texts and attempts to derive the best compression scheme, matching different methods with each other recursively.

canewsin commented 2 years ago

I was exploring various options for https://github.com/canewsin/zeronet-rs to compress protocol message size. As you know all messages are just plaintext and msgpack encoded, for py client there should be normal standard, for rust to rust client there should be proper compression algo for messages.

canewsin commented 2 years ago

also for ser/de, I was exploring https://github.com/rkyv/rkyv for rust to rust.

purplesyringa commented 2 years ago

I'd say you're talking about premature optimization. Don't get me wrong, of course there are use cases for custom compression protocols, but as ZeroNet shares files of all kinds, zstandard should work for you. As for serialization, I'd say any binary format is fine, including msgpack (take a look at rmp-serde). The overhead should be pretty much negligible, and if you really want to optimize something, perhaps you should be thinking about transparently converting content.json to a binary format for transmission.

canewsin commented 2 years ago

On compatibility side all that work is completed, I was taking about future clients that are native one's to rust client. I was happy to see, you adopted rust as your one of your lang.

canewsin commented 2 years ago

I was now working on UI server and WebSocket connections, I tried to contact you on ZeroMail, but didn't get reply from you.

canewsin commented 2 years ago

converting content.json to a binary format for transmission.

I do had some plans for the raw json files as well

purplesyringa commented 2 years ago

Guess I'm closing this as it's not an issue. If you have any other questions, feel free to message me here.