Closed canewsin closed 2 years ago
This is a custom implementation. It's undocumented and doesn't even have a matching decompressor. The main difference is that, unlike other compression formats, Taco does not prefer any particular algorithm, and compresses structured data rather than binary files. For instance, the core of bzip2 is the Burrows-Wheeler transform, the core of gzip is LZ, the core of zstandard is entropy encoding. Plus Huffman code, which is usually used in all of these to make compression somewhat more efficient. Although this makes these compression tools universal, they become inefficient in any domain-specific circumstance. For instance, I could write a compressor that easily beats the above for Unicode text in an hour. That's what Taco's about: it detects structure in texts and attempts to derive the best compression scheme, matching different methods with each other recursively.
I was exploring various options for https://github.com/canewsin/zeronet-rs to compress protocol message size. As you know all messages are just plaintext and msgpack encoded, for py client there should be normal standard, for rust to rust client there should be proper compression algo for messages.
also for ser/de, I was exploring https://github.com/rkyv/rkyv for rust to rust.
I'd say you're talking about premature optimization. Don't get me wrong, of course there are use cases for custom compression protocols, but as ZeroNet shares files of all kinds, zstandard should work for you. As for serialization, I'd say any binary format is fine, including msgpack (take a look at rmp-serde). The overhead should be pretty much negligible, and if you really want to optimize something, perhaps you should be thinking about transparently converting content.json to a binary format for transmission.
On compatibility side all that work is completed, I was taking about future clients that are native one's to rust client. I was happy to see, you adopted rust as your one of your lang.
I was now working on UI server and WebSocket connections, I tried to contact you on ZeroMail, but didn't get reply from you.
converting content.json to a binary format for transmission.
I do had some plans for the raw json files as well
Guess I'm closing this as it's not an issue. If you have any other questions, feel free to message me here.
Never heard of taco, are there any existing impls or it's custom compression standard ?