This is still in flight but I wanted to get some feedback from the tooling side before we go too far on the IETF spec for dictionary-compressed responses.
We are considering creating a new file/stream format that adds a 35-byte header before the compressed stream with a magic signature (DCB) and sha-256 hash of the dictionary that was used to compress the resource.
Currently the dictionary hash is sent in a separate header but there may be value in putting the hash in the file itself and removing the need for an extra header.
Optimally, if we go down this path it would be useful for the brotli cli and API's to support generating and decompressing these streams directly rather than wrapping their output in more tooling.
On compression:
Add a flag for generating "Dictionary-Compressed Brotli"
Flag limits the compression window to 16 MB
Flag requires a dictionary to be specified
Output stream is prefixed with DCB + sha-256 hash of dictionary
On decompression:
Add a flag for decompressing "Dictionary-Compressed Brotli" (or autodetect from the magic signature)
Flag requires a dictionary be specified
Flag sets compression window max to 16 MB
On decompression, if the hash doesn't match the provided dictionary, fail
Does this sound reasonable and make sense to add if we do go down the route of specifying a stream prefix for the dictionary-compressed streams? Are there any concerns/suggestions on the plan itself?
This is still in flight but I wanted to get some feedback from the tooling side before we go too far on the IETF spec for dictionary-compressed responses.
We are considering creating a new file/stream format that adds a 35-byte header before the compressed stream with a magic signature (
DCB) and sha-256 hash of the dictionary that was used to compress the resource.Currently the dictionary hash is sent in a separate header but there may be value in putting the hash in the file itself and removing the need for an extra header.
Optimally, if we go down this path it would be useful for the brotli cli and API's to support generating and decompressing these streams directly rather than wrapping their output in more tooling.
On compression:
DCB+ sha-256 hash of dictionaryOn decompression:
Does this sound reasonable and make sense to add if we do go down the route of specifying a stream prefix for the dictionary-compressed streams? Are there any concerns/suggestions on the plan itself?