search

facebook / zstd

Zstandard - Fast real-time compression algorithm
http://www.zstd.net
Other
23.33k stars 2.07k forks source link

Feature request: Standard dictionary for web usage #3100

Open tuxayo opened 2 years ago

tuxayo commented 2 years ago

Describe the solution you'd like End goal is to have web browsers and servers use zstd the same way they now use brotli and gzip. IIUC the first step would be to have a dictionary optimized for web usage. https://github.com/Fyrd/caniuse/issues/4065#issuecomment-674579092

There doesn't seem to be a similar ticket. Feel free to close as dup if it's the case.

felixhandte commented 2 years ago

Hi @tuxayo,

We've discussed this a bunch internally, and we've had related conversations on various issues (#1669, #315, #2853, mozilla/standards-positions#105). The summary is that this is something we would like to do. But it requires time that we have not yet been able to invest.

tuxayo commented 1 year ago

Hi, thanks for the update @felixhandte

Still no time for this? Wouldn't the gain over brotli bring a bit better first loading experience of facebook's websites? Make the HTML, CSS, JS be ready a bit faster so it can display the text content while waiting for media content. (which should still take the same time)

daiaji commented 1 year ago

I think it is beneficial for everyone to extend zstd to HTTP/3. Since zstd can provide better compression ratio and speed than the existing scheme by using the appropriate dictionary, why not do it?

Moreover, the compression algorithm used in the existing HTTP protocol does lack more modernization, such as multi-core support and better dictionaries.

At this stage, there is really no compression algorithm more suitable for HTTP/3 than ZTSD.

gvollant commented 1 year ago

Somes people requested this feature at https://bugzilla.mozilla.org/show_bug.cgi?id=1301878

brunoais commented 1 year ago

At this stage, there is really no compression algorithm more suitable for HTTP/3 than ZTSD.

For dynamic content only. For static content brotli is still the best choice.

jefer94 commented 10 months ago

You should implement cache with Brotli for dynamic content and Zstandard for each other dynamic contents, I really wish to see Zstandard on Gecko, Webkit and Blink, do you have any idea when this will release for them, right now Zstandard was release on Chrome using Flags, so I can use it, but my clients can not

The results of implement Brotli before save the content in cache are very good, and it should reduce the response time x10, x15 (in heavy contents)

ekinnear commented 5 months ago

On our end, I think we'd be interested in incorporating a web-focused dictionary as we consider implementing zstandard support.

If we can drive even bigger wins for web content, that is an added bonus for the story for deploying zstandard in the browser (assuming there's a good way to indicate when the dictionary is in use, etc).