search

awslabs / llrt

LLRT (Low Latency Runtime) is an experimental, lightweight JavaScript runtime designed to address the growing demand for fast and efficient Serverless applications.
Apache License 2.0
8.04k stars 355 forks source link

response.text() does not support content-encoding #355

Closed nabetti1720 closed 4 months ago

nabetti1720 commented 5 months ago

When the following code is executed, the result of response.text() is garbled. The same result is obtained when accept-encoding is set to something else(such as br, deflate, zstd).

And although I have not tried it, I suspect that response.json() will produce the same result.

// reproduction.js
const main = async () => {
  try {
    const response = await fetch('https://google.com', {headers: {'accept-encoding': 'gzip'}});
    console.log(response.headers.get('content-encoding'));
    const responseText = await response.text();
    console.log('vvv');
    console.log(responseText.substring(0, 100));
    console.log('^^^');
  } catch(e) {
    console.log(e);
  }
}

main();

result of llrt:

shinya@MBA2022M2 llrt-test % ./llrt reproduction.js
gzip
vvv
// This part is actually garbled.
^^^

result of bun:

shinya@MBA2022M2 llrt-test % bun reproduction.js
gzip
vvv
<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="ja"><head><meta content
^^^
richarddavison commented 5 months ago

This is a must have. We already heavily rely on zstd for reading LLRT bytecode format. Gzip and deflate (zip) decompression can be added using flate2 crate with zlib-ng backend. Compression & decompression are also requirements of WinterCG compression stream with said algorithms. For now (since streaming is not yet in place), decompression can be done when reading from response here: https://github.com/awslabs/llrt/blob/8fa91565bd7446eca010bb93aac336a0fe3ba0ac/llrt_core/src/modules/http/response.rs#L135

nabetti1720 commented 5 months ago

I know that many sites return content-encoding as br when accept-encoding is "gzip, deflate, br, zstd", but do you plan to support brotli?

richarddavison commented 5 months ago

I know that many sites return content-encoding as br when accept-encoding is "gzip, deflate, br, zstd", but do you plan to support brotli?

Yes brotli as well

nabetti1720 commented 5 months ago

After this implementation is complete, We also need to decide on the default accept-encoding.

For example,

Personally, I think that within the scope of WinterCG, challenging settings that lead to improved performance could be incorporated.

nabetti1720 commented 4 months ago

Hi @richarddavison, are you already working on this issue? If you haven't already, would you give me the opportunity to work on it?

richarddavison commented 4 months ago

Hi @richarddavison, are you already working on this issue? If you haven't already, would you give me the opportunity to work on it?

Hi @nabetti1720! I'm not, please go ahead 👌

richarddavison commented 4 months ago

Landed in https://github.com/awslabs/llrt/pull/360