feat: Implementation of decompression in fetch()

nabetti1720 commented 1 month ago

Issue #355 (if available)

Description of changes

Implementation of decompression in fetch() Supported Encodings: gzip, deflate, br, zstd

Execution Sample

content-encoding.js:

const main = async () => {
  try {
    const response = await fetch('https://www.meta.com/', {headers: {"accept-encoding": process.argv[2]}});
    console.log(response.headers.get('content-encoding'));
    const responseText = await response.text();
    console.log('vvv');
    console.log(responseText.substring(0, 100));
    console.log('^^^');
  } catch(e) {
    console.log(e);
  }
}

main();

accept-encoding: zstd

shinya@MBA2022M2 llrt-test % ./llrt content-encoding.js zstd   
zstd
vvv
<!DOCTYPE html><html id="facebook" class="_a4ch _aber" lang="ja" dir="ltr"><head><link data-default-
^^^

accept-encoding: br

shinya@MBA2022M2 llrt-test % ./llrt content-encoding.js br  
br
vvv
<!DOCTYPE html><html id="facebook" class="_a4ch _aber" lang="ja" dir="ltr"><head><link data-default-
^^^

accept-encoding: gzip

shinya@MBA2022M2 llrt-test % ./llrt content-encoding.js gzip
gzip
vvv
<!DOCTYPE html><html id="facebook" class="_a4ch _aber" lang="ja" dir="ltr"><head><link data-default-
^^^

accept-encoding: deflate

shinya@MBA2022M2 llrt-test % ./llrt content-encoding.js deflate
undefined       // NOTE: This website does not support deflate.
vvv
<!DOCTYPE html><html id="facebook" class="_a4ch _aber" lang="ja" dir="ltr"><head><link data-default-
^^^

accept-encoding: gzip, deflate, br, zstd

shinya@MBA2022M2 llrt-test % ./llrt content-encoding.js "gzip, deflate, br, zstd"
zstd            // NOTE: The encoding is determined on the server side.
vvv
<!DOCTYPE html><html id="facebook" class="_a4ch _aber" lang="ja" dir="ltr"><head><link data-default-
^^^

ToDo

[x] Add test code. (Response.text() or response.json() does not work well in Mock Server testing...)
~~[ ] Verify deflate decompression on the actual website. (I would like to know if anyone has found it anywhere. :)~~
[x] Default setting for accept-encoding. (Is there a specification that cannot be overridden by a header method when Request::builder() is executed? Also, what about the encoding priority order?)

Checklist

[x] Created unit tests in tests/unit and/or in Rust for my feature if needed
[x] Ran make fix to format JS and apply Clippy auto fixes
[x] Made sure my code didn't add any additional warnings: make check
[x] Updated documentation if needed (API.md/README.md/Other)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

nabetti1720 commented 1 month ago

I suggest to encode (or hardcode) gzip, deflate, br and zstd "hello world" bytes and read the response as plain texts. Ideally test in Rust and not JS so we can mock the http responses.

Yes, I added the test that way. One of the remaining ToDo, checking deflate decompression on the actual website, can be substituted for this test if it seems difficult.

The remaining issue is the default accept-encoding and pre-allocating vectors.

richarddavison commented 1 month ago

Yes, I added the test that way. One of the remaining ToDo, checking deflate decompression on the actual website, can be substituted for this test if it seems difficult.

It's not ideal to have CI depend on external resources. These tests should suffice.

nabetti1720 commented 1 month ago

It's not ideal to have CI depend on external resources. These tests should suffice.

Sorry for the misunderstanding. We did not intend to incorporate access to external resources into the CI test, we just wanted to see how it worked on a real website. :)

richarddavison commented 1 month ago

@nabetti1720 please rebase from main. I have a minor changed CI

nabetti1720 commented 1 month ago

@nabetti1720 please rebase from main. I have a minor changed CI

I ran it as soon as possible.

nabetti1720 commented 1 month ago

Hi @richarddavison, sorry for the delay. All issues were addressed.

awslabs / llrt