RequestError: unknown compression method

[metalwarrior665]

Describe the bug This website throws RequestError: unknown compression method every time. Not sure if this should be fixed in libraries or it is just an edge case. It works with Puppeteer.

To Reproduce

const startUrls = [{ url: 'https://www.tripplite.com/products/display-mount-accessories-dual-bar-mounts~56-203'}]
const requestList = await Apify.openRequestList('start-urls', startUrls);

const crawler = new Apify.CheerioCrawler({
    requestList,
    handlePageFunction: async (context) => {
        const { url, userData: { label } } = context.request;
        log.info('Page opened.', { label, url });
    },
});

log.info('Starting the crawl.');
await crawler.run();

Expected behavior A clear and concise description of what you expected to happen.

System information: {"apifyVersion":"2.1.0","apifyClientVersion":"2.0.2","osType":"Linux","nodeVersion":"v16.13.0"}

Additional context Sample run - https://console.apify.com/view/runs/le9zcQFmeteY00dnR

[B4nan]

cc @szmarczak

[szmarczak]

require('axios')('https://www.tripplite.com/products/display-mount-accessories-dual-bar-mounts~56-203', { decompress: true, headers: { 'accept-encoding': 'deflate' } }).then(x => x.headers).then(console.log)

I tried axios and the same happens. Not sure how browsers decode this.

[szmarczak]

Ok got it, it's missing zlib headers. inflateRaw works like a charm, but this (99%) won't be supported by Got. I guess we could retry with decompress: false?

[mnmkng]

Can we somehow integrate this retry into got-scraping? Those decompression errors are not frequent, but common. E.g. https://github.com/apify/apify-js/issues/373 https://github.com/apify/apify-js/issues/266

[szmarczak]

https://github.com/apify/got-scraping/pull/63

[B4nan]

Should be addressed in latest got-scraping via https://github.com/apify/got-scraping/pull/64