apify / got-scraping

HTTP client made for scraping based on got.
422 stars 32 forks source link

Slower compared to got and fetch #84

Closed VoidMonk closed 11 months ago

VoidMonk commented 1 year ago

Hi, I'm trying got-scraping for a project, and noticed that it's almost 2x slower compared to got and native fetch.

Here's some basic test code:

import { got, gotScraping } from 'got-scraping';

const url = 'https://httpbin.org/html';
let t, r;

t = performance.now();
r = await fetch(url);
console.log(`fetch took ${performance.now() - t}ms`);

t = performance.now();
r = await got(url);
console.log(`got took ${performance.now() - t}ms`, JSON.stringify(r.timings.phases));

t = performance.now();
r = await gotScraping(url);
console.log(`gotScraping took ${performance.now() - t}ms`, JSON.stringify(r.timings.phases));

Results (similar timing difference on multiple runs, even with other URLs):

fetch took 987.2981999991462ms

got took 960.8908999999985ms {"wait":2,"dns":2,"tcp":234,"tls":477,"request":1,"firstByte":235,"download":3,"total":954}

gotScraping took 1676.9162999996915ms {"wait":719,"request":null,"firstByte":239,"download":2,"total":960}

My test machine is running Windows 11 and Node.js 18.12.1.

Why is got-scraping is slower, if not any faster, and can it be improved?