search

apify / got-scraping

HTTP client made for scraping based on got.
422 stars 32 forks source link

Possible to set timeout for ALPN in browser-headers hook #130

Closed rasmushorndrup closed 4 months ago

rasmushorndrup commented 4 months ago

Currently we're experiencing issues where the resolveProtocol promise never resolves in the browser-headers hook, due to no timeout being set. Should it reuse the timeout set in the timeout.lookup in request options?

barjin commented 4 months ago

Hello and thank you for your interest in this package!

Could you please share the URL where you experienced this / some reproduction scenario? All parts of the request processing should indeed respect the respective timeouts, so this is a potential issue - but we would still want to see it happening before we try to fix it :)

Cheers!

rasmushorndrup commented 4 months ago

A site like https://www.cyclebrother.com should reproduce the error, where the server does not respond.

Setting a request timeout or lookup timeout, is not respected in the browser-headers hook

await gotScraping.get({ url: "https://www.cyclebrother.com", timeout: { request: 5000, lookup: 2000 } })

Got around it by just setting our own resolveProtocol in the options object