Is it possible to change the proxyUrl with the beforeRequest hook?

[szmarczak]

Hey!

First off all thanks for this amazing package!

As the title suggests, is there any way to set the proxyUrl through the beforeRequest hook? Ideally I'd like to do this on the beforeRetry hook but we cannot access the options object from there.

It should sort of act like a proxy rotation system - if the request fails, call a function which returns a new proxy and set it as the new proxyUrl.

Thanks!

Originally posted by @PhilBookst in https://github.com/apify/got-scraping/discussions/76

[szmarczak]

It should be possible to do that, I'll add a test for this. You should be able to override error.options.context.proxyUrl.

[VoidMonk]

Any way to remove the set proxy before a retry?

Setting error.options.context.proxyUrl to undefined, null or an empty string in the beforeRetry hook still makes a retry request with the originally set proxyUrl.

@szmarczak We'll appreciate any suggestions or unit test results from you. Thanks.

[yovanoc]

Would like to know this too, thanks

[yovanoc]

as I understand, if we extends the got-scraping client our hooks will be called after got-scraping ones, so the proxy hook is called before we can change the proxyUrl