I forked this some time ago (at https://github.com/RhinoSecurityLabs/CloudScraper) because I was running into issues on the latest commit here. I happened to check my fork just now and noticed some people have been using it and have even got a PR to add more domains to check.
I'd rather not maintain a fork though so wanted to push the changes upstream if possible.
Resolve #9
Add --no-verify flag for skipping TLS verification.
Sites with broken TLS can still be useful so failing in these cases isn't always beneficial (I suspect most the time it's not, but didn't want to make this assumption so throwing this behind a flag for now).
Always strip extra whitespace from the URL.
Previously this only stripped extra chars on the right when the link did not contain http which (iirc) caused issues in some cases.
Update user agent.
I don't remember what the issue here was but iirc some site was acting up with the user agent that was set before.
I forked this some time ago (at https://github.com/RhinoSecurityLabs/CloudScraper) because I was running into issues on the latest commit here. I happened to check my fork just now and noticed some people have been using it and have even got a PR to add more domains to check.
I'd rather not maintain a fork though so wanted to push the changes upstream if possible.
httpwhich (iirc) caused issues in some cases.