search

sayem314 / hooman

http interceptor to hoomanize cloudflare requests
https://www.npmjs.com/package/hooman
MIT License
146 stars 18 forks source link

Response code 403 (Forbidden) #7

Open wiktorpikosz opened 4 years ago

wiktorpikosz commented 4 years ago

Hey,

I used codemanki/cloudscraper but author decided to archived his project and I looking for a alternative.

I tracked a lot of websites and cloudscraper usually works fine but for one website is a problem. But when the project was archived, it isn't good prognostic for future. So I tested your solution.

When I try load it: https://vitals.com - I get error:

{ "name": "HTTPError", "timings": { "start": 1589285452689, "socket": 1589285453194, "lookup": 1589285453195, "connect": 1589285453195, "secureConnect": 1589285453195, "upload": 1589285453197, "response": 1589285453359, "end": 1589285453395, "phases": { "wait": 505, "dns": 1, "tcp": 0, "tls": 0, "request": 2, "firstByte": 162, "download": 36, "total": 706 } } }

In the stack is: HTTPError: Response code 403 (Forbidden) at PromisableRequest.request.once (/root/cloudflare-bot/node_modules/got/dist/source/as-promise/index.js:124:28) at process._tickCallback (internal/process/next_tick.js:68:7)

Could you tell me, if your library should work fine for this example and I'm doing wrong something or your library don't support it.

sayem314 commented 4 years ago

I can't even use that site in the browser. image

wiktorpikosz commented 4 years ago

Thanks for your fast answer!

I don't have this problem: image

Maybe you have a ban, because has tested something on your localhost :D

You can change IP? VPN?

sayem314 commented 4 years ago

Yes, VPN worked. Their site is very well protected, I will try to make it work but no guarantee. You can always use puppeteer tho.

wiktorpikosz commented 4 years ago

Ok, thanks!

Yeah, but pupeeteer uses headless and needs instance of chromium if I understand their description. It's barier for me.

msjonker commented 4 years ago

I've run into the same error with a site that I am scraping, and the way I got around it was to enable http2 in the got request.