search

sayem314 / hooman

http interceptor to hoomanize cloudflare requests
https://www.npmjs.com/package/hooman
MIT License
148 stars 18 forks source link

Request returns 'Please enable javascript and reload the page' #21

Closed marcojacobsNL closed 4 years ago

marcojacobsNL commented 4 years ago

What would you like to discuss?

Hi there,

I'm currently trying to find a library able to scrape pages behind a cloudflare protection. I have a simple example set up in my code, but every time I run the code, it returns with:

Please turn JavaScript on and reload the page.

Here is my example code:

(async () => { try { const response = await hooman.get( "https://www.slamjam.com/en_BE/man/footwear/sneakers/low/nike-special-project/dunk-low-sp-sneakers/J188431.html" ); console.log(response.body); //=> '<!doctype html> ...' } catch (error) { console.log(error.response.body); //=> 'Internal server error ...' } })();

I can't seem to figure out how to fix this, I thought hooman would take care of the javascript challenge? Any help in the good direction is much appreciated!

Checklist

sayem314 commented 4 years ago

If a website require you to use javascript then it can't be bypassed with any nodejs-http based library. You should look into puppeteer.