Bloomberg detects you

[Kikobeats]

Example URL

https://www.bloomberg.com/news/articles/2019-01-15/here-are-five-volatility-charts-keeping-wall-street-up-at-night

When you enter performing the request using puppeteer the website automatically redirects you to Terms of Service forcing you to verify you are a human with captcha checkbox.

[NikolaiT]

does somebody have an idea how to prevent this?

does bloomberg track mouse movement?

[franz101]

@NikolaiT From a short inspection they use perimeterx for bot detection.

I was able to fetch the document with the following header: let url = 'https://www.bloomberg.com/news/articles/2019-01-15/here-are-five-volatility-charts-keeping-wall-street-up-at-night' var headers = { 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0.3 Safari/605.1.15', 'Accept-Language': 'en-us', 'Accept-Encoding': 'br, gzip, deflate', 'Connection': 'keep-alive' } let data = await fetch(url,{ method: 'GET', headers: headers}) let body = await data.text()

As you can see it works by simply changing the user agent to: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0.3 Safari/605.1.15

@paulirish let me know if that works.

[Kikobeats]

I tested your code snippet and it doesn't work to me 🤔

[momala454]

it detect the IP