search

gigobyte / HLTV

The unofficial HLTV Node.js API
MIT License
397 stars 107 forks source link

some useragents are permanently banned from cloudflare #450

Closed Revilum closed 3 years ago

Revilum commented 3 years ago

I had some issues where I would get banned by cloudflare way too often but i couldn't reproduce the issue. I heavily reduced the amount of requests but the issue was still occurring and the amount of requests being banned was the same. Even after some stress tests i found out that spamming one page still took a long time until I was banned so I had one final idea: Since the release of v3, random useragents are used getting data from hltv so this might be the cause of the issue as this is the only other random thing that could influence the random bans from cloudflare. After installing this addon for firefox and selecting 'Internet Explorer' 6.0 as my useragent. When visiting hltv I'll get the following error message: https://i.imgur.com/oFuw1nH.png After some research I found this article.

gigobyte commented 3 years ago

Wow, thanks for the info! That's really useful. I'll try to configure the random user agent library so that it doesn't return IE.

Revilum commented 3 years ago

Wow, thanks for the info! That's really useful. I'll try to configure the random user agent library so that it doesn't return IE.

it's not just IE. other versions work too so my guess is that there is a pattern on what is banned or it was randomly chosen by HLTV.

Revilum commented 3 years ago

After some testing I made this list of useragents not working. The only thing I could find out was the all IE v6 are banned so MSIE 6.0 in any way causes the issue. There may be some more troublesome agents because I managed to find 4 agents in 100 agents that were banned which way less than the api has this problem. In v2 the problem was occurring or at least not as bad so that might be a place to start.

gigobyte commented 3 years ago

I think it just blocks old useragents. I'll try to filter them out.