403 when fetching rankings

[programerr01]

It appears that there's anti-bot protection enabled by cloudflare for /api/ranking/* page

[TISHARP]

Yes I've found this problem is for the most part resolved by Selenium. I haven't created a way to solve the Cloud Flair with Selenium, however using Selenium does help improve the fetching of the data making us able to capture 100% of the data (or close to it).

One down side of using Selenium is you have to install the chrome web-driver (or another web-driver) before you are able to use it...

For that reason I was thinking of having a secondary file for selenium use... What do you think about that?

[programerr01]

@TISHARP using selenium isn't very effective , atleast in this context. Selenium makes more sense in situation where there's dynamic data or complex dynamic html rendering on client side. This endpoint provides well formatted json response so would be better if direct http request could be made.

There are few libraries for bypassing cloudflare but so far i couldn't make them work with the specified endpoint

[rahulharpal1603]

https://github.com/baoliay2008/lccn_predictor

Hi, you can try to refer to the codebase of this repo. I think they have managed to bypass Cloudflare. They are using Python and crawler.

[programerr01]

Hii @rahulharpal1603 , i am not sure what libraries they have used but i personally used cloudscrape and it worked Repo