search

S1M0N38 / soccerapi

soccerapi, an unambitious soccer odds scraper ⚽️
MIT License
154 stars 37 forks source link

Problems scraping bet365 #41

Open DjarDjar opened 2 years ago

DjarDjar commented 2 years ago

So I did all the ReadME instructions (installed the docker and ran it) and everything ran smoothly until I tried to call the api.competitions() method. What I got as a result of that is just: {} I think this issue might be cloudflare related but I have no idea how to fix it. Is there anyone else facing the same problem? My code : 

from soccerapi.api import ApiBet365
api = ApiBet365()
competitions = api.competitions()
print(competitions)
DjarDjar commented 2 years ago

UPDATE So I cloned a local copy of the repository and tried to understand how the code works. Turns out it actually is cloudflare-related. So in the competitions function in the ApiBet365 class I changed the code to only return the table_country variable. Like this: https://prnt.sc/ktqM0kbE2OP9 And what I get as a response is: https://prnt.sc/_ka0Pgpo-OEB So the issue is definitely caused by some type of cloudflare detection. @S1M0N38 maybe you know what could've broken?

DjarDjar commented 2 years ago

UDATE 2 So looks like the problem is related to the x-net-sync-term. More specifically it looks like you can't have the same x-net-sync-term tokens for different requests. The method provided by the docker doesn't work because of that. Btw am I the only one experiencing this issue? I think If there was a way to check what headers a request has before sending it would easily fix the issue because that way we can detect whenever an x-net-sync term would be sent and just simply not send it but extract it and give it to the api.

Jmqcooper1 commented 6 months ago

Did you ever get this fixed?