search

S1M0N38 / soccerapi

soccerapi, an unambitious soccer odds scraper ⚽️
MIT License
161 stars 36 forks source link

Problems scraping bet365 #41

Open DjarDjar opened 2 years ago

DjarDjar commented 2 years ago

So I did all the ReadME instructions (installed the docker and ran it) and everything ran smoothly until I tried to call the api.competitions() method. What I got as a result of that is just: {} I think this issue might be cloudflare related but I have no idea how to fix it. Is there anyone else facing the same problem? My code : 

from soccerapi.api import ApiBet365
api = ApiBet365()
competitions = api.competitions()
print(competitions)
DjarDjar commented 2 years ago

UPDATE So I cloned a local copy of the repository and tried to understand how the code works. Turns out it actually is cloudflare-related. So in the competitions function in the ApiBet365 class I changed the code to only return the table_country variable. Like this: https://prnt.sc/ktqM0kbE2OP9 And what I get as a response is: https://prnt.sc/_ka0Pgpo-OEB So the issue is definitely caused by some type of cloudflare detection. @S1M0N38 maybe you know what could've broken?

DjarDjar commented 2 years ago

UDATE 2 So looks like the problem is related to the x-net-sync-term. More specifically it looks like you can't have the same x-net-sync-term tokens for different requests. The method provided by the docker doesn't work because of that. Btw am I the only one experiencing this issue? I think If there was a way to check what headers a request has before sending it would easily fix the issue because that way we can detect whenever an x-net-sync term would be sent and just simply not send it but extract it and give it to the api.

Jmqcooper1 commented 11 months ago

Did you ever get this fixed?