Chiang97912 / bet365.com

The scraper of bet365.com
MIT License
112 stars 48 forks source link

Pstk cookie issue #15

Closed dochenaj closed 3 years ago

dochenaj commented 3 years ago

I get the cookie the first time but on repeat runs of the script. It never receives that header.

And on such occasions when the cookie is not sent, the server redirects the request to localhost. I supposed, this was an IP blocking issue but it also happens with my local IP which can still access the website on the browser.

But the problem is more pronounced on my VPS that has a UK IP address.

I have done a lot of tweaking on the headers, yet to no avail.

HMaker commented 3 years ago

They return HTTP 400, 500 or 200 with empty response when headers are wrong. If that problem also happens with your local IP but you can access their website on the browser, then use the browser to inspect the network traffic and check if their API has changed.

dochenaj commented 3 years ago

@HMaker, I wonder how headers can be wrong. What do you suppose constitute a wrong header? My biggest challenge is that the problem is random, with the same headers it works for sometimes and doesn't work at other times.

HMaker commented 3 years ago

Wrong headers means missing headers or invalid values. I have never seen redirections due to wrong headers yet. Which endpoint you are acessing? I think you should show some code.

HMaker commented 3 years ago

Check if response.url is https://www.bet365.com/defaultapi/sports-configuration and response.status_code is 200. You can check Response.history to see if you are being redirected.

dochenaj commented 3 years ago

When it fails, response.url is http://localhost.

What worries me is that without changing the code in any way, it works sometimes and other times doesn't work. I can't figure out why this is happening. My guess is that it is probably a bet365 server issue.

Like I pointed out, the initial request returns a 302 response redirecting to http://localhost.com

HMaker commented 3 years ago

Do you also have that redirections to localhost on browser? If so, I think your IP address is blacklisted.

dochenaj commented 3 years ago

I thought that too. But if the IP was blacklisted, it shouldn't be able work at all.