search

mahendrakalkura / www.bet365.com

44 stars 16 forks source link

In-Play channel does not return full message. ['InPlay_20_0', 'LIInPlay_20_0'] #1

Closed xtonousou closed 6 years ago

xtonousou commented 6 years ago

I have these topics, ['InPlay_20_0', 'LIInPlay_20_0']

the output is

#P__time,S_988C94B9D5D74975BA863F73D0661746000003
sending message: '#\x03P\x01__time,S_988C94B9D5D74975BA863F73D0661746000003\x00'
received message: 100M33-kKvO5ztf3IYp
sending message: '\x16\x00InPlay_20_0\x01'
sending message: '\x16\x00LIInPlay_20_0\x01'
received message: __timeF|IN;TI=20171026161053631;UF=55;|
['__time', 'F|IN;TI=20171026161053631;UF=55;|']
received message: InPlay_20_0F|
['InPlay_20_0', 'F|']
received message: EMPTYF|IN;EM=1;TO=LIInPlay_20_0;|
['EMPTY', 'F|IN;EM=1;TO=LIInPlay_20_0;|']

as you can see in ['InPlay_20_0', 'F|'] the rest of the second part is missing, what am I doing wrong?

mahendrakalkura commented 6 years ago

How are you using this? Can you show me some sample code?

xtonousou commented 6 years ago 
#! /usr/bin/env python3.6

from requests import get
from requests import post
from random import choice
from ws4py.client.threadedclient import WebSocketClient
from wsaccel import patch_ws4py
from user_agents import get_random_user_agent

class WebSockets(WebSocketClient):

    _USER_AGENT = get_random_user_agent(u'mobile')
    _ODDS_SERVER_LIST = [u'1', u'2', u'3', u'11']
    _URL_CONNECTION = u'wss://premws-pt%s.365lpodds.com/zap/' % (
        choice(_ODDS_SERVER_LIST))
    _URL_SESSION_ID = u'https://www.bet365.com/?#/AS/B1/'

    _HEADERS = [
        (u'Sec-WebSocket-Extensions',
         u'permessage-deflate;client_max_window_bits'),
        (u'Sec-WebSocket-Protocol', u'zap-protocol-v1'),
        (u'Sec-WebSocket-Version', u'13'),
        (u'Connection', u'Upgrade'),
        (u'User-Agent', _USER_AGENT),
    ]

    _DELIMITERS_RECORD = u'\x01'
    _DELIMITERS_FIELD = u'\x02'
    _DELIMITERS_HANDSHAKE = u'\x03'
    _DELIMITERS_MESSAGE = u'\x08'

    _ENCODINGS_NONE = u'\x00'

    _TYPES_TOPIC_LOAD_MESSAGE = u'\x14'
    _TYPES_DELTA_MESSAGE = u'\x15'
    _TYPES_SUBSCRIBE = u'\x16'
    _TYPES_PING_CLIENT = u'\x19'
    _TYPES_TOPIC_STATUS_NOTIFICATION = u'\x23'

    _TOPICS = [
        'InPlay_20_0',
        'LIInPlay_20_0',
    ]

    _MESSAGES_SESSION_ID = u'%s%sP%s__time,S_%%s%s' % (
        _TYPES_TOPIC_STATUS_NOTIFICATION,
        _DELIMITERS_HANDSHAKE,
        _DELIMITERS_RECORD,
        _ENCODINGS_NONE,
    )

    _MESSAGES_SUBSCRIPTION = u'%s%s%%s%s' % (
        _TYPES_SUBSCRIBE,
        _ENCODINGS_NONE,
        _DELIMITERS_RECORD,
    )

    def __init__(self):
        super(WebSockets, self).__init__(
            self._URL_CONNECTION, headers=self._HEADERS)

    def connect(self):
        super(WebSockets, self).connect()

    def disconnect(self):
        print(u'closing connection...')

    def opened(self):
        session_id = self._fetch_session_id()
        if not session_id:
            self.disconnect()
            return
        message = self._MESSAGES_SESSION_ID % session_id
        print(message)
        self._send(message)

    def closed(self, code, reason=None):
        print(u'closed connection')
        print(u'code:', code)
        print(u'reason:', reason)

    def received_message(self, message):
        message = str(message)
        print(u'received message:', message)
        message = message.split(self._DELIMITERS_MESSAGE)
        while len(message):
            a = message.pop()
            b = a[0]
            if b == u'1':
                for topic in self._TOPICS:
                    m = self._MESSAGES_SUBSCRIPTION % topic
                    self._send(m)
                continue
            if b in [self._TYPES_TOPIC_LOAD_MESSAGE,
                     self._TYPES_DELTA_MESSAGE]:
                matches = a.split(self._DELIMITERS_RECORD)
                path_config = matches[0].split(self._DELIMITERS_FIELD)
                pair = path_config.pop()
                read_it_message = pair[1:]
                l = a[(len(matches[0]) + 1):]
                print([read_it_message, l])
                continue

    def _send(self, message):
        print(u'sending message:', repr(message))
        self.send(message)

    def _fetch_session_id(self):
        response = None
        try:
            response = get(self._URL_SESSION_ID)
        except Exception:
            pass
        if not response:
            return
        session_id = response.cookies[u'pstk']
        return session_id

if __name__ == u'__main__':
    try:
        patch_ws4py()
        web_sockets = WebSockets()
        web_sockets.connect()
        web_sockets.run_forever()
    except KeyboardInterrupt:
        web_sockets.disconnect()
pxpwoa commented 6 years ago

I met the same problem. EMPTY may mean that the server don't have this topic, but I did see the chorme sended the topic to the server. So I am confused about it.

mahendrakalkura commented 6 years ago

EMPTY may mean that the server don't have this topic

Correct.

chorme sended the topic to the server

Do you mean to say that you see reply to a topic in Chrome but not using this script?

pxpwoa commented 6 years ago

sorry,my problem is different. The connection was closed after the client sended topics to the server except 'CONFIG_10_0'.

I connected to (wss://premws-pt3.365lpodds.com/zap/) to get streaming data(web is 'https://www.365-777.com/#/IP/'). Finding in the chorme DevTools, chrome sends all topics in one string as '\x16\x00CONFIG_10_0,OVInPlay_10_0,Media_L10_Z0,XL_L10_Z0_C1_W1\x00'.Then, chrome receives messages about these topics one by one. I try to do it in my code. However, the client just received the message about CONFIG_10_0, then connection was closed.

I try to send only one topic one time, the conection was closed except sending 'CONFIG_10_0'.

I guess that no permission to read some topics cause the problem.

mahendrakalkura commented 6 years ago

You have the send the topics in a specific order. It can be different for each use-case. Copy it from what you see in Chrome.

xtonousou commented 6 years ago

Still the same, cannot receive full message, or the connection closes with the code 1006.

pxpwoa commented 6 years ago

Chrome requests two wss simultaneously. Here is what Chrome recorded (I made some modification) :

Request URL:wss://premws-pt3.365lpodds.com/zap/?uid=8347050473008863 14:39:25.544 #P__time,S_6D379FCE2BE3F1878EA86B037F047F48000003 sended from chorme 14:39:25.931 100M33-0UK6z8SxMrcy
14:39:25.933 CONFIG_10_0,OVInPlay_10_0,Media_L10_Z0,XL_L10_Z0_C1_W1 sended from chorme 14:39:25.933 __timeF|IN;TI=20171103074409624;UF=55;|
14:39:26.443 CONFIG_10_0F|CG;AM=0;AQ=0;CB=0;CM=1;CP=0;CW=1;DM=0..... 14:39:26.443 Media_L10_Z0F|EV;AU=0;C1=1;C2=35120382;CB=ASAUCKFJNCNZPFPGSB......
14:39:26.568 OVInPlay_10_0F|CL;CD=1;FF=;HM=1777#全场赛果^1778#下一个进球^10124#.....
14:39:27.171 6V68117822C1A_10_0 sended from chorme 14:39:27.495 6V68117822C1A_10_0F|EV;AM=;AS=;AU=0;C1=1;C2=35120382;C3=68362347;.....
..... no more send from chrome

Request URL:wss://pshudws.365lpodds.com/zap/?uid=631700528200908 14:39:25.542 #Ptime,S_6D379FCE2BE3F1878EA86B037F047F48000003 sended from chorme 14:39:25.921 100IRUD06-toQwMb5f19ZG
14:39:25.930 S_6D379FCE2BE3F1878EA86B037F047F48000003F|ER;RT=2;IT=S_6D379FCE2BE3F1878EA86B037F047F48000003;|
14:39:26.024 commandgetMessage6D379FCE2BE3F1878EA86B037F047F48000003SPTBK10 sended from chorme 14:39:26.442 __timeF|IN;TI=20171103074409859;UF=55;| 14:39:26.533 -11111_SPTBK_10_MSGF
14:39:28.949 timeU|IN;TI=20171103074412759;UF=55;| ..... no more send from chrome

xtonousou commented 6 years ago

I implemented this with two different .py scripts (two websocket connections) with the same user agent, session id and cookies. Still the same, i cannot receive in play frames at all, or i do not receive full message.

I forked the repo and added a new branch dev. Please take a look at it, I put example outputs for each case on README.md

xtonousou commented 6 years ago

@mahendrakalkura my problem is that i see the topics in chrome but i cant retrieve them using the script. Moreover, the error 1006 appears and closes the connection.

xtonousou commented 6 years ago

Did anyone solve this?

mahendrakalkura commented 6 years ago

as you can see in ['InPlay_20_0', 'F|'] the rest of the second part is missing, what am I doing wrong?

Fixed.

(
    u'received message:', 
    u'\x14LHInPlay_1_3\x01F|\x08\x14CONFIG_1_3\x01F|CG;AM=0;AQ=0;CB=0;CM=1;CP=0;CW=1;DM=0;DR=;EE=0;EM=0;EW=0;FC= 998,129,95,197,171,10,25,41,47,49,57,58,60,82,85,87,139,147,149,150,199,28,181,13,126,44,999;IF=1;IT=CONFIG_1_3;MB=0;MC=1;MP=0;MR=1;MT=0;OP=0;PB=1;RE=;SB=1;SS=0;WB=0;WC=1;WI=;WL=0;WR=1;WT=0;|'
)
xtonousou commented 6 years ago

Hello sir,

In your last comment, the content of the topic is still missing I still cannot receive the topics' content...

This is a proof of concept, using a remote connection (Slovakia)

opening connection...
opened connection
fetching session id...
(u'session id:', 'ED6B35CE9B0A4E0B8FC7E143238C90A4000003')
(u'sending message:', "u'#\\x03P\\x01__time,S_ED6B35CE9B0A4E0B8FC7E143238C90A4000003\\x00'")
(u'received message:', u'100\x02M11-LIkGQCIVBTrr\x00')
(u'sending message:', "u'\\x16\\x00__host\\x01'")
(u'sending message:', "u'\\x16\\x00CONFIG_1_3\\x01'")
(u'sending message:', "u'\\x16\\x00HL_L1_Z3_C1_W4\\x01'")
(u'sending message:', "u'\\x16\\x00HR_L1_Z3_C1_W4\\x01'")
(u'sending message:', "u'\\x16\\x00InPlay_1_3\\x01'")
(u'sending message:', "u'\\x16\\x00LHInPlay_1_3\\x01'")
(u'sending message:', "u'\\x16\\x00Media_l1_Z3\\x01'")
(u'sending message:', "u'\\x16\\x00OVInPlay_1_3\\x01'")
(u'sending message:', "u'\\x16\\x00XI_1_3\\x01'")
(u'sending message:', "u'\\x16\\x00XL_L1_Z3_C1_W4\\x01'")
(u'received message:', u'\x14__time\x01F|IN;TI=20171127010935691;UF=55;|')
[u'__time',
 u'F|IN;TI=20171127010935691;UF=55;|']
(u'received message:', u'\x14EMPTY\x01F|IN;EM=1;TO=__host;|\x08\x14CONFIG_1_3\x01F|CG;AM=0;AQ=0;CB=0;CM=1;CP=0;CW=1;DM=0;DR=;EE=0;EM=0;EW=0;FC= 998,129,95,197,171,10,25,41,47,49,57,58,60,82,85,87,139,147,149,150,199,28,181,13,126,44,999;IF=1;IT=CONFIG_1_3;MB=0;MC=1;MP=0;MR=1;MT=0;OP=0;PB=1;RE=;SB=1;SS=0;WB=0;WC=1;WI=;WL=0;WR=1;WT=0;|')
[u'CONFIG_1_3',
 u'F|CG;AM=0;AQ=0;CB=0;CM=1;CP=0;CW=1;DM=0;DR=;EE=0;EM=0;EW=0;FC= 998,129,95,197,171,10,25,41,47,49,57,58,60,82,85,87,139,147,149,150,199,28,181,13,126,44,999;IF=1;IT=CONFIG_1_3;MB=0;MC=1;MP=0;MR=1;MT=0;OP=0;PB=1;RE=;SB=1;SS=0;WB=0;WC=1;WI=;WL=0;WR=1;WT=0;|']
[u'EMPTY',
 u'F|IN;EM=1;TO=__host;|']
closed connection
(u'code:', 1006)
(u'reason:', 'Going away')

I tried tornado and gevent clients, the results are almost the same.

mahendrakalkura commented 6 years ago

Can you try and confirm via a VPN from another country?

xtonousou commented 6 years ago

Well I also tried here in Greece, but with the gr tld. It's very difficult to find working VPN for bet365.com

xtonousou commented 6 years ago

Tested with Python2.7 and Python3.6.

mahendrakalkura commented 6 years ago

Can you try it on a cloud server (AWS or GCP) in order to bypass your country-level firewall?

I am sorry. I don't know how to help you otherwise.

pxpwoa commented 6 years ago

I can't parsing websocket messages about inpaly animation. Someone can tell me which part of websocket messages of 365bet inplay channel transmits the information of the inpaly animatiom.

mahendrakalkura commented 6 years ago

That would be InPlay_*.

pxpwoa commented 6 years ago

Sorry, I did't say it clearly. As we konw, 365bet use abbreviation system to declare which type the information is in the websocket messages. Like 'OD=4/5', OD means odds. In 365bet inplay web, there is a window to show matchlive animatiomin in which we can see the possession exchanging, freethrow missing in basketball game. But, I can't find any message about the animatiomin in the websocket.

mahendrakalkura commented 6 years ago

You should be able to find it in their JS codes.

Example (unminified code): https://gist.github.com/mahendrakalkura/77159876de7d56157fc413e3b5215bcd (see line #20613).

finanshow commented 4 years ago

Sorry, I did't say it clearly. As we konw, 365bet use abbreviation system to declare which type the information is in the websocket messages. Like 'OD=4/5', OD means odds. In 365bet inplay web, there is a window to show matchlive animatiomin in which we can see the possession exchanging, freethrow missing in basketball game. But, I can't find any message about the animatiomin in the websocket.

Have you ever figure out how to get those in-Play Events message from websocket?

mahendrakalkura commented 4 years ago

No, not yet.

victorratts13 commented 3 years ago

hey guys, see my project https://github.com/victorratts13/pulldata-bet-api

victorratts13 commented 3 years ago

how i can convert this to the XML file? Screen Capture_select-area_20200814010824