Chiang97912 / bet365.com

The scraper of bet365.com
MIT License
112 stars 48 forks source link

Unable to authenticate #9

Open otac0074 opened 4 years ago

otac0074 commented 4 years ago

i think this problem

bet365 update..

스크린샷 2020-03-03 오전 12 34 30

P + session_id + '????'

added

tedmax100 commented 4 years ago

i was also unable to authenticate too.

otac0074 commented 4 years ago

i was also unable to authenticate too.

How do you generate that value?

mjgerace commented 4 years ago

I started having this issue today as well.

Here is the new source code, still cannot figure out where that value is coming in: https://gist.github.com/mjgerace/ae144c6bcf434c68cf85f1b686d3ce83

otac0074 commented 4 years ago

I started having this issue today as well.

Here is the new source code, still cannot figure out where that value is coming in: https://gist.github.com/mjgerace/ae144c6bcf434c68cf85f1b686d3ce83

i think keyworld 'NST' or 'token'

mjgerace commented 4 years ago

I would agree, although I cannot figure out what this means in the context of that file. nodecrypto also seems very interesting. Perhaps related to https://nodejs.org/api/crypto.html#crypto_diffiehellman_generatekeys_encoding

otac0074 commented 4 years ago

I would agree, although I cannot figure out what this means in the context of that file. nodecrypto also seems very interesting.

I was trying

if you knows this problem

tell to me,

me to

스크린샷 2020-03-03 오후 2 40 46

and boot.nst what is mean?

otac0074 commented 4 years ago

i find you file

스크린샷 2020-03-03 오후 2 42 37 스크린샷 2020-03-03 오후 2 43 10

but nothing 'h' functions

mjgerace commented 4 years ago

I am investigating the 'h' business now. This is really interesting, as is your find. What is the name of that file?

otac0074 commented 4 years ago

I am investigating the 'h' business now. This is really interesting, as is your find. What is the name of that file?

just bet365 mainpage source

no another file

otac0074 commented 4 years ago
스크린샷 2020-03-03 오후 2 50 58
mjgerace commented 4 years ago

I am stumped, will revisit this tomorrow.

Smart828 commented 4 years ago

Any thoughts? Since last night, I have been trying to figure out how they can form this parameter.

mjgerace commented 4 years ago

I'm gunna look at it later

otac0074 commented 4 years ago

difficult..

lucifer-v commented 4 years ago

I figure out what is the '???' in "P + session_id + '????'"。 First you can find a B365SimpleEncrypt.decrypt() in bet365 source code as follow: 图片

And, you get two string about nstToken from response of "https://www.288-365.com"。As show in picture. 图片

Then, you bind two string with dot , after that, use B365SimpleEncrypt.decrypt() function decrypt the string( PHP Code I use), as follow: 图片

you will calculate "????", which is the surffix value of handshake message of wss of "wss://premws-pt3.365lpodds.com/zap/"。 图片

And I also find out in request "wss://pshudws.365lpodds.com/zap/", you may get another string looks like the nstToken in request "wss://premws-pt3.365lpodds.com/zap/"。 图片

If you work on the string with B365SimpleEncrypt.decrypt(), you will get the result of upstream "command" message in request "wss://premws-pt3.365lpodds.com/zap/". 图片

But, even I have done this all, I can not receive valid match data. Do you have any more brain holes?

mjgerace commented 4 years ago

@lucifer-v

You have made it further than I did. I did the regex part like this in python. Please note d_value here is NOT ??? by your logic, I am not running decrypt on it.

def get_d_value(user_agent) -> str:
    headers = {
        'Cookie': SESSION_COOKIE_HEADER,
        'User-Agent': user_agent
    }

    req = requests.get(url=HOME_PAGE, headers=headers)
    regex = r'boot&&boot.nst&&boot.nst\((.*)\)'
    matches = re.search(regex, req.text)

    if not matches:
        raise Exception(
            'Session', 'Did not find D value.')

    group = matches.group(0).split("\"")
    d_value = "{}.{}".format(group[1], group[3])
    print(d_value)
    return d_value

To get the nstToken (pre-decryption), but you are drawing important correlations between the values. My initial assumption was that this value was the D value, but after seeing your response, it is clear it is not.

Also worth noting: boot.nst value changes dynamically without refreshing the page. Just view the source a couple of times and see it continuously change.

lucifer-v commented 4 years ago

@mjgerace In one refresh, the D_value can be calculated by the value find in the response of "www.288-365.com" passed into B365SimpleEncrypt.decrypt(). you can have a try.

You know: "boot.nst( "str1", "str2" )" Dvalue (without "D") = B365SimpleEncrypt.decrypt( "str1"+"."+"str2" )

mjgerace commented 4 years ago

@lucifer-v

I was just pointing out that the value seems to change repeatedly. IE, the nst ("str1", "str2") changes if you inspect the page source without refreshing (dynamically).

lucifer-v commented 4 years ago

@mjgerace Yes it will change when you open the page source. you can also inspect the value in this "Network " window, in "response" Tab, the value is fixed, and which is looks like the value used in the whole wss connection 图片

And I paste the code of B365SimpleEncrypt, you can transfer it to python: (function(e) { var t = (function() { function e() {} return e.encrypt = function(t) { var n, i = "", o = t.length, r = 0, s = 0; for (r = 0; o > r; r++) { for (n = t.substr(r, 1), s = 0; s < e.MAP_LEN; s++) if (n == e.charMap[s][0]) { n = e.charMap[s][1]; break } i += n } return i } , e.decrypt = function(t) { var n, i = "", o = t.length, r = 0, s = 0; for (r = 0; o > r; r++) { for (n = t.substr(r, 1), s = 0; s < e.MAP_LEN; s++) { if (":" == n && ":|~" == t.substr(r, 3)) { n = "\n", r += 2; break } if (n == e.charMap[s][1]) { n = e.charMap[s][0]; break } } i += n } return i } , e.MAP_LEN = 64, e.charMap = [["A", "d"], ["B", "e"], ["C", "f"], ["D", "g"], ["E", "h"], ["F", "i"], ["G", "j"], ["H", "k"], ["I", "l"], ["J", "m"], ["K", "n"], ["L", "o"], ["M", "p"], ["N", "q"], ["O", "r"], ["P", "s"], ["Q", "t"], ["R", "u"], ["S", "v"], ["T", "w"], ["U", "x"], ["V", "y"], ["W", "z"], ["X", "a"], ["Y", "b"], ["Z", "c"], ["a", "Q"], ["b", "R"], ["c", "S"], ["d", "T"], ["e", "U"], ["f", "V"], ["g", "W"], ["h", "X"], ["i", "Y"], ["j", "Z"], ["k", "A"], ["l", "B"], ["m", "C"], ["n", "D"], ["o", "E"], ["p", "F"], ["q", "0"], ["r", "1"], ["s", "2"], ["t", "3"], ["u", "4"], ["v", "5"], ["w", "6"], ["x", "7"], ["y", "8"], ["z", "9"], ["0", "G"], ["1", "H"], ["2", "I"], ["3", "J"], ["4", "K"], ["5", "L"], ["6", "M"], ["7", "N"], ["8", "O"], ["9", "P"], ["\n", ":|~"], ["\r", ""]], e } )(); e.B365SimpleEncrypt = t }

When you get any progress, please inform us. thanks

otac0074 commented 4 years ago

@mjgerace Yes it will change when you open the page source. you can also inspect the value in this "Network " window, in "response" Tab, the value is fixed, and which is looks like the value used in the whole wss connection 图片

And I paste the code of B365SimpleEncrypt, you can transfer it to python: (function(e) { var t = (function() { function e() {} return e.encrypt = function(t) { var n, i = "", o = t.length, r = 0, s = 0; for (r = 0; o > r; r++) { for (n = t.substr(r, 1), s = 0; s < e.MAP_LEN; s++) if (n == e.charMap[s][0]) { n = e.charMap[s][1]; break } i += n } return i } , e.decrypt = function(t) { var n, i = "", o = t.length, r = 0, s = 0; for (r = 0; o > r; r++) { for (n = t.substr(r, 1), s = 0; s < e.MAP_LEN; s++) { if (":" == n && ":|~" == t.substr(r, 3)) { n = "\n", r += 2; break } if (n == e.charMap[s][1]) { n = e.charMap[s][0]; break } } i += n } return i } , e.MAP_LEN = 64, e.charMap = [["A", "d"], ["B", "e"], ["C", "f"], ["D", "g"], ["E", "h"], ["F", "i"], ["G", "j"], ["H", "k"], ["I", "l"], ["J", "m"], ["K", "n"], ["L", "o"], ["M", "p"], ["N", "q"], ["O", "r"], ["P", "s"], ["Q", "t"], ["R", "u"], ["S", "v"], ["T", "w"], ["U", "x"], ["V", "y"], ["W", "z"], ["X", "a"], ["Y", "b"], ["Z", "c"], ["a", "Q"], ["b", "R"], ["c", "S"], ["d", "T"], ["e", "U"], ["f", "V"], ["g", "W"], ["h", "X"], ["i", "Y"], ["j", "Z"], ["k", "A"], ["l", "B"], ["m", "C"], ["n", "D"], ["o", "E"], ["p", "F"], ["q", "0"], ["r", "1"], ["s", "2"], ["t", "3"], ["u", "4"], ["v", "5"], ["w", "6"], ["x", "7"], ["y", "8"], ["z", "9"], ["0", "G"], ["1", "H"], ["2", "I"], ["3", "J"], ["4", "K"], ["5", "L"], ["6", "M"], ["7", "N"], ["8", "O"], ["9", "P"], ["\n", ":|~"], ["\r", ""]], e } )(); e.B365SimpleEncrypt = t }

When you get any progress, please inform us. thanks

php b365simpleEncrypt source

can i get that?

elpaxel commented 4 years ago

They changed boot.nst[] to two variables var order=['WJcVaW=='] and var loadingflags=['ES1RgM8F13q8jNKI1ryxD2+AxGKXzFGQJLF0Odaeaet=']

lucifer-v commented 4 years ago

@otac0074 PHP version, you change ext name to "php" by yourself.

Bet365SimpleEncrypt.txt

some samples as follow: // command // commandnstFkBeXg==.4iswQ11b6Xlqbc5WxU3AX3cxm+iGCmDQ8AE4wr/5Buo=SPTBK $val = "iAeUaW==.KY26tHHRMaB0RSLz7xJdaJS7C+YjfCgtOdhK61/Le4E="; $val = "eEPUaW==.pXXEoxBRljjF16VK3FcrXbqDLECBWOgWoStFH7GG4rd="; $val = "HmyUaW==.UnOZs53sCPWceRF0q+5COVUe1mjj1/sbkW5gD3dWReb="; var_dump( Bet365SimpleEncrypt::decrypt($val) );

teocns commented 4 years ago
def decryptToken(t):
    n = ""
    i = ""
    o = len(t)
    r = 0
    s = 0
    MAP_LEN = 64
    charMap = [["A", "d"], ["B", "e"], ["C", "f"], ["D", "g"], ["E", "h"], ["F", "i"], ["G", "j"], ["H", "k"], ["I", "l"], ["J", "m"], ["K", "n"], ["L", "o"], ["M", "p"], ["N", "q"], ["O", "r"], ["P", "s"], ["Q", "t"], ["R", "u"], ["S", "v"], ["T", "w"], ["U", "x"], ["V", "y"], ["W", "z"], ["X", "a"], ["Y", "b"], ["Z", "c"], ["a", "Q"], ["b", "R"], ["c", "S"], ["d", "T"], ["e", "U"], ["f", "V"], [
        "g", "W"], ["h", "X"], ["i", "Y"], ["j", "Z"], ["k", "A"], ["l", "B"], ["m", "C"], ["n", "D"], ["o", "E"], ["p", "F"], ["q", "0"], ["r", "1"], ["s", "2"], ["t", "3"], ["u", "4"], ["v", "5"], ["w", "6"], ["x", "7"], ["y", "8"], ["z", "9"], ["0", "G"], ["1", "H"], ["2", "I"], ["3", "J"], ["4", "K"], ["5", "L"], ["6", "M"], ["7", "N"], ["8", "O"], ["9", "P"], ["\n", ":|~"], ["\r", ""]]
    for r in range(0, o):
        n = t[r]
        for s in range(0, MAP_LEN):
            if ":" == n and ":|~" == t[r:3]:
                n = "\n"
                r = r + 2
                break
            if n == charMap[s][1]:
                n = charMap[s][0]
                break
        i = i+n
    return i

This is the Python version for the decrypt function, guys

otac0074 commented 4 years ago
def decryptToken(t):
    n = ""
    i = ""
    o = len(t)
    r = 0
    s = 0
    MAP_LEN = 64
    charMap = [["A", "d"], ["B", "e"], ["C", "f"], ["D", "g"], ["E", "h"], ["F", "i"], ["G", "j"], ["H", "k"], ["I", "l"], ["J", "m"], ["K", "n"], ["L", "o"], ["M", "p"], ["N", "q"], ["O", "r"], ["P", "s"], ["Q", "t"], ["R", "u"], ["S", "v"], ["T", "w"], ["U", "x"], ["V", "y"], ["W", "z"], ["X", "a"], ["Y", "b"], ["Z", "c"], ["a", "Q"], ["b", "R"], ["c", "S"], ["d", "T"], ["e", "U"], ["f", "V"], [
        "g", "W"], ["h", "X"], ["i", "Y"], ["j", "Z"], ["k", "A"], ["l", "B"], ["m", "C"], ["n", "D"], ["o", "E"], ["p", "F"], ["q", "0"], ["r", "1"], ["s", "2"], ["t", "3"], ["u", "4"], ["v", "5"], ["w", "6"], ["x", "7"], ["y", "8"], ["z", "9"], ["0", "G"], ["1", "H"], ["2", "I"], ["3", "J"], ["4", "K"], ["5", "L"], ["6", "M"], ["7", "N"], ["8", "O"], ["9", "P"], ["\n", ":|~"], ["\r", ""]]
    for r in range(0, o):
        n = t[r]
        for s in range(0, MAP_LEN):
            if ":" == n and ":|~" == t[r:3]:
                n = "\n"
                r = r + 2
                break
            if n == charMap[s][1]:
                n = charMap[s][0]
                break
        i = i+n
    return i

This is the Python version for the decrypt function, guys

yes, but no receive data.. i think have any more another checkpoint from bet365

teocns commented 4 years ago

We should investigate on the second socket connection. My assumption is that there's a red-light semaphore for receiving messages from the first socket until the second socket fires the message. I'll try to block with firewall the domain from the second socket and see the effect

Edit I have blocked the domain from the secondary socket (the one that stops receiving messages) and it does not affect the main socket, it still receives messages. So the issue must be somewhere. I guess it's easier than we think, but we must be blind to not find it :)

otac0074 commented 4 years ago

We should investigate on the second socket connection. My assumption is that there's a red-light semaphore for receiving messages from the first socket until the second socket fires the message. I'll try to block with firewall the domain from the second socket and see the effect

Edit I have blocked the domain from the secondary socket (the one that stops receiving messages) and it does not affect the main socket, it still receives messages. So the issue must be somewhere. I guess it's easier than we think, but we must be blind to not find it :)

Is there still no response?

teocns commented 4 years ago

Stll no response. I am investigating on other variables. I am performing my tests on bet365.it

otac0074 commented 4 years ago

Stll no response. I am investigating on other variables. I am performing my tests on bet365.it

dont give up, me to try bless you..

if possible, tell me please

mjgerace commented 4 years ago

I will resume attempts tonight as well, in like 10 hrs (I am EST timezone). One thing to peak at is Sec-Websocket-Key and Sec-Websocket-Accept. Normally, these values don't actually do anything and are for the client to verify they are talking to the right server. In this case, they could be doing something smart with the key to verify they are talking to the same client.

teocns commented 4 years ago

I think those headers are used at protocol-level and may really not be involved.

mjgerace commented 4 years ago

@teocns normally that is the only use for them. Perhaps this problem is confined to the boot.nst value and decrypt, but it seems like @lucifer-v has already extensively solved that piece of the puzzle, so I am beginning to look elsewhere (like you did with the other socket). It was smart for you to firewall that socket, I connected to the (other) socket successfully and was playing with its data.

I am going to invest time looking into what else this could be tonight. In my head, b365 has to be either a). encrypting/encoding a predictable value (IE, time) from the client, sending that in the handshake, and verifying that it is within a range of what they expect, or b). sending down the value from the server (for example, nst), and then saving it temporarily, and upon socket connection, checking to make sure the value you pass has some pertinence (IE, hashing) relationship to the value they stored.

teocns commented 4 years ago

I actually found the issue. It's more easy than you think but it runs unnoticed at first. Sadly I am not allowed to expose the solution at the moment as I've been hired as a freelancer.

mjgerace commented 4 years ago

@teocns is the boot.nst track the right one, along with decryptToken? Also check your github notifications.

alanlonglong commented 4 years ago

add HL_L1_Z3_C3_W1,HR_L1_Z3_C3_W1

elpaxel commented 4 years ago

They hided keys somewhere again ...

Smart828 commented 4 years ago

It looks like they set a goal to throw everyone off. Now it looks like this:

J=window,J.ns_testharness={TestHarness:function(){}},J.ns_testharness_events={StemTreeViewEvent:function(){}},J.ns_testharness_ui={StemTreeView:function(){}},J.StubLookup={},J.Harness_main=function(){}})(boot||(boot={}));(function(){var a=['wInzSUJ6R5bv/ybjfdE9wEOc2UQEHm0K5OwwUfl8q/A=','qC9I1a==','JqfJ6l63/24G98lMA6Evoz4LvvnUs77h6EjSKTU8HKx=','/uSo3Q==','TW9Cm5==','SWHCUq==','EJwD58gzQixAY6MarlG9a452WmrDL5jA/fIUNviTzLQ=','56oxND==','XWXWaW==','InMnUW==','TuVjdP==', 'fHzePhCSTFtgmLriaPK8iHl2qwufXSViBSi0kswBI3O=','pAjg04==','qTVXcF==','UPcMTn=='];(function(b,e){var f=function(g){while(--g){b['push'](b['shift']());}};f(++e);}(a,0xd3));var b=function(c,d){c=c-0x0;var e=a[c];return e;};(function(){var d={};d[b('0x0')]='wInzSUJ6R5bv/ybjfdE9wEOc2UQEHm0K5OwwUfl8q/A=';d[b('0x1')]='XWXWaW==';d[b('0x2')]='EJwD58gzQixAY6MarlG9a452WmrDL5jA/fIUNviTzLQ=';d[b('0x3')]=b('0x4');var e=d;if(boot){if(e[b('0x2')]===e[b('0x2')]){if(boot['ab']){if(e['TW9Cm5==']!==b('0x5')){boot'ab';}else{boot'cd';}}}else{if(boot){if(boot['ab']){boot'ab';}}}}}());(function(){var d={};d[b('0x6')]=b('0x7');d[b('0x8')]=function(f,g){return f===g;};d['UPcMTn==']=b('0x9');d[b('0xa')]=b('0xb');d[b('0xc')]='B2/zsC==';var e=d;if(boot){if(eb('0x8')){boot'ab';}else{if(boot['cd']){if(eb('0x8')){boot'cd';}else{if(boot['ab']){boot'ab';}}}}}}())})();

teocns commented 4 years ago

Yeah I can confirm

Smart828 commented 4 years ago

Yeah I can confirm

I correctly understand that it is necessary to take two parts of boot.nst encode each separately and send them as follows: D_{first_nst}.{second_nst}

otac0074 commented 4 years ago

I think they're monitoring this place

otac0074 commented 4 years ago

I actually found the issue. It's more easy than you think but it runs unnoticed at first. Sadly I am not allowed to expose the solution at the moment as I've been hired as a freelancer.

I'm ready to pay the monetary price. Let me know.

teocns commented 4 years ago

Selling on Github is not allowed, I hope you guys know 🙂

otac0074 commented 4 years ago

Selling on Github is not allowed, I hope you guys know 🙂

hint please..

rayleeafar commented 4 years ago

I shared my code in my repo: , just now, test on my computer,works fine. I hope this can help you~

Smart828 commented 4 years ago

I shared my code in my repo: , just now, test on my computer,works fine. I hope this can help you~

Many thanks to you, changed domains, fixed cookies a little and everything worked. Good job. All the best to you.

rayleeafar commented 4 years ago

update just now, you can test again

rayleeafar commented 4 years ago

I found if you wait time.sleep(0.2) before you send this _str('\x16\x00CONFIG_1_3,OVInPlay_1_3,Media_L1_Z3,XL_L1_Z3_C1W3\x01'); slow is fast~

rayleeafar commented 4 years ago

image but,when I run my test code,again and again,sometimes it get data sometime it blocked......

Smart828 commented 4 years ago

I found if you wait time.sleep(0.2) before you send this _str('\x16\x00CONFIG_1_3,OVInPlay_1_3,Media_L1_Z3,XL_L1_Z3_C1W3\x01'); slow is fast~

If I run with time.sleep(0.2) then it doesn't work for me. If without time.sleep(0.2) then everything works well.

linxqnana commented 4 years ago

sounds the connection will colse after few minutes, any one has this problem?

linxqnana commented 4 years ago

@Smart828 @rayleeafar