carcabot / tiktok-signature

Generate tiktok signature token using node
756 stars 286 forks source link

Did TikTok just change up a lot of stuff again? I can not get any data. #86

Closed robindz closed 4 years ago

robindz commented 4 years ago

Pretty much the title, looks like a whole lot of parameters got added to the endpoint and I'm not sure where to start really.

robindz commented 4 years ago

I made a fresh clone of the repo, installed playwright & playwright-webkit and tried to get a signature through the following command: node browser.js "https://m.tiktok.com/api/item_list/?count=30&id=6805697443515663361&type=1&secUid=&maxCursor=0&minCursor=0&sourceType=8&appId=1233" and this resulted in the following error:

Error: Evaluation failed: TypeError: undefined is not an object (evaluating 'window.byted_acrawler.sign')
    at WKExecutionContext.evaluateWithArguments (C:\Users\robin\source\repos\tiktok-test\tiktok-signature\node_modules\playwright-webkit\lib\webkit\wkExecutionContext.js:74:23)
    at async evaluateExpression (C:\Users\robin\source\repos\tiktok-test\tiktok-signature\node_modules\playwright-webkit\lib\javascript.js:164:16)
    at async FrameManager.waitForSignalsCreatedBy (C:\Users\robin\source\repos\tiktok-test\tiktok-signature\node_modules\playwright-webkit\lib\frames.js:90:24)
    at async FrameExecutionContext.evaluateInternal (C:\Users\robin\source\repos\tiktok-test\tiktok-signature\node_modules\playwright-webkit\lib\dom.js:36:16)
    at async Signer.init (C:\Users\robin\source\repos\tiktok-test\tiktok-signature\index.js:70:5)
    at async main (C:\Users\robin\source\repos\tiktok-test\tiktok-signature\browser.js:8:9)
  -- ASYNC --
    at Frame.evaluate (C:\Users\robin\source\repos\tiktok-test\tiktok-signature\node_modules\playwright-webkit\lib\helper.js:79:23)
    at C:\Users\robin\source\repos\tiktok-test\tiktok-signature\node_modules\playwright-webkit\lib\page.js:286:61
    at Page._attributeToPage (C:\Users\robin\source\repos\tiktok-test\tiktok-signature\node_modules\playwright-webkit\lib\page.js:370:20)
    at Page.evaluate (C:\Users\robin\source\repos\tiktok-test\tiktok-signature\node_modules\playwright-webkit\lib\page.js:286:21)
    at Page.evaluate (C:\Users\robin\source\repos\tiktok-test\tiktok-signature\node_modules\playwright-webkit\lib\helper.js:80:31)
    at Signer.init (C:\Users\robin\source\repos\tiktok-test\tiktok-signature\index.js:70:21)
    at async main (C:\Users\robin\source\repos\tiktok-test\tiktok-signature\browser.js:8:9)

So I'm assuming something definitely got changed on their end. I've a feeling the endpoint itself might have changed to (for some reason I can still get a signature & verifyFp on my Raspberry Pi, but when using trending.py I get the following response:

{"code":"10000","from":"","type":"verify","version":"","region":"va","subtype":"slide","detail":"lJfMBXJCp0qsA8dOcUQyG3iIPu*h9FmP701Lo9lazaceKT1sgGbku4dhWZ3ZIfm9Dnr0nDK8w86C509ZHFLJZFGhdMZ5d3SQJis5oMOCRmDQnAcOYdqF2SzbpcmmFYQQV0q83HaKoZqPRO6Sbask4qMHKgnMkZSeQhRPzV0vs*zdMhj9d-a12VZmndrsSKBe3OBnS7QtgiimUwdXgjIjemVYHihSoIg.","verify_event":"","fp":"verify_kdjamid1_7D32cydO_sSjl_41Lc_8rd0_vbn8nVtmv2fW"}
liamengland1 commented 4 years ago

I get the following response:

That is a captcha

robindz commented 4 years ago

Just tried the following on my Raspberry Pi (fresh repo clone):

pi@chaengpi:~/scripts/tiktok-test/tiktok-signature $ DEBUG=pw:api node listen.js
TikTok Signature server started
(node:2309) UnhandledPromiseRejectionWarning: browserType.launch: Protocol error (Playwright.enable): Browser has been closed.
=========================== logs ===========================
[browser] <launching> /usr/bin/chromium-browser --inspector-pipe --headless --no-startup-window
[browser] <launched> pid=2320
[browser] Fontconfig warning: "/etc/fonts/fonts.conf", line 100: unknown element "blank"
============================================================
Note: use DEBUG=pw:api environment variable and rerun to capture Playwright logs.Error
    at /home/pi/scripts/tiktok-test/tiktok-signature/node_modules/playwright-webkit/lib/webkit/wkConnection.js:103:63
    at new Promise (<anonymous>)
    at WKSession.send (/home/pi/scripts/tiktok-test/tiktok-signature/node_modules/playwright-webkit/lib/webkit/wkConnection.js:102:16)
    at Function.connect (/home/pi/scripts/tiktok-test/tiktok-signature/node_modules/playwright-webkit/lib/webkit/wkBrowser.js:52:37)
    at WebKit._connectToTransport (/home/pi/scripts/tiktok-test/tiktok-signature/node_modules/playwright-webkit/lib/server/webkit.js:32:38)
    at WebKit._innerLaunch (/home/pi/scripts/tiktok-test/tiktok-signature/node_modules/playwright-webkit/lib/server/browserType.js:90:36)
    at async ProgressController.run (/home/pi/scripts/tiktok-test/tiktok-signature/node_modules/playwright-webkit/lib/progress.js:75:28)
    at async WebKit.launch (/home/pi/scripts/tiktok-test/tiktok-signature/node_modules/playwright-webkit/lib/server/browserType.js:62:25)
    at async Signer.init (/home/pi/scripts/tiktok-test/tiktok-signature/index.js:43:22)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:2309) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2)
(node:2309) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

and

pi@chaengpi:~/scripts/tiktok-test/tiktok-signature $ DEBUG=pw:api node listen.js
TikTok Signature server started
events.js:292
      throw er; // Unhandled 'error' event
      ^

Error: read ECONNRESET
    at Pipe.onStreamRead (internal/stream_base_commons.js:205:27)
Emitted 'error' event on Socket instance at:
    at emitErrorNT (internal/streams/destroy.js:84:8)
    at processTicksAndRejections (internal/process/task_queues.js:84:21) {
  errno: -104,
  code: 'ECONNRESET',
  syscall: 'read'
}

Both of these error occur (randomly)

robindz commented 4 years ago

I've been messing around for about two hours now and I'm fairly certain something changed with window.byted_acrawler.sign because I keep getting an error that states: Error: Evaluation failed: TypeError: undefined is not an object (evaluating 'window.byted_acrawler.sign')

Platform: Windows 10 10.0.18363 Build 18363 Node version: v12.16.1 Playwright version: 1.3.0

@carcabot

carcabot commented 4 years ago

@GodsWithin the repo is still working, probably you got banned and that's why you don't have access to window.byted_crawler.sign function. Try changing your server or IP to see if it's working or use proxy with playwright.

bagastepe commented 4 years ago

I get the following response:

That is a captcha I also got response like this, but after waiting for a while it worked again

{ "code": "10000", "from": "", "type": "verify", "version": "", "region": "va", "subtype": "slide", "detail": "eBbwLokGfaLjXVYaWnSk1MvzmNFjxvIqp-mRmsRuk43mLC5PAaTw4BQxsAyHZ8m5yTO-y8Q6An6LO0BOQF5OBAYho9MBZYEC3DCHy34PgiC93ejRyNXxxJAud0HprbvXF9m0U2H0EDA6DU*77rPIi0pdNmCbPP5tTravXU9JWMElyIj4Ed8B0r-FPDOyxXzI-RgPGNmeHw4oouZ46ftWEcIsUOjqE.", "verify_event": "", "fp": "verify_kdjpto2n_ovlRRIRu_6HQ0_4N6X_BaDu_pYTHNjSoDDKv" }

bjrambo commented 4 years ago

do not use that.

u use this endpoint.

i test it resolve.

https://t.tiktok.com/api/item_list/?aid=1988&app_name=tiktok_web&device_platform=web&referer=&user_agent=&cookie_enabled=true&screen_width=2560&screen_height=1440&browser_language=ko&browser_platform=&browser_name=&browser_version=&browser_online=true&timezone_name=Asia%2FSeoul&priority_region=&appId=1180&region=KR&appType=t&isAndroid=false&isMobile=false&isIOS=false&OS=mac&did=&count=30&id= 6805697443515663361&type=1&secUid=3&maxCursor=0&minCursor=0&sourceType=8&language=ko

node browser.js "https://t.tiktok.com/api/item_list/?aid=1988&app_name=tiktok_web&device_platform=web&referer=&user_agent=&cookie_enabled=true&screen_width=2560&screen_height=1440&browser_language=ko&browser_platform=&browser_name=&browser_version=&browser_online=true&timezone_name=Asia%2FSeoul&priority_region=&appId=1180&region=KR&appType=t&isAndroid=false&isMobile=false&isIOS=false&OS=mac&did=&count=30&id= 6805697443515663361&type=1&secUid=3&maxCursor=0&minCursor=0&sourceType=8&language=ko"

// get to _signature 

https://t.tiktok.com/api/item_list/?aid=1988&app_name=tiktok_web&device_platform=web&referer=&user_agent=&cookie_enabled=true&screen_width=2560&screen_height=1440&browser_language=ko&browser_platform=&browser_name=&browser_version=&browser_online=true&timezone_name=Asia%2FSeoul&priority_region=&appId=1180&region=KR&appType=t&isAndroid=false&isMobile=false&isIOS=false&OS=mac&did=&count=30&id= {0}&type=1&secUid=3&maxCursor=0&minCursor=0&sourceType=8&language=ko&verifyFp={1}&_signature={2}

if try, tell me plz.

bjrambo commented 4 years ago

If you request api for the last TikTok, you must use PC User-Agent in header request.

header = "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36", ( windows or mac ETC only use to PC user agent

but, there is no need to modify the useragent in index.js file. ( https://github.com/carcabot/tiktok-signature/blob/master/index.js#L6 ) no need fix it.

bjrambo commented 4 years ago

e.g for PHP)

        //First get to _signature, not use to user - agent, and no need fix index.js files.
        $ch = curl_init();
        curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'POST');
        // i use to Docker. so i request to myserver. it same node browser.js "urlurlurl~~"
        curl_setopt($ch, CURLOPT_URL, "http://localhost:8080/signature");
        curl_setopt ($ch, CURLOPT_POSTFIELDS, "urlurlurl~~");
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
        curl_setopt($ch, CURLOPT_TIMEOUT, 30);
        curl_setopt($ch, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_V4);
        $result = curl_exec($ch);
        curl_close($ch);
        $src_data = json_decode($result);
        debugPrint($src_data);

        $url = "urlurlurl~~&verifyFp={$src_data->verifyFp}&_signature={$src_data->signature}";

        debugPrint($url);
        $ch = curl_init();
        $headers = array(
            "Host: t.tiktok.com",
            "Cookie: {$src_data->cookies}",
            // must be use pc agent
            "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36",
            "Referer: https://www.tiktok.com/@dj_hyo",
        );

        curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
        curl_setopt($ch, CURLOPT_URL, $url);
        curl_setopt($ch, CURLOPT_ENCODING, 'gzip');
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
        curl_setopt($ch, CURLOPT_TIMEOUT, 30);
        curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
        curl_setopt($ch, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_V4);
        if (curl_error($ch))
        {
            print_r(curl_error($ch));
        }

        $result = curl_exec($ch);
        curl_close($ch);
        $src_data = json_decode($result);
toxicity1985 commented 4 years ago

Use m or t doesn't care. But know they have add a captcha so sometimes it works and sometimes not. The way is to add a timer between the request but it's not ok for all the request.

toxicity1985 commented 4 years ago

And for the moment, tiktok have some issue with their website. Sometimes, i can see the page, sometimes not. Sometimes, i can see the video and sometimes not.

toxicity1985 commented 4 years ago

`

{"code":"10000","from":"","type":"verify","version":"","region":"va","subtype":"slide","detail":"PnZnAi4mDrLOr8X-Qm8scYydxJEH9rGBS0Ejxd7iLk3hj0EvCxAcWAk9s9N2wQ0DZYUdwXsdftbLw6TulvIgMgqhOKe1NU34delBuiU2DxWSOIoV6mxevVjXSdUL43EExwYAOmKhuY8zGpKEj-9ROJ0sBwVtlC2-RXpAfXLhTrghxGy1navti8j8JuIyfJnm3eW7b0Y4pq-CbsIjVQaN2Lyqs.","verify_event":"","fp":"verify_kdk0ns1v_JL2UPEDP_htJj_4mBg_BIu8_HdptEiBTrUGJ"}

{"code":"10000","from":"","type":"verify","version":"","region":"va","subtype":"slide","detail":"L1alswuh7bIlfynS1TVAOh5TrD9v*tiOHLZgZ3pui6rhkujf8ynVjiIqJngUsdaIlp9ABF8hbeJ-4Ime0BE-z6VAN2KgTOTdMpzJtOPDUGikuYqI5f9D7XduZmlLIFaCXmYw6JZlZFmBYz5Ui*n8bCoQF0Cnf*vEYflfSJ4MJtYYs5pOEb1UEpaIPTt1mtQDxMLUe9xR9n4HwBswWlEFO51IUvNSVnU.","verify_event":"","fp":"verify_kdk0nvmb_ANVMYCEo_EtKj_4qiB_835u_q20qgiVkH3EW"}
{"statusCode":0,"userInfo":{"user":{"id":"6799709771164992518","uniqueId":"_lauryneg","nickname":"_lauryneg","avatarThumb":"https://p16-va-tiktok.ibyteimg.com/img/musically-maliva-obj/df35191efb1d309a9b4780d7ddedbe07~c5_100x100.jpeg","avatarMedium":"https://p16-va-tiktok.ibyteimg.com/img/musically-maliva-obj/df35191efb1d309a9b4780d7ddedbe07~c5_720x720.jpeg","avatarLarger":"https://p16-va-tiktok.ibyteimg.com/img/musically-maliva-obj/df35191efb1d309a9b4780d7ddedbe07~c5_1080x1080.jpeg","signature":"21\nInstagram 👉🏻 @_lauryneg","verified":false,"secUid":"MS4wLjABAAAAVucKvb0tIWNKu3_aFyEycONyEgfZg008b9Z8Lr3JXSo9QQlqpi3YTo60q4UXvEAm","secret":false,"relation":0,"openFavorite":false},"stats":{"followingCount":254,"followerCount":1331,"heartCount":14700,"videoCount":139,"diggCount":0,"heart":14700},"shareMeta":{"title":"Join TikTok and see what I've been up to!","desc":"TikTok: Make Every Second Count"}}}

`

bagastepe commented 4 years ago

I just loop it until I get the actual result (code != '10000')

robindz commented 4 years ago

@GodsWithin the repo is still working, probably you got banned and that's why you don't have access to window.byted_crawler.sign function. Try changing your server or IP to see if it's working or use proxy with playwright.

Is there a recommended interval duration to perform checks? In my case on the posts of a single user.

@carcabot

daviddanielarch-zz commented 4 years ago

If you need to make lots of requests just use the server functionality, it will only makes one request from time to time to update the token.

robindz commented 4 years ago

@daviddanielarch I'm talking about requests sent to endpoints, not for getting credentials.

toxicity1985 commented 4 years ago

If you need to make lots of requests just use the server functionality, it will only makes one request from time to time to update the token.

Server functionality doesn't work when we make a second call. We have an error with the Signer.

bjrambo commented 4 years ago

All you have to do is change the manual. We've worked this out.

Right? @GodsWithin

toxicity1985 commented 4 years ago

Hello,

So Tiktok implements a captha way. One solution i found and didn't test for the moment is https://2captcha.com/2captcha-api#solving_tiktok. You have to send some cookie detail and they give you back what you need to bypass the captcha. I will test it this week so i can tell you if it's working or not.