search

davidteather / TikTok-Api

The Unofficial TikTok API Wrapper In Python
https://davidteather.github.io/TikTok-Api
MIT License
4.88k stars 976 forks source link

TikTokCaptchaError in some API functions using custom_verifyFp #407

Closed skyleradams closed 3 years ago

skyleradams commented 3 years ago

Describe the bug

Hello,

I downloaded this library and seem to be able to run some methods OK but others are consistently throwing TikTokCaptchaError.

Setup Process

  1. Log in to tiktok site through chrome browser
  2. Acquire s_v_web_id from cookies
  3. Pip install latest version of tiktokapi
  4. Run below code

The buggy code

Modified example code:

# this works and returns 10 ids
trending = api.trending(count=10, custom_verifyFp="key_goes_here")

# this does NOT work and returns a captchaerror
tiktoks = api.byUsername("americanredcross", count=10, custom_verifyFp="key_goes_here")

This is not an exhaustive list, I have not tried all the api methods to check if they work

Changes I have tried

Is anyone able to reproduce this?

skyleradams commented 3 years ago

Other things I've noticed:

@davidteather sometimes specifies a custom did - is that also from cookie information?

I've looked through closed issues and the closest I see is #385, before custom_verifyFp seemed to be required.

Desktop (please complete the following information):

OS: MacOS Mojave TikTokApi Version - 3.8.3

davidteather commented 3 years ago

Can you post the entire error trace that throws the TikTokCaptchaException?

skyleradams commented 3 years ago

Sure. Sorry, I should have done that initially.

ERROR:root:Tiktok response: 
 <!DOCTYPE html>
<html>
    <Head>
        <meta charset="utf-8">
        <title>TikTok</title>
        <link rel="shortcut icon" type="image/x-icon" id="favicon">
        <meta name="screen-orientation" content="portrait">
        <meta name="x5-orientation" content="portrait">
        <meta name="format-detection" content="telephone=no">
        <meta name="viewport" content="width=device-width, initial-scale=1, user-scalable=no, minimum-scale=1, maximum-scale=1, minimal-ui, viewport-fit=cover">
        <meta name="apple-mobile-web-app-capable" content="yes">
        <meta name="applicable-device" content="pc,mobile"/>
        <link rel="dns-prefetch" href="https://sf16-scmcdn-va.ibytedtos.com" />
        <script async src="https://sf16-scmcdn-va.ibytedtos.com/goofy/log-sdk/collect/collect-tcpy.js"></script>
        <script>
            const option = {"title":"tiktok-verify-page","iid":"0","did":"0","app_name":"tiktok","aid":1284,"favicon":"https://s16.tiktokcdn.com/musical/resource/mtact/static/images/tiktok-logo/logo.png","mobileIcons":["https://s16.tiktokcdn.com/musical/resource/mtact/static/images/tiktok-logo/tiktok_m.png","https://s16.tiktokcdn.com/musical/resource/mtact/static/images/tiktok-logo/tiktok_m2x.png","https://s16.tiktokcdn.com/musical/resource/mtact/static/images/tiktok-logo/tiktok_m3x.png"],"icons":["https://s16.tiktokcdn.com/musical/resource/mtact/static/images/tiktok-logo/tiktok_w.png","https://s16.tiktokcdn.com/musical/resource/mtact/static/images/tiktok-logo/tiktok_w2x.png","https://s16.tiktokcdn.com/musical/resource/mtact/static/images/tiktok-logo/tiktok_w3x.png"],"region":"va","type":"slide","lang":"en"};

            if (!option.region) {
                option.region = 'va';
            }
            var verifyTime = new Date().getTime();
            (function(win, export_obj) {
                win['TeaAnalyticsObject'] = export_obj;
                if (!win[export_obj]) {
                    function _collect() {
                        _collect.q.push(arguments);
                    }
                    _collect.q = _collect.q || [];
                    win[export_obj] = _collect;            
                }
                win[export_obj].l = +new Date();
            })(window, 'collectEvent');

            window.collectEvent('page.init', {
                app_id: option.region === 'cn' ? 2018 : 2740,
                channel: option.region === 'boe' ? 'cn' : option.region,
                log: true,
            });

            window.collectEvent('page.start');
            window.collectEvent('page.verify_page_load', {
                aid: option.aid,
                product_host: location.host,
                product_path: location.pathname,
                time: new Date().getTime(),
                is_success: 0,
                duration: new Date().getTime() - verifyTime
            })
            window.onbeforeunload = function () {
                window.collectEvent(document.readyState === 'complete' ? 'page.verify_page_close' : 'page.verify_page_load_close', {
                    product_host: location.host,
                    product_path: location.pathname,
                    aid: option.aid,
                    time: new Date().getTime(),
                    fp: (document.cookie.match(/s_v_web_id=(\w+)/) || [])[1],
                    is_success: Number(!!window.verify_is_success)
                })
            }
        </script>
        <script> window.captchaHost = "//verification-va.byteoversea.com";</script>
        <script src="https://s0.ipstatp.com/sec-sdk/sec_sdk_build/2.2.3/captcha.js"></script>
        <script async src="https://sf16-muse-va.ibytedtos.com/obj/eden-va2/fviylclsjeh7bogubfbd/tt-webapp/starling.browser.js"></script>
    </Head>
    <body>
        <div class="content">
            <div class="app_icon"></div>
            <div class="verify-wrap">
                <div id="verify-ele"></div>
            </div>
            <p class="page-desc" id="verifyEle"></p>
        </div>
    </body>
    <style>
        body {
            background: #EDF0F5;
            display: flex;
            min-height: 500px;
        }
        .content { 
            width: 300px;
            margin: auto;
        }

        .verify-wrap {
            min-height: 306px;
        }

        @media (min-width: 1281px) {
            .content {
                width: 380px;
            }

            .verify-wrap {
                min-height: 386px;
            }
        }

        .captcha_verify_container {
            border: 1px solid #E8E8E8;
        }

        .captcha_verify_container #verify-bar-close {
            display: none;
        }

        .app_icon img {
            margin-bottom: 20px;
        }

        .page-desc{
            margin-top: 32px;
            font-family: PingFangSC-Medium;
            font-size: 12px;
            color: #505050;
            line-height: 19px;
        }

        .page-desc span {
            font-family: 'PingFangSC-Regular';
            color: #505050;

        }
    </style>
    <script>
        const pageDescKey = 'user_verify_page_description';
        const ua = navigator.userAgent;
        const isMobile = /Android|webOS|iPhone|iPod|BlackBerry|Windows Phone|iPad/i.test(ua);

        window.onload = function () {
            window.verify_is_success = true;
            document.querySelector('#favicon').href = option.favicon;
            document.title = option.title || 'security verification';
            const icon = isMobile ? option.mobileIcons : option.icons;
            if (option.icons || true) {
                let img = document.createElement('img');
                img.srcset = `${icon[0]} 1x,
                    ${icon[1]} 2x,
                    ${icon[2]} 3x`;
                img.alt = 'app logo';
                img.src = icon[0];
                document.querySelector('.app_icon').appendChild(img);
            }

            const hosts = {
                cn: '//verify.snssdk.com',
                boe: '//boe-verify.snssdk.com',
                sg: '//verify-sg.byteoversea.com',
                va: '//verification-va.byteoversea.com'
            };

            window.collectEvent(location.href === document.referrer ? 'page.verify_page_refresh' : 'page.verify_page_init', {
                duration: new Date().getTime() - verifyTime,
                aid: option.aid,
                region: option.region,
                fp: (document.cookie.match(/s_v_web_id=(\w+)/) || [])[1],
                is_success: 1,
                product_host: location.host,
                product_path: location.pathname,
            })

            const starling = new Starling({
                api_key: '5dc26cf008d511e9b571e1bc0c9e23b5',
                namespace: 'Captcha',
                locale: option.lang,
                zone: (option.region || 'SG').toUpperCase(),
                test: false,
                fallbackLang: ['en'],
            });
            starling.load((texts) => {
                let desc = document.querySelector('#verifyEle');
                desc.innerText = texts[pageDescKey];
            });

            window.renderCaptcha({
                ele: 'verify-ele',
                host: hosts[option.region],
                aid: option.aid || 1284,
                iid: option.iid || '0',
                did: option.did || '0',
                lang: option.lang.match(/zh/) ? 'zh-Hant' : option.lang,
                region: option.region === 'boe' ? 'cn' : option.region,
                app_name: option.app_name || 'tiktok_reflow',
                hideCloseBtn: true,
                verify_data: JSON.stringify({
                  subtype: option.type,
                }),
                successCb: successCb,
                feedbackSubmitCb: feedbackSubmitCb,
                autoClose: false
            });

            // tt embed iframe
            if (window.self !== window.top && window.name.indexOf('__tt_embed__') !== -1) {
                const resizeData = JSON.stringify({
                    signalSource: window.name,
                    height: 600,
                });
                window.parent.postMessage(resizeData, '*');
            }
        }

        function successCb() {
            location.reload();
        }

        function feedbackSubmitCb() {
            window.renderCaptcha({
                ele: 'verify-ele',
                host: hosts[option.region],
                aid: option.aid || 1284,
                iid: option.iid || '0',
                did: option.did || '0',
                lang: option.lang.match(/zh/) ? 'zh-Hant' : option.lang,
                region: option.region,
                app_name: option.app_name || 'tiktok_reflow',
                hideCloseBtn: true,
                verify_data: JSON.stringify({
                  subtype: option.type,
                }),
                successCb: successCb,
                feedbackSubmitCb: feedbackSubmitCb,
                autoClose: false
            });
        }
    </script>
</html>
Traceback (most recent call last):
  File "/Users/skyleradams/Documents/github/tiktok_wrapped/TikTok_Api/TikTokApi/tiktok.py", line 1124, in getUser
    )[1].split("</script>")[0]
IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/skyleradams/Documents/github/tiktok_wrapped/test.py", line 36, in <module>
    tiktoks = api.byUsername("americanredcross", count=10, custom_verifyFp="verify_ki97npvl_35AVCpWz_sBzJ_4g8y_93Dq_ZDTTv1bQAmpq")
  File "/Users/skyleradams/Documents/github/tiktok_wrapped/TikTok_Api/TikTokApi/tiktok.py", line 518, in byUsername
    data = self.getUserObject(username, **kwargs)
  File "/Users/skyleradams/Documents/github/tiktok_wrapped/TikTok_Api/TikTokApi/tiktok.py", line 1087, in getUserObject
    return self.getUser(username, **kwargs)["userInfo"]["user"]
  File "/Users/skyleradams/Documents/github/tiktok_wrapped/TikTok_Api/TikTokApi/tiktok.py", line 1130, in getUser
    raise TikTokCaptchaError()
TikTok_Api.TikTokApi.exceptions.TikTokCaptchaError: TikTok blocks this request displaying a Captcha 
Tip: Consider using a proxy or a custom_verifyFp as method parameters
rocailler commented 3 years ago

i have the same, and nothing helps. I tried a proxy - the same.

skyleradams commented 3 years ago

@rocailler @davidteather I think I figured this out - you need to specify did and custom_did

I noticed that did=did appears in a couple places in documentation and tracked it down to here

Here is my updated example code that works: 

user = api.getUser("tiktokusername", did="user_goes_here", custom_did="did_goes_here", custom_verifyFp="verifyfp_goes_here")

user_object = user["userInfo"]["user"]

posts = api.userPosts(
            user_object["id"],
            user_object["secUid"],
            count=10,
            did="did_goes_here", custom_did="did_goes_here", custom_verifyFp="verifyfp_goes_here"
        )

for p in posts:
    print(p)
rocailler commented 3 years ago

it doesn't work for me, i've tried it before. did isn't a problem. The funniest thing is that is not always the same error. for list of profiles sometimes it passed, sometimes it is captcha, sometimes TikTokApi.exceptions.TikTokNotFoundError, sometimes even requests.exceptions.SSLError . I have no idea what is wrong, i thought that if captcha occur, then it is not possible to pass it without resolving captcha.

8bitgentleman commented 3 years ago

@skyleradams What does "user_goes_here" mean? Is that the username or some other user object?

davidteather commented 3 years ago

TikTokApi.exceptions.TikTokNotFoundError means the user doesn't exist/tiktok just isn't retuning data

@8bitgentleman You shouldn't need to pass the did="user_goes_here" as it's not utilized in the code. You can pass custom_did="some_string_here" as a parameter if that helps. You can pass any random string you want or random number.

skyleradams commented 3 years ago

@8bitgentleman yeah typo on my part for user_goes_here it should be a custom did string

@davidteather does "any random number" imply that it doesn't matter what the value is? I thought the comment here meant that the custom did should match the web_id_v2 cookie value: https://github.com/davidteather/TikTok-Api/blob/73a7f7aadaba57de22c7438c43776376d86b209f/examples/downloadTikTok.py#L9

rocailler commented 3 years ago

TikTokApi.exceptions.TikTokNotFoundError means the user doesn't exist/tiktok just isn't retuning data

I know but the user exists and have a posts. this is a bug related to this topic. All exception are related, TikTokNotFoundError is the worst - tiktok isn't returning data because of ban/ssl/certificate/or something else, and than i don't know if it really doesn't exist or it's a bug.

is api still working to you and other users? i'm trying everything (proxy, verifyfp, did,....)

I'm curios, in the error response i have the same region value that @skyleradams. It can't be changed. I don't know why it is always 'va'. Maybe it's a part of problem? my ideas are over... :(

,"region":"va","type":"slide","lang":"en"};

davidteather commented 3 years ago

does "any random number" imply that it doesn't matter what the value is?

It should be the tt_webid_v2/tt_webid cookie's value's but they don't seem to track it so :shrug: it DOES matter for subsequent api calls as that's how they seem to track what TikTok's you've viewed and match it with the preview of a video.

christophershultz commented 2 years ago

@davidteather

I know it's been some time since this discussion took place and I was wondering if a resolution was ever found. I've also attempted a proxy bypass with no luck. I'm primarily getting the Captcha error when using api.by_hashtag(), but not when using api.trending() as was mentioned above. Thanks in advance!!

Anos995 commented 2 years ago

بالتأكيد. ، البداية.


ERROR:root:Tiktok response: 
 <!DOCTYPE html>
<html>
    <Head>
        <meta charset="utf-8">
        <title>TikTok</title>
        <link rel="shortcut icon" type="image/x-icon" id="favicon">
        <meta name="screen-orientation" content="portrait">
        <meta name="x5-orientation" content="portrait">
        <meta name="format-detection" content="telephone=no">
        <meta name="viewport" content="width=device-width, initial-scale=1, user-scalable=no, minimum-scale=1, maximum-scale=1, minimal-ui, viewport-fit=cover">
        <meta name="apple-mobile-web-app-capable" content="yes">
        <meta name="applicable-device" content="pc,mobile"/>
        <link rel="dns-prefetch" href="https://sf16-scmcdn-va.ibytedtos.com" />
        <script async src="https://sf16-scmcdn-va.ibytedtos.com/goofy/log-sdk/collect/collect-tcpy.js"></script>
        <script>
            const option = {"title":"tiktok-verify-page","iid":"0","did":"0","app_name":"tiktok","aid":1284,"favicon":"https://s16.tiktokcdn.com/musical/resource/mtact/static/images/tiktok-logo/logo.png","mobileIcons":["https://s16.tiktokcdn.com/musical/resource/mtact/static/images/tiktok-logo/tiktok_m.png","https://s16.tiktokcdn.com/musical/resource/mtact/static/images/tiktok-logo/tiktok_m2x.png","https://s16.tiktokcdn.com/musical/resource/mtact/static/images/tiktok-logo/tiktok_m3x.png"],"icons":["https://s16.tiktokcdn.com/musical/resource/mtact/static/images/tiktok-logo/tiktok_w.png","https://s16.tiktokcdn.com/musical/resource/mtact/static/images/tiktok-logo/tiktok_w2x.png","https://s16.tiktokcdn.com/musical/resource/mtact/static/images/tiktok-logo/tiktok_w3x.png"],"region":"va","type":"slide","lang":"en"};

            if (!option.region) {
                option.region = 'va';
            }
            var verifyTime = new Date().getTime();
            (function(win, export_obj) {
                win['TeaAnalyticsObject'] = export_obj;
                if (!win[export_obj]) {
                    function _collect() {
                        _collect.q.push(arguments);
                    }
                    _collect.q = _collect.q || [];
                    win[export_obj] = _collect;            
                }
                win[export_obj].l = +new Date();
            })(window, 'collectEvent');

            window.collectEvent('page.init', {
                app_id: option.region === 'cn' ? 2018 : 2740,
                channel: option.region === 'boe' ? 'cn' : option.region,
                log: true,
            });

            window.collectEvent('page.start');
            window.collectEvent('page.verify_page_load', {
                aid: option.aid,
                product_host: location.host,
                product_path: location.pathname,
                time: new Date().getTime(),
                is_success: 0,
                duration: new Date().getTime() - verifyTime
            })
            window.onbeforeunload = function () {
                window.collectEvent(document.readyState === 'complete' ? 'page.verify_page_close' : 'page.verify_page_load_close', {
                    product_host: location.host,
                    product_path: location.pathname,
                    aid: option.aid,
                    time: new Date().getTime(),
                    fp: (document.cookie.match(/s_v_web_id=(\w+)/) || [])[1],
                    is_success: Number(!!window.verify_is_success)
                })
            }
        </script>
        <script> window.captchaHost = "//verification-va.byteoversea.com";</script>
        <script src="https://s0.ipstatp.com/sec-sdk/sec_sdk_build/2.2.3/captcha.js"></script>
        <script async src="https://sf16-muse-va.ibytedtos.com/obj/eden-va2/fviylclsjeh7bogubfbd/tt-webapp/starling.browser.js"></script>
    </Head>
    <body>
        <div class="content">
            <div class="app_icon"></div>
            <div class="verify-wrap">
                <div id="verify-ele"></div>
            </div>
            <p class="page-desc" id="verifyEle"></p>
        </div>
    </body>
    <style>
        body {
            background: #EDF0F5;
            display: flex;
            min-height: 500px;
        }
        .content { 
            width: 300px;
            margin: auto;
        }

        .verify-wrap {
            min-height: 306px;
        }

        @media (min-width: 1281px) {
            .content {
                width: 380px;
            }

            .verify-wrap {
                min-height: 386px;
            }
        }

        .captcha_verify_container {
            border: 1px solid #E8E8E8;
        }

        .captcha_verify_container #verify-bar-close {
            display: none;
        }

        .app_icon img {
            margin-bottom: 20px;
        }

        .page-desc{
            margin-top: 32px;
            font-family: PingFangSC-Medium;
            font-size: 12px;
            color: #505050;
            line-height: 19px;
        }

        .page-desc span {
            font-family: 'PingFangSC-Regular';
            color: #505050;

        }
    </style>
    <script>
        const pageDescKey = 'user_verify_page_description';
        const ua = navigator.userAgent;
        const isMobile = /Android|webOS|iPhone|iPod|BlackBerry|Windows Phone|iPad/i.test(ua);

        window.onload = function () {
            window.verify_is_success = true;
            document.querySelector('#favicon').href = option.favicon;
            document.title = option.title || 'security verification';
            const icon = isMobile ? option.mobileIcons : option.icons;
            if (option.icons || true) {
                let img = document.createElement('img');
                img.srcset = `${icon[2000000]} 1x,
                    ${icon[2000000]} 2x,
                    ${icon[2000000]} 3x`;
                img.alt = 'app logo';
                img.src = icon[200000];
                document.querySelector('.app_icon').appendChild(img);
            }

            const hosts = {
                cn: '//verify.snssdk.com',
                boe: '//boe-verify.snssdk.com',
                sg: '//verify-sg.byteoversea.com',
                va: '//verification-va.byteoversea.com'
            };

            window.collectEvent(location.href === document.referrer ? 'page.verify_page_refresh' : 'page.verify_page_init', {
                duration: new Date().getTime() - verifyTime,
                aid: option.aid,
                region: option.region,
                fp: (document.cookie.match(/s_v_web_id=(\w+)/) || [])[1],
                is_success: 1,
                product_host: location.host,
                product_path: location.pathname,
            })

            const starling = new Starling({
                api_key: '5dc26cf008d511e9b571e1bc0c9e23b5',
                namespace: 'Captcha',
                locale: option.lang,
                zone: (option.region || 'SG').toUpperCase(),
                test: false,
                fallbackLang: ['en'],
            });
            starling.load((texts) => {
                let desc = document.querySelector('#verifyEle');
                desc.innerText = texts[pageDescKey];
            });

            window.renderCaptcha({
                ele: 'verify-ele',
                host: hosts[option.region],
                aid: option.aid || 1284,
                iid: option.iid || '0',
                did: option.did || '0',
                lang: option.lang.match(/zh/) ? 'zh-Hant' : option.lang,
                region: option.region === 'boe' ? 'cn' : option.region,
                app_name: option.app_name || 'tiktok_reflow',
                hideCloseBtn: true,
                verify_data: JSON.stringify({
                  subtype: option.type,
                }),
                successCb: successCb,
                feedbackSubmitCb: feedbackSubmitCb,
                autoClose: false
            });

            // tt embed iframe
            if (window.self !== window.top && window.name.indexOf('__tt_embed__') !== -1) {
                const resizeData = JSON.stringify({
                    signalSource: window.name,
                    height: 600,
                });
                window.parent.postMessage(resizeData, '*');
            }
        }

        function successCb() {
            location.reload();
        }

        function feedbackSubmitCb() {
            window.renderCaptcha({
                ele: 'verify-ele',
                host: hosts[option.region],
                aid: option.aid || 1284,
                iid: option.iid || '0',
                did: option.did || '0',
                lang: option.lang.match(/zh/) ? 'zh-Hant' : option.lang,
                region: option.region,
                app_name: option.app_name || 'tiktok_reflow',
                hideCloseBtn: true,
                verify_data: JSON.stringify({
                  subtype: option.type,
                }),
                successCb: successCb,
                feedbackSubmitCb: feedbackSubmitCb,
                autoClose: false
            });
        }
    </script>
</html>
Traceback (most recent call last):
  File "/Users/skyleradams/Documents/github/tiktok_wrapped/TikTok_Api/TikTokApi/tiktok.py", line 1124, in getUser
    )[1].split("</script>")[0]
IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/skyleradams/Documents/github/tiktok_wrapped/test.py", line 13536, in <module>
    tiktoks = api.byUsername("americanredcross", count=100099, custom_verifyFp="verify_ki97npvl_35AVCpWz_sBzJ_4g8y_93Dq_ZDTTv1bQAmpq")
  File "/Users/skyleradams/Documents/github/tiktok_wrapped/TikTok_Api/TikTokApi/tiktok.py", line 5180, in byUsername
    data = self.getUserObject(username, **kwargs)
  File "/Users/skyleradams/Documents/github/tiktok_wrapped/TikTok_Api/TikTokApi/tiktok.py", line 10897, in getUserObject
    return self.getUser(username, **kwargs)["userInfo"]["user"]
  File "/Users/skyleradams/Documents/github/tiktok_wrapped/TikTok_Api/TikTokApi/tiktok.py", line 1130, in getUser
    raise TikTokCaptcha()
TikTok_Api.TikTokApi.exceptions.TikTokCaptcha: TikTok blocks this request displaying a Captcha 
Tip: Consider using a proxy or a custom_verifyFp as method parameters

لدي نفس الشيء ، ولا شيء يساعد. حاولت الوكيل - نفس الشيء.
Anos995 commented 2 years ago

2000000

Anos995 commented 2 years ago

826

Anos995 commented 2 years ago

826