Cannot scrape comments with generated verifyFp and signature

[Aze1337]

EDIT - Solution by @carcabot here - https://github.com/carcabot/tiktok-signature/issues/117#issuecomment-819608213

Hey there,

so I set it up and I tried to scrape a list of comments with the generated verifyFp & signature & also used the exact same cookies

check below, this is a working request which I have manually taken from a working request in the network tab of chrome api link - https://www.tiktok.com/api/comment/list/?aid=1988&app_name=tiktok_web&device_platform=web_pc&referer=&root_referer=&user_agent=Mozilla%2F5.0+(Windows+NT+10.0%3B+Win64%3B+x64)+AppleWebKit%2F537.36+(KHTML,+like+Gecko)+Chrome%2F89.0.4389.114+Safari%2F537.36&cookie_enabled=true&screen_width=1920&screen_height=1080&browser_language=en-US&browser_platform=Win32&browser_name=Mozilla&browser_version=5.0+(Windows+NT+10.0%3B+Win64%3B+x64)+AppleWebKit%2F537.36+(KHTML,+like+Gecko)+Chrome%2F89.0.4389.114+Safari%2F537.36&browser_online=true&ac=4g&timezone_name=America%2FToronto&priority_region=&verifyFp=verify_kn7rd9iy_NhT695jM_yiiE_4JfB_8ppr_uP0h3xcbNF1J&appId=1233&region=CA&appType=m&isAndroid=false&isMobile=false&isIOS=false&OS=windows&did=6948478889082340870&insert_ids=&aweme_id=6936702371682684166&cursor=0&count=20&app_language=en&current_region=CA&fromWeb=1&channel_id=0&_signature=_02B4Z6wo00901I-D5VwAAIDCU8eYL-NkVTyPg-HAAEOR6f

the response is the following

this is the link of a request that is using the generated verifyFp and tiktok signature https://www.tiktok.com/api/comment/list/?aid=1988&app_name=tiktok_web&device_platform=web_pc&referer=&root_referer=&user_agent=Mozilla%2F5.0+(Windows+NT+10.0%3B+Win64%3B+x64)+AppleWebKit%2F537.36+(KHTML,+like+Gecko)+Chrome%2F89.0.4389.114+Safari%2F537.36&cookie_enabled=true&screen_width=1920&screen_height=1080&browser_language=en-US&browser_platform=Win32&browser_name=Mozilla&browser_version=5.0+(Windows+NT+10.0%3B+Win64%3B+x64)+AppleWebKit%2F537.36+(KHTML,+like+Gecko)+Chrome%2F89.0.4389.114+Safari%2F537.36&browser_online=true&ac=4g&timezone_name=America%2FToronto&priority_region=&verifyFp=verify_kn7rd9iy_NhT695jM_yiiE_4JfB_8ppr_uP0h3xcbNF1J&appId=1233&region=CA&appType=m&isAndroid=false&isMobile=false&isIOS=false&OS=windows&did=6948478889082340870&insert_ids=&aweme_id=6936702371682684166&cursor=0&count=20&app_language=en&current_region=CA&fromWeb=1&channel_id=0&_signature=_02B4Z6wo00901I-D5VwAAIDCU8eYL-NkVTyPg-HAAEOR6f

the response is

it doesn't works , any idea why ? thank you very much

[Michaelmala]

Hi @Aze1337 did you added the "tt_webid_v2" cookie value to the request headers?

[Aze1337]

Hi @Aze1337 did you added the "tt_webid_v2" cookie value to the request headers?

Hey! let me show you

this is the cookie of the request that doesn't works (trying to scrape comments)

and here is the request that im trying to replicate (do you see anything im missing ?)

thanks

[chdzma]

Hi @Aze1337 I have the same problem, did you find any solution?

I know that signing the url directly with a browser works, executing the script in the console on a tiktok page

[Aze1337]

Hi @Aze1337 I have the same problem, did you find any solution?

I know that signing the url directly with a browser works, executing the script in the console on a tiktok page

Hey! I still haven't found it yet. Can you send me the script to execute on console ? I'm guessing it could be because we need to be logged in order to see the comments page, and it's probably generated without taking in consideration the sessionid cookie ?

are you trying to do the same thing as me ? (comments scraping api) ?

edit - I'm 100% sure it has nothing to do with cookies as I have sent the exact same request as a normal one but only with the sessionid cookie only (which is the cookie used to know the account you logged in, if you remove it - you cant see comments) - it's 100% the signature/verifyFp that is at fault - maybe it's generated differently when you are logged into an account vs not logged and that could causes it (???) but I have no idea on how to understand tik tok way to generate it. we need you! haha @carcabot

[chdzma]

@Aze1337

You can go to the tiktok page: tiktok.com and open dev tools, in the console section you can run the script as follows:

window.byted_acrawler.sign({"url":"url_with_parameters_to_sign"})

Once signed, you add the _signature parameter with the result and you can test it with postman. In my case I do not pass cookies, you can open the url in a private tab and it will return the results.

I have tried everything and before with the chromedp library it worked :(

[carcabot]

You have to pass also verfyFp cookie along with _signature and also request must embed cookie tt_webid_v2 and sessionid since comment listing requires authentication.

This being said, follow this small tutorial to generate the correct signature.

Using this library run command:

node browser "https://www.tiktok.com/api/comment/list/?aid=1988&app_name=tiktok_web&insert_ids=&aweme_id=6936702371682684166&cursor=0&count=20&app_language=en"

Response should be similar but not mandatory to be exact like this since signature is changing based on timestamp:

{"signature":"_02B4Z6wo00f01AU9KygAAIDBunKE8Gn7VMwFHC-AAGE-81","verifyFp":"verify_af3296c06bac8502b2c63f4dc8d9b160","cookies":"MONITOR_WEB_ID=6816677712255075841; csrf_session_id=d02d710e56754f91b46b27cce4337768; _abck=80E5F2F4B4F99572385ADF72E39B6949~-1~YAAQn4B8VlDQRHJ4AQAAvpj10AW0jRBJwl8kobHSBLR+Bh2hdkb4zpIggjI8p/O6u1CjzdTv8lY7gFQEUb1nXtePkaT41CGRAxQL3eouFwUqlrCLxfvSHZztjvMJuDBssrP180lAtvOKw2Kgw71Mcj3uvNQ2ux13oH8K/VM39ISom10RxJULasUvYI28RA/gro6RUeASUS+QUdTwStmuzCIsT0CAcY046WZHv5hni2RDT2NQ9yE+F+82ucoZadvy0v6ZPBRWFaWFjEqYvjqgfSJHOn4QO0lq6KDR6qy2tt6iLeRh82a8oyEiZ7/xhmOy8dTZZzXDqeyhdp7++iyYQ9SkY5xbx24nQNUKmljfJHTCcJ15jDRHUh7DsgZ3~-1~-1~-1; s_v_web_id=verify_af3296c06bac8502b2c63f4dc8d9b160"}

Create a python file with the following content let's say: comment.py

import requests

signature = "_02B4Z6wo00f01AU9KygAAIDBunKE8Gn7VMwFHC-AAGE-81"
verifyFp = "verify_af3296c06bac8502b2c63f4dc8d9b160"

referer = "https://www.tiktok.com/@ondymikula/video/6847563020290346245"

url = "https://www.tiktok.com/api/comment/list/?aid=1988&app_name=tiktok_web&insert_ids=&aweme_id=6936702371682684166&cursor=0&count=20&app_language=en" + \
    "&verifyFp=" + verifyFp + \
    "&_signature=" + signature

request = requests.get(url, headers={"method": "GET",
                                     "accept-encoding": "gzip, deflate",
                                     "cookie": "tt_webid_v2=1234567890; sessionid=1a70bc1117090fadbcb9cad568134888",
                                     "Referer": referer,
                                     "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36"
                                     })
data = request.text
print(data)

And now you can run the script

python comment.py

You should be sure that user agent used by tiktok-signature library is the same with python file and session id is valid.

If you don't pass the sessionid cookie you will get error

{"status_code":8,"status_msg":"Login expired","log_pb":{"impr_id":"2021041415192701019021822645168B9F"}}

Greetings.

[Aze1337]

You have to pass also verfyFp cookie along with _signature and also request must embed cookie tt_webid_v2 and sessionid since comment listing requires authentication.

This being said, follow this small tutorial to generate the correct signature.

Using this library run command:

node browser "https://www.tiktok.com/api/comment/list/?aid=1988&app_name=tiktok_web&insert_ids=&aweme_id=6936702371682684166&cursor=0&count=20&app_language=en"

Response should be similar but not mandatory to be exact like this since signature is changing based on timestamp:

{"signature":"_02B4Z6wo00f01AU9KygAAIDBunKE8Gn7VMwFHC-AAGE-81","verifyFp":"verify_af3296c06bac8502b2c63f4dc8d9b160","cookies":"MONITOR_WEB_ID=6816677712255075841; csrf_session_id=d02d710e56754f91b46b27cce4337768; _abck=80E5F2F4B4F99572385ADF72E39B6949~-1~YAAQn4B8VlDQRHJ4AQAAvpj10AW0jRBJwl8kobHSBLR+Bh2hdkb4zpIggjI8p/O6u1CjzdTv8lY7gFQEUb1nXtePkaT41CGRAxQL3eouFwUqlrCLxfvSHZztjvMJuDBssrP180lAtvOKw2Kgw71Mcj3uvNQ2ux13oH8K/VM39ISom10RxJULasUvYI28RA/gro6RUeASUS+QUdTwStmuzCIsT0CAcY046WZHv5hni2RDT2NQ9yE+F+82ucoZadvy0v6ZPBRWFaWFjEqYvjqgfSJHOn4QO0lq6KDR6qy2tt6iLeRh82a8oyEiZ7/xhmOy8dTZZzXDqeyhdp7++iyYQ9SkY5xbx24nQNUKmljfJHTCcJ15jDRHUh7DsgZ3~-1~-1~-1; s_v_web_id=verify_af3296c06bac8502b2c63f4dc8d9b160"}

Create a python file with the following content let's say: comment.py

import requests

signature = "_02B4Z6wo00f01AU9KygAAIDBunKE8Gn7VMwFHC-AAGE-81"
verifyFp = "verify_af3296c06bac8502b2c63f4dc8d9b160"

referer = "https://www.tiktok.com/@ondymikula/video/6847563020290346245"

url = "https://www.tiktok.com/api/comment/list/?aid=1988&app_name=tiktok_web&insert_ids=&aweme_id=6936702371682684166&cursor=0&count=20&app_language=en" + \
    "&verifyFp=" + verifyFp + \
    "&_signature=" + signature

request = requests.get(url, headers={"method": "GET",
                                     "accept-encoding": "gzip, deflate",
                                     "cookie": "tt_webid_v2=1234567890; sessionid=1a70bc1117090fadbcb9cad568134888",
                                     "Referer": referer,
                                     "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36"
                                     })
data = request.text
print(data)

And now you can run the script

python comment.py

You should be sure that user agent used by tiktok-signature library is the same with python file and session id is valid.

If you don't pass the sessionid cookie you will get error

{"status_code":8,"status_msg":"Login expired","log_pb":{"impr_id":"2021041415192701019021822645168B9F"}}

Greetings.

Thank you very much for your response! I really appreciate the time you put into this! I'm going to give it a try with what you just said and edit my post haha hopefully I can get it to work hehe

@carcabot Thank you so much! it does works with the python file! Thanks a lot! I'm going to rewrite it for my project in c# and see if I can still get it to works there!

also wondering , if you see this message before I do my own tests - is it the same url you must sign in order to like a comment or post one ?

thanks a lot again!

[Aze1337]

@carcabot Hey - I was wondering - how do you know which url is used to sign ? for example, the link you put in your example does works for scraping comments. but for liking a comment or posting one - how would I find that by myself ? thanks

edit -

for liking comment I tried signing this url -https://www.tiktok.com/api/comment/digg/?aid=1988&app_name=tiktok_web&device_platform=web&referer=&root_referer=&&cookie_enabled=true&screen_width=1920&screen_height=1080&browser_language=en-US&browser_platform=Win32&browser_online=true&ac=4g&timezone_name=America%2FToronto&priority_region=&appId=1233&region=CA&appType=m&isAndroid=false&isMobile=false&isIOS=false&OS=windows&cid=" + cid + "&aweme_id=6936702371682684166&channel_id=0&digg_type=1

tried this one too

"https://www.tiktok.com/api/comment/digg/?aid=1988&app_name=tiktok_web&device_platform=web&referer=&root_referer=&browser_language=en-US&appId=1233&cid=" + cid + "&aweme_id=6936702371682684166&channel_id=0&digg_type=1";

tried this too

https://www.tiktok.com/api/comment/digg/?aid=1988&app_name=tiktok_web&insert_ids=&aweme_id=6936702371682684166&cursor=0&count=20&app_language=en&digg_type=1

wouldn't let me.. im still trying to figure out which url it is . ya need to teach us how to find it haha!

[carcabot]

In general the signature should be the same for all kind of requests signed with this algorithm. I personally didn't checked if this feature works but I'm sure that it should do without issues.

All you have to do is to watch in dev tools how request is made and replicate it in the same way. Also in Dev Tools you can see the request url for each action you do on TikTok.

[Aze1337]

In general the signature should be the same for all kind of requests signed with this algorithm. I personally didn't checked if this feature works but I'm sure that it should do without issues.

All you have to do is to watch in dev tools how request is made and replicate it in the same way. Also in Dev Tools you can see the request url for each action you do on TikTok.

Alright thanks a lot for your time. For some reasons I can't find the good api to sign but I'll try hard on it later tonight and hopefully find out which one it is

edit - I found what is needed to sent to comment! hopefully I'll find now for liking a comment. you need to be very careful at the parameters you send. thanks again to you @carcabot i'll try buying you coffees when my project generate it's first revenue!

[youssefavx]

Hey guys when I run:

node browser "https://www.tiktok.com/api/comment/list/?aid=1988&app_name=tiktok_web&insert_ids=&aweme_id=6936702371682684166&cursor=0&count=20&app_language=en"

I get this error:

page.evaluate: Evaluation failed: TypeError: Cannot read property 'sign' of undefined
    at eval (eval at evaluate (:303:29), <anonymous>:2:40)
    at UtilityScript.evaluate (<anonymous>:305:22)
    at UtilityScript.<anonymous> (<anonymous>:1:44)
Error
    at Object.captureStackTrace (/Volumes/Transcend/tiktok-signature/node_modules/playwright-chromium/lib/utils/stackTrace.js:51:19)
    at Connection.sendMessageToServer (/Volumes/Transcend/tiktok-signature/node_modules/playwright-chromium/lib/client/connection.js:69:48)
    at Proxy.<anonymous> (/Volumes/Transcend/tiktok-signature/node_modules/playwright-chromium/lib/client/channelOwner.js:64:61)
    at /Volumes/Transcend/tiktok-signature/node_modules/playwright-chromium/lib/client/frame.js:177:42
    at Frame._wrapApiCall (/Volumes/Transcend/tiktok-signature/node_modules/playwright-chromium/lib/client/channelOwner.js:77:34)
    at Frame.evaluate (/Volumes/Transcend/tiktok-signature/node_modules/playwright-chromium/lib/client/frame.js:176:21)
    at /Volumes/Transcend/tiktok-signature/node_modules/playwright-chromium/lib/client/page.js:360:60
    at Page._attributeToPage (/Volumes/Transcend/tiktok-signature/node_modules/playwright-chromium/lib/client/page.js:202:20)
    at Page.evaluate (/Volumes/Transcend/tiktok-signature/node_modules/playwright-chromium/lib/client/page.js:360:21)
    at Signer.init (/Volumes/Transcend/tiktok-signature/index.js:70:21)
    at async main (/Volumes/Transcend/tiktok-signature/browser.js:8:5)

Any idea what this is about? I upgraded node recently after noticing a syntax error with =. Not sure if related.

[youssefavx]

Never mind! All I had to do was add '.js' to browser and it worked!

node browser.js "https://www.tiktok.com/api/comment/list/?aid=1988&app_name=tiktok_web&insert_ids=&aweme_id=6936702371682684166&cursor=0&count=20&app_language=en"