Open davidteather opened 3 years ago
Issue-Label Bot is automatically applying the label feature_request
to this issue, with a confidence of 0.99. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.
Hi David, has the issued been solved, at least for a certain amount of time? How am I supposed to use the verifyfp? My ByHashtag is not working. Will you update a new version soon? Thank you so much for your help!
I used exactly the code you wrote there regarding the captcha. However, it still returned the same error as before. Did I not implement it correctly?
I downloaded the new version 3.7.8 and included the cookie but it still did not return me anything. I was just wondering if the issue still happening to you?
Did you end up solving a captcha before including that cookie?
Yes! I did.
It returned the same error and I was not asked to solve another captcha so the cookie was the same.
May I ask how your code that worked well look like? Sorry for keep asking questions. Just really need help on this.
The following works for me which also worked for the test suite
from TikTokApi import TikTokApi
import logging
api = TikTokApi.get_instance(logging_level=logging.INFO)
tiktoks = api.byHashtag('funny', custom_verifyFp="verify_khgp75kb_wubFGH9K_YApB_4evl_8Tw3_FVz6N3YxEhxI")
Oh it worked now!! Thank you so much!! That's really helpful!!
@davidteather I tested here using a single cookie value for valueFp.
I created an application doing 1 user post request each 5 seconds (0.2 req/s). Basically I'm crawling TikTok User Posts.
TikTok are blocking randomly. I didn't found any pattern for their blocks.
Everytime I see they blocked my application I manually enter to the tiktok in the browser with same cookie my application are using and I solve the captcha. After that my application starts collecting data again. I had cases where my application collected only for 1 minutes after solving the captcha and other that the application collected for 13 hours straight. Apparently more time pass more I was being blocked:
Green marks -> manually solved the captcha using the same cookie.
The verifyFp itself changed for my browser so I think they have some mecanism to create new cookie after some time. I also tried to use the same verifyFp I was using in the browser (manually changed the cookie) and after that the discover page does not works for me even after the captcha resolution:
I noticed that other API methods still works (for example user recommendation). They are deploying this on each api endpoint right now and not all endpoints has this protection or they will protect only some of TikTok endpoints.
By that I think the only solution will definitely solve this problem will be the automatic captcha solver and make the signer generate a new valid verifyFp when its necessary (of course people can generate it one time and re-use while its valid).
Don't know if any of this help. If I have some time I will try to create a PoC for captcha resolution but Its not my area of expertise.
Although I haven't checked which captcha provider TikTok utilizes, this tip may still be valid. Check to see if the captcha has an audio captcha mode, which then you can easily utilize a speech recognition module to automatically solve it (speech recognition is a lot better than most people realize, even with the 'white noise'). I did this for another unrelated website that uses FunCaptcha a few weeks ago and it works smoothly. Definitely an avenue worth pursuing if it offers an audio mode.
Would it be possible to write a script to solve the captcha? Something that takes a picture of the captcha and selects a block of pixels and compares them to a bunch of same sized blocks shifted over.
Would it be possible to write a script to solve the captcha?
Yes I linked an article above that might be helpful. I might have time over thanksgiving break to experiment with this further.
Here's some websites that might be useful but are outdated so may require changes
Would it be possible to write a script to solve the captcha?
Yes I linked an article above that might be helpful. I might have time over thanksgiving break to experiment with this further.
Here's some websites that might be useful but are outdated so may require changes
I have solution for solve captcha. (Get X and Y coords of puzzle). But TikTok recording mouse moves and clicks on buttons. the problem is that the payload is generated from this data. and I don't know how to emulate it. the source code of the captcha is too confusing
I don't know how to emulate it
I think you'll need to launch a browser using pyppeteer/playwright/selenium to solve the captcha by feigning a mouse move and then grabbing that cookie after you solve it
Я не знаю, как это подражать
Я думаю, вам нужно запустить браузер с использованием pyppeteer / playwright / selenium, чтобы решить капчу, симулируя движение мыши, а затем схватив этот файл cookie после его решения.
The thing is, I want to make a solution without using a browser (For better performance) Even if you do this, you need to understand how the captcha logic works in order to at least run it.
Started digging into this more and found the captcha html page, it's weird though because not every s_v_web_id cookie is valid from there :\
When I solve the captcha from https://sf16-scmcdn-va.ibytedtos.com/goofy/secsdk-captcha/va/2.15.21/index.html and then use the s_v_web_id cookie from the sf16-scmcdn-va.ibytedtos.com domain (as opposed to the one from www.tiktok.com), calls to trending work. But when I try byUsername, getUser fails with an IndexError exception when it tries to scrape the html.
Uhm you might wanna check out nightly branch they're not doing much prevention on the "t.tiktok.com" url with cookies so just like install that code or change BASE_URL in your code to "t.tiktok" instead of "m.tiktok" it's temporary for sure but eh scrape while you can
Seems like t.tiktok.com works for most of the data scraping functions, although it does not work for downloading videos.
As I mentioned in #397 I have a working solution for solving captchas (puzzle sliders). I will be able to share the tool (Python and PyTorch based, something like this) if someone tells me exact requests limits, i.e. I need a solution for this carcabot/tiktok-signature issue#105. I did some research, I already have where to start.
https://github.com/tolgatasci/musically-tiktok-api-python maybe someone will understand this code. @davidteather @nuqz maybe help you
I've written a register app to register tiktok accounts, exports cookies, another login app to relogin and export cookies with auto slide captcha resolver, a signer server to sign every api endpoints that requires signature, and a comment app via sending http requests.
Preview for signer app (several sensitive arguments hidden here): curl --header "Content-Type: application/json" \ --request POST \ --data '{"url":"https://www.tiktok.com/comment/publish", "referer":"https://www.tiktok.com/", "ua":"ua"}' \ http://120.55.164.73:8888/api/sign
I've written a register app to register tiktok accounts, exports cookies, another login app to relogin and export cookies with auto slide captcha resolver, a signer server to sign every api endpoints that requires signature, and a comment app via sending http requests.
I wonder how you wrote the solver
I wrote it using opencv-python lib(pip install opencv-python)
I wrote it using opencv-python lib(pip install opencv-python)
oh cool. can you show how you did? I want to study the code how to solve captcha. i tried to implement but failed
Like this:https://pythonmana.com/2021/02/20210223164623700z.html
Thank you
U r welcome.
Hi!
I bring to the topic the solution made by @carcabot which obtain acces to the webpage by generating signatures dynamically, thus without having to resolve any captcha https://github.com/carcabot/tiktok-signature.
I have made small tests with the package and I can parse different TikTok urls automatically without having any Captcha error. It might help solving the problem for scrapping TikTok author's information.
Hope it helps!
Hi!
I bring to the topic the solution made by @carcabot which obtain acces to the webpage by generating signatures dynamically, thus without having to resolve any captcha https://github.com/carcabot/tiktok-signature.
I have made small tests with the package and I can parse different TikTok urls automatically without having any Captcha error. It might help solving the problem for scrapping TikTok author's information.
Hope it helps!
can you please explain what is tiktok signature have to do along with captcha i noticd that on my chrome there is no captcha but on burp suit always there is acptcha and i do not know why any help please?
The current solution isn't great as verifyFp probably expires after some given amount of time. We need to find a way to solve the captcha for a given verifyFp cookie.
More information
344
342