search

AAndyProgram / SCrawler

🏳️‍🌈 Media downloader from any sites, including Twitter, Reddit, Instagram, Threads, Facebook, OnlyFans, YouTube, Pinterest, PornHub, XHamster, XVIDEOS, ThisVid etc.
https://discord.gg/uFNUXvFFmg
GNU General Public License v3.0
1.23k stars 81 forks source link

TikTok only downloads the latest 30 posts #75

Closed AAndyProgram closed 1 year ago

AAndyProgram commented 1 year ago

TikTok is currently parsed through HTML. I tried to write a request which returns me a json response. But with any combination of options, I got the same response. The request has a cursor parameter. This parameter means date in json format. Every response returns a cursor value that developer should use to get the next media data block. But this parameter is deprecated on their side or doesn't work at all. I haven't seen any requests in the browser's DevTools containing this parameter.

If you know how to solve this issue, you can post a suggestion in a comment on this post.

OMGerEDU commented 1 year ago

Hey there, I think I know your problem, The problem is that the "Lazy loader" only loads the X videos the website set, I had the same problem with Twitter on the past, what you need to do is to set a task to browse down on the website until it find new posts, meanwhile save all posts (I used Xpath and compare titles VS old ones, but Tiktok is different)

AAndyProgram commented 1 year ago

Any code sample (link or code text) to see how it works?

AAndyProgram commented 1 year ago

Fixed. Please update to the latest release.