search

freezingdata / snh-userscript-tiktok

SNH-Script to acquire data from TikTok
GNU General Public License v3.0
8 stars 0 forks source link

TikTok Module only collects first 30 Videos #1

Closed drangzt1450 closed 2 years ago

drangzt1450 commented 2 years ago

The module only seems to collect the first 30 TikTok videos even when logged into TikTok. I replaced the original script (that had the same problem) with the newest and it has the same problem. User Agent configured as suggested to be most current.

freezingdata commented 2 years ago

Hello drangzt450,

one week ago, we've fixed this bug. TikTok recently changed an API entry point from m.tiktok.com to www.tiktok.com. Did you've used the recent commit of the tiktok modul? The open source modules dosn't update itself by the SNH update system, you have to replace it manually.

Fix

drangzt1450 commented 2 years ago

I replaced the older TikTok module with the new TikTok module and the same issue remains...it will only collect the first 30 Videos.

freezingdata commented 2 years ago

I've released a modul version with additional verbose output. Please do the following

  1. Replace the current modul with the new one
  2. Start the SNH an create a Scriptcase with TikTok modul 3.0.3
  3. LogIn in TikTok and open the profil of your choice (I used https://www.tiktok.com/@sheyla_hdz to test)
  4. Collect Postings
  5. After the collection is finished, Export the logfile. (Help->Export Supportfile)
  6. Please upload the file.

Thanks

drangzt1450 commented 2 years ago

I followed the directions provided. I verified that the module was 3.0.3. The same issue when I scraped a random TikTok profile...stopped scraping at 30 videos. Here is the support file

(File Removed for privacy reasons)

freezingdata commented 2 years ago

Hello, I can see the problem. The API calls of the website seems not to be captured. Perhaps TikTok has a different API Endpoint than expected. Next week, I will send you a special version of the module, to determine the possible problem.

Thanks

freezingdata commented 2 years ago

We updated the TikTok Modul to avoide hopefully the issues on yout side. Please test the new commit.

drangzt1450 commented 2 years ago

The new module commit appears to be working I was able to collect all 157 Videos from my target. I had to refresh the page several times in order for the timeline option to activate but when it did, it worked!

freezingdata commented 2 years ago

Ok, thank you for the testing. We will take a deeper look into the timeline detection behaviour.