Russell-Newton / TikTokPy

Extract data from TikTok without needing any login information or API keys.
https://pypi.org/project/tiktokapipy/
MIT License
213 stars 28 forks source link

[BUG] TikTok seems to have changed things #82

Open bobcogneato opened 11 months ago

bobcogneato commented 11 months ago

Describe the bug TikTok got rid of #SIGI_STATE and replaced it with #__UNIVERSAL_DATA_FOR_REHYDRATION__, now the data exists in ex /__DEFAULT_SCOPE__/webapp.video-detail/itemInfo and has different keys so the mapper doesn't work anymore.

To Reproduce Steps to reproduce the behavior:

  1. Scrape any page. Timeout occurs because SIGI_STATE isn't attached.

Expected behavior Well, you know...

Version Information Please include what versions of pydantic, playwright, and tiktokapipy you have installed (can be found with pip freeze).

playwright==1.30.0
playwright-stealth==1.0.5
pydantic==2.5.2
pydantic_core==2.14.5
tiktokapipy==0.2.4.post2

System Information Please provide information about the system you're running the library on. What's the OS? Is it a desktop, server, VM, or container? Various linux (ubuntu/debian) machines Region Information Where in the world are you located? If you're not in the US, it's likely that TikTok treats your region differently, which may present some challenges. US Additional context Add any other context about the problem here.

RobertModus commented 11 months ago

As of today, i only see static and user data in the new state object, eg __$UNIVERSAL_DATA$__.__DEFAULT_SCOPE__["webapp.user-detail"] but no webapp.video-detail. Video data should be gathered through the DeferredItemListIterator(self._api, "post", self.sec_uid) magic anyways, so shmaybe not a big change needed?

Russell-Newton commented 11 months ago

I'll need some time to address this. At the moment, I'm pretty busy with research and various applications, but I'll try to get to this soon.