[BUG] Unable to do any kind of request

[reddere]

Describe the bug I am unable to do any kind of request. This error pops up: tiktokapipy.TikTokAPIError: Data scraping unable to complete in 30.0s (retries: 0)

To Reproduce Here is the code you can run:

from tiktokapipy.api import TikTokAPI 

with TikTokAPI() as api:
    video = api.video('https://www.tiktok.com/@ahormozi/video/7242371997324315946')
    print(video)

Expected behavior print out the scraped Videoobject

Version Information pydantic==1.10.9 playwright==1.35.0 playwright==1.35.0 tiktokapipy==0.1.13.post1 Please include what versions of pydantic, playwright, and tiktokapipy you have installed (can be found with pip freeze).

Additional context Here is the full traceback:

Traceback (most recent call last): File "C:\Users\path\venv\lib\site-packages\tiktokapipy\api.py", line 430, in _scrape_data data = self._extract_and_dump_data(content, extras_json, data_model) File "C:\Users\path\venv\lib\site-packages\tiktokapipy\api.py", line 469, in _extract_and_dump_data parsed = data_model.parse_raw(data) File "pydantic\main.py", line 549, in pydantic.main.BaseModel.parse_raw File "C:\Users\path\venv\lib\site-packages\tiktokapipy\models\raw_data.py", line 130, in parse_obj return super().parse_obj(obj) File "pydantic\main.py", line 526, in pydantic.main.BaseModel.parse_obj File "pydantic\main.py", line 341, in pydantic.main.BaseModel.init pydantic.error_wrappers.ValidationError: 1 validation error for VideoResponse VideoPage field required (type=value_error.missing) Traceback (most recent call last): File "c:\Users\path\main.py", line 11, in video = api.video('https://www.tiktok.com/@ahormozi/video/7242371997324315946') File "C:\Users\path\venv\lib\site-packages\tiktokapipy\api.py", line 348, in video
response, api_extras = self._scrape_data( File "C:\Users\path\venv\lib\site-packages\tiktokapipy\api.py", line 445, in _scrape_data raise TikTokAPIError( tiktokapipy.TikTokAPIError: Data scraping unable to complete in 30.0s (retries: 0)

[Russell-Newton]

A few questions:

1. What region are you in?
2. Can you try running the API with headless=False in the constructor and taking a recording of your screen so I can see what the browser sees? Screenshots of the loaded page could also work, but it's tricky to time the screenshot capture.

As a side note, I'm working on a new version of the library that should hopefully

[jasonxubw]

Getting the same error with api.challenge() .

[Russell-Newton]

@jasonxubw It's unlikely you're getting the exact same error. Please copy the traceback.

@jasonxubw @reddere if both of you could set a data dump file in the API constructor and send the dumped data here, that would be very helpful. I also need to know what region you both are in and what operating system you're both using.

[jasonxubw]

Region: U.S. O.S.: Windows 10 Data dump: https://jpst.it/3h7b7

My code:

    with TikTokAPI(data_dump_file="dump") as api:
        challenge = api.challenge("fyp", 3)
        for video in challenge.videos:
            print(video.stats.play_count)

Full stack trace:

Traceback (most recent call last):
  File "C:\Users\path\miniconda3\lib\site-packages\tiktokapipy\api.py", line 430, in _scrape_data
    data = self._extract_and_dump_data(content, extras_json, data_model)
  File "C:\Users\path\miniconda3\lib\site-packages\tiktokapipy\api.py", line 469, in _extract_and_dump_data
    parsed = data_model.parse_raw(data)
  File "pydantic\main.py", line 549, in pydantic.main.BaseModel.parse_raw
  File "pydantic\main.py", line 526, in pydantic.main.BaseModel.parse_obj
  File "pydantic\main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for ChallengeResponse
ChallengePage
  none is not an allowed value (type=type_error.none.not_allowed)
Exception in callback SyncBase._sync.<locals>.<lambda>(<Task cancell...twork.py:187>>) at C:\Users\path\miniconda3\lib\site-packages\playwright\_impl\_sync_base.py:100
handle: <Handle SyncBase._sync.<locals>.<lambda>(<Task cancell...twork.py:187>>) at C:\Users\path\miniconda3\lib\site-packages\playwright\_impl\_sync_base.py:100>
Traceback (most recent call last):
  File "C:\Users\path\miniconda3\lib\asyncio\events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "C:\Users\path\miniconda3\lib\site-packages\playwright\_impl\_sync_base.py", line 100, in <lambda>
    task.add_done_callback(lambda _: g_self.switch())
  File "C:\Users\path\miniconda3\lib\site-packages\playwright\_impl\_helper.py", line 273, in impl
    result = cast(
  File "C:\Users\path\miniconda3\lib\site-packages\playwright\_impl\_impl_to_api_mapping.py", line 123, in wrapper_func
    return handler(
  File "C:\Users\path\miniconda3\lib\site-packages\tiktokapipy\api.py", line 381, in capture_api_extras
    response = route.request.response()
  File "C:\Users\path\miniconda3\lib\site-packages\playwright\sync_api\_generated.py", line 326, in response
    return mapping.from_impl_nullable(self._sync(self._impl_obj.response()))
  File "C:\Users\path\miniconda3\lib\site-packages\playwright\_impl\_sync_base.py", line 104, in _sync
    return task.result()
asyncio.exceptions.CancelledError
Exception in callback SyncBase._sync.<locals>.<lambda>(<Task cancell...twork.py:187>>) at C:\Users\path\miniconda3\lib\site-packages\playwright\_impl\_sync_base.py:100
handle: <Handle SyncBase._sync.<locals>.<lambda>(<Task cancell...twork.py:187>>) at C:\Users\path\miniconda3\lib\site-packages\playwright\_impl\_sync_base.py:100>
Traceback (most recent call last):
  File "C:\Users\path\miniconda3\lib\asyncio\events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "C:\Users\path\miniconda3\lib\site-packages\playwright\_impl\_sync_base.py", line 100, in <lambda>
    task.add_done_callback(lambda _: g_self.switch())
  File "C:\Users\path\miniconda3\lib\site-packages\playwright\_impl\_helper.py", line 273, in impl
    result = cast(
  File "C:\Users\path\miniconda3\lib\site-packages\playwright\_impl\_impl_to_api_mapping.py", line 123, in wrapper_func
    return handler(
  File "C:\Users\path\miniconda3\lib\site-packages\tiktokapipy\api.py", line 381, in capture_api_extras
    response = route.request.response()
  File "C:\Users\path\miniconda3\lib\site-packages\playwright\sync_api\_generated.py", line 326, in response
    return mapping.from_impl_nullable(self._sync(self._impl_obj.response()))
  File "C:\Users\path\miniconda3\lib\site-packages\playwright\_impl\_sync_base.py", line 104, in _sync
    return task.result()
asyncio.exceptions.CancelledError
Traceback (most recent call last):
  File "C:/Users/path/Documents/GeorgiaTech/Deep-Learning-2/final/data/main.py", line 123, in <module>
    main()
  File "C:/Users/path/Documents/GeorgiaTech/Deep-Learning-2/final/data/main.py", line 26, in main
    download_video()
  File "C:/Users/path/Documents/GeorgiaTech/Deep-Learning-2/final/data/main.py", line 71, in download_video
    challenge = api.challenge("fyp", 3)
  File "C:\Users\path\miniconda3\lib\site-packages\tiktokapipy\api.py", line 291, in challenge
    response, api_extras = self._scrape_data(
  File "C:\Users\path\miniconda3\lib\site-packages\tiktokapipy\api.py", line 445, in _scrape_data
    raise TikTokAPIError(
tiktokapipy.TikTokAPIError: Data scraping unable to complete in 30.0s (retries: 0)

[reddere]

1. What region are you in?

Italy, EU

2. Can you try running the API with headless=False in the constructor and taking a recording of your screen so I can see what the browser sees? Screenshots of the loaded page could also work, but it's tricky to time the screenshot capture.

This is the last page that is shown before crashing:

[reddere]

@reddere if both of you could set a data dump file in the API constructor and send the dumped data here, that would be very helpful. I also need to know what region you both are in and what operating system you're both using.

Sure @Russell-Newton , just sent it to your email as I cannot upload json files here. Region is Italy and Windows 10 is the OS

[Russell-Newton]

@jasonxubw your issue seems to be related to a change TikTok made with how challenges are loaded. I think this should be fixed with the v0.2 changes I've been working on. @reddere your issue seems to be caused by TikTok sending you to a login page. I'm not sure what might be causing this, I can't seem to recreate it, but it may be fixed by the v0.2 changes.

It could be worth trying out the new stuff:

pip install -U git+https://github.com/Russell-Newton/TikTokPy.git@v0.2-overhaul

I removed some features, but what you two are doing shouldn't be affected.

[reddere]

Thanks @Russell-Newton, hope you could find more information with dump file I sent you on the email. Also, I tried updating with the CLI command you sent and an ImportError pops up:

Traceback (most recent call last):
  File "c:\Users\path\using_tiktokpy.py", line 1, in <module>
    from tiktokapipy.api import TikTokAPI 
  File "C:\Users\path\venv\lib\site-packages\tiktokapipy\api.py", line 32, in <module>
    from tiktokapipy.models import DeferredIterator, TikTokDataModel
  File "C:\Users\path\venv\lib\site-packages\tiktokapipy\models\__init__.py", line 24, in <module>
    from pydantic.main import ModelMetaclass
ImportError: cannot import name 'ModelMetaclass' from 'pydantic.main' (C:\Users\path\venv\lib\site-packages\pydantic\main.py)

Probably it's because tiktokpy is expecting a precise version of pydantic, as of now pydantic's version is 2.0b3

[Russell-Newton]

@reddere it actually looks like tiktokapipy didn't update properly because I didn't set the versioning correctly in this branch. Try the install command from above again and make sure that tiktokapipy is set to version 0.2.0a1 afterwards

[reddere]

Updated and this works, thanks @Russell-Newton! There are some bad news though, all the links provided are accessible except for the ones related to the actual video, which are the links given by "play_addr" and "download_addr", when I make a request on my browser it returns "Access Denied".

Also, I tried using tiktokpy to fetch videos from a public user profile with this code

from tiktokapipy.api import TikTokAPI 

with TikTokAPI(headless=False, data_dump_file='here.txt') as api:
    user = api.user('silvio.berlusconi', video_limit=2)
    for video in user.videos:
        print(video)

This does not work as it returns the same error given by first issue, what can we do to fix it? tiktokapipy.TikTokAPIError: Data scraping unable to complete in 30.0s (retries: 0)

I will send your email the dump file for this one

[Russell-Newton]

...all the links provided are accessible except for the ones related to the actual video, which are the links given by "play_addr" and "download_addr", when I make a request on my browser it returns "Access Denied".

This has been an issue with video downloading in the past. It's a tricky issue for me to debug because I can't recreate it consistently. See #35 and #44.

As for the other issue, this seems to be once again related to TikTok sending you to a login page. I'm not sure why you're the first person who's posted about this issue, but I'll see what I can do to address it. One thing you could try is setting navigation_retries=2 in the API constructor.

[Russell-Newton]

Also fingers crossed the changes I just pushed should work. You can try reinstalling the library, and it should be set to version 0.2.0a2 now. The changes attempt to bypass the login page by going immediately to the redirect link specified by the login page. I've no idea if it will actually work, but it's worth a shot.

[reddere]

I updated the library to version 0.2.0a2 and tried both running with and without navigation_retries=2 in the API constructor, still no success. Same error: tiktokapipy.TikTokAPIError: Data scraping unable to complete in 30.0s (retries: 2)

Well, at least I gave it a shot. Is there anything I can do to help you figure this out even further? I can screenshare on Discord too (Discord username: @robb0312) @Russell-Newton

[Russell-Newton]

@reddere when you go to TikTok in incognito mode on a normal browser, does it try to automatically take you to a login page when you view an account's page?

[reddere]

Yep @Russell-Newton ! Tried requesting both with a tiktok video URL and a tiktok profile URL, both redirect me straight to login page.

[Russell-Newton]

That's really interesting. Some followup questions then @reddere:

1. If you refresh, does it take you to the correct page or stay on the login page?
2. Can you try sending me a copy of the login page HTML? You should be able to get it with CTRL+S.
3. Can you share the link of the login page when it auto-navogates you to it?

[reddere]

Sure @Russell-Newton

1. If you refresh, does it take you to the correct page or stay on the login page?

Nope. Tested on both video URL and profile URL. Tried refreshing more times too, still on login page.

2. Can you try sending me a copy of the login page HTML? You should be able to get it with CTRL+S.

Sure, but I cannot upload the htm file here, I just sent the file to your mail.

3. Can you share the link of the login page when it auto-navigates you to it?

Sure, here it is: https://www.tiktok.com/login?redirect_url=https%3A%2F%2Fwww.tiktok.com%2F%40silvio.berlusconi&lang=en&enter_method=mandatory

The link is generated by the code pasted at #issuecomment-1613360584

[Russell-Newton]

@reddere I just pushed a new change as v0.2.0a3, this time with the goal of attempting to stop the login page redirect. This time, there's also a TikTokAPIWarning that informs you that it got sent to a login page. Because I can't seem to recreate this issue on my end, you'll have to try again, and this time it'll be a bit easier to tell what's going on with that warning.

[reddere]

@Russell-Newton Cool, I updated to v0.2.0a3, still not able to get the data. If this can help, if you set a VPN to Italy, you should be able to encounter this problem too. On my end, when I set my VPN to Atlanta (US), it does not redirect me to the login page. It just shows me the video. So yeah, even if this seems very odd, you can recreate this by setting a VPN on Italy or to any other country that has this issue.

[Russell-Newton]

I think I know exactly what the issue is. TikTok has had to instate changes and is instating changes in the EU to comply with it's data privacy regulations. Therese regulations are much more strict than their US counterparts. I guess to comply with the mandatory opt-in standard set by the EU and to achieve pseudonymisation, TikTok currently requires all EU users to log in to use their platform.

With this in mind, there isn't anything I can do. The only solution is to use a VPN or proxy (proxy information can be passed to the API, see the constructor documentation). At the moment then, some features of TikTokPy won't work 100% within the EU without a VPN or proxy.

[Russell-Newton]

This issue should be resolved in the official version 0.2.0 release. The newly discovered issue of EU library users being unable to access TikTok user information is something that I won't be able to fix unless I can adapt the user collection to also work in line with #21, which is a WIP. I'm closing this issue as resolved.

[reddere]

Sure. thanks.