Russell-Newton / TikTokPy

Extract data from TikTok without needing any login information or API keys.
https://pypi.org/project/tiktokapipy/
MIT License
213 stars 28 forks source link

[BUG] seeing 2 different errors in log using version 0.2.1 #59

Closed stolenvw closed 1 year ago

stolenvw commented 1 year ago

tiktokapipy==0.2.1 playwright==1.32.1 pydantic==2.0.2 pydantic_core==2.1.2

Error 1. Only seen this one happen 3 times in past 9 hours

Failed: TypeError: 'NoneType' object is not iterable
Traceback (most recent call last):
  File "/home/user/TwitchBot/bot/modules/tiktok.py", line 27, in get_tiktoks
    async for video in user.videos:
  File "/home/user/.venv/lib/python3.10/site-packages/tiktokapipy/util/deferred_collectors.py", line 90, in __anext__
    await self._fetch_async()
  File "/home/user/.venv/lib/python3.10/site-packages/tiktokapipy/util/deferred_collectors.py", line 215, in _fetch_async
    for item in converted.item_list:
TypeError: 'NoneType' object is not iterable

Error 2. This one has happened 72 times in past 9 hours

JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
  File "/home/user/TwitchBot/bot/modules/tiktok.py", line 27, in get_tiktoks
    async for video in user.videos:
  File "/home/user/.venv/lib/python3.10/site-packages/tiktokapipy/util/deferred_collectors.py", line 90, in __anext__
    await self._fetch_async()
  File "/home/user/.venv/lib/python3.10/site-packages/tiktokapipy/util/deferred_collectors.py", line 207, in _fetch_async
    raw = await make_request_async(
  File "/home/user/.venv/lib/python3.10/site-packages/tiktokapipy/util/queries.py", line 163, in make_request_async
    return await sign_and_get_request_async(
  File "/home/user/.venv/lib/python3.10/site-packages/tiktokapipy/util/signing.py", line 600, in sign_and_get_request_async
    json = await resp.json()
  File "/home/user/.venv/lib/python3.10/site-packages/playwright/async_api/_generated.py", line 617, in json
    return mapping.from_maybe_impl(await self._impl_obj.json())
  File "/home/user/.venv/lib/python3.10/site-packages/playwright/_impl/_network.py", line 547, in json
    return json.loads(await self.text())
  File "/usr/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Russell-Newton commented 1 year ago

These errors should no longer appear in version 0.2.2. The first error was because I tried to iterate over something that could be None before checking if it is actually None. The second is a byproduct of not raising StopIteration at the correct time and the occasional credentials issue. This second one spits out a warning now in case it happens again, telling you to try again if you believe it should be able to continue iteration.

If you would prefer that the iterator raise a TikTokAPIError instead of just issuing a warning, let me know.

stolenvw commented 1 year ago

hmmm, this is the start of code i use. By my thinking it should not be trying to iterate over something when i am telling it to only get 1. But guessing it has to do some iterating to get to that one video.

warning should be fine (hoping it dont happen alot and flood logs)

async with AsyncTikTokAPI(navigation_retries=1) as api:
                    user = await api.user(i.username, video_limit=1)
                    async for video in user.videos:
Russell-Newton commented 1 year ago

The way it works is that it has to grab videos from TikTok's API, which it does in bulk. It throws out any videos you don't want (set by the video_limit). It does some extra stuff on these bulk videos, which includes some iteration. This was where the TypeError came up.

You can suppress warnings of a specific type, in the case for tiktokapipy, the warning type is TikTokAPIWarning.

It's worth noting that if you're only getting one video, it's possible that no video will actually get fetched if the erroneous situation comes up. This error is suppressed, so it won't break anything, but you'll see that the async for loop won't actually run on anything.

stolenvw commented 1 year ago

Im okay with the loop not running on things now and then, right now its checking every 10min for new videos from 2 users, first thing i do in the async for loop is to check the video.id against the stored video.id from the last video from that user i got.

cdove99 commented 1 year ago

TypeError: 'NoneType' object is not iterable seems to be happening on video comments as well.

Russell-Newton commented 1 year ago

All of these issues should be fixed in 0.2.4. Closing as complete.