[BUG] Pydantic Validation Error (APIResponse.hasMore)

Describe the bug I'm utilizing FastAPI in conjunction with aiohttp to download TikTok videos. Recently I've been dealing with a strange Pydantic data validation error. It happens on and off. By this I mean some times I can issue many requests and successfully fetch and download videos by hashtag, other times I get met with this error and am unable to get any thing back. Then it'll start working again without any code changes. I'm not sure if this is a mismatch between TikTokPy's Pydantic Model for the expected response and TikTok's actual response or if it's TikTok possibly rate limiting.

This is the error:

{
  "detail": 
    "An error occurred while retrieving videos: 1 validation error for APIResponse
    hasMore  
    Field required [
      type=missing,
      input_value={'log_pb': {'impr_id': '2...10101, 'status_msg': ''},
      input_type=dict
    ]
  For further information visit https://errors.pydantic.dev/2.4/v/missing"
}

This is the chunk of code that fetches and downloads the videos, which is based off the documentation:

async def download_video(self, video: str, hashtag: str):
    if not video.image_post:
        logger.info("Attempting to download video: %s", video.id)
        filename = f"{hashtag.lower()}-{video.id}.mp4"
        url = video.video.download_addr
        file_path = os.path.join(self.video_directory, filename)
        try:
            async with self.session.get(
                url, headers={"referer": "https://www.tiktok.com/"}
            ) as response:
                if response.status != 200:
                    logger.error(
                        "Received HTTP %s when trying to fetch %s. Response Text: %s",
                        response.status,
                        url,
                        await response.text(),
                    )
                    return None

                with open(file_path, "wb") as file:
                    file.write(await response.read())
                logger.info("Video: %s download successful", filename)
                return file_path
        except Exception as e:
            logger.error("Error while saving the file: %s", e)
            return None

async def get_videos_by_hashtag(self, hashtag: str):
    local_videos = self.search_videos_local(hashtag)
    if local_videos:
        logger.info(
            "Found locally saved videos for hashtag %s: %s", hashtag, local_videos
        )
        return local_videos

    async with AsyncTikTokAPI() as api:
        try:
            logger.info("Attempting to retrieve videos for hashtag: #%s", hashtag)
            challenge = await api.challenge(f"{hashtag}", video_limit=3)
        except Exception as e:
            logger.error("Error while making call to tiktokapipy: %s", e)
            raise
        tasks = [
            self.download_video(video, hashtag) async for video in challenge.videos
        ]
        video_paths = await asyncio.gather(*tasks)
        return video_paths

I'm downloading the videos concurrently so I re-use the same aiohttp client session:


async def __aenter__(self):
    async with AsyncTikTokAPI() as api:
        self.session = ClientSession(
            cookies={
                cookie["name"]: cookie["value"]
                for cookie in await api.context.cookies()
                if cookie["name"] == "tt_chain_token"
            }
        )
    return self

async def __aexit__(self):
    if self.session:
        await self.session.close()

To Reproduce Steps to reproduce the behavior: It's hard to reproduce consistently. Sometimes I'm testing locally and it works and then all the sudden will start giving me that error over and over. Sometimes, I'll start my machine cold and spin up uvicorn/FastAPI and it'll start failing on the first try, continue failing for an hour+ and then start working again.

Expected behavior It should consistently fetch, download, and serve the videos.

Version Information Please include what versions of pydantic, playwright, and tiktokapipy you have installed: pydantic==2.4.0 pydantic-extra-types==2.1.0 pydantic-settings==2.0.3 pydantic_core==2.10.0 playwright==1.38.0 tiktokapipy==0.2.3 fastapi==0.103.2 uvicorn==0.23.2

System Information Local: MacOS 14.0 Sonoma

Docker: Base Image: ubuntu20.04

Region Information London, UK

Russell-Newton / TikTokPy

[BUG] Pydantic Validation Error (APIResponse.hasMore) #76