Russell-Newton / TikTokPy

Extract data from TikTok without needing any login information or API keys.
https://pypi.org/project/tiktokapipy/
MIT License
213 stars 28 forks source link

[QUESTION] Video Downloading Error 403 #35

Open alexempie opened 1 year ago

alexempie commented 1 year ago

Cannot download any video by down_addr from challenge collection. It returns 403 Forbidden error

for video in challenge.videos requests.get(video.video.download_addr)

How to fix that?

Russell-Newton commented 1 year ago

I'm not experiencing this problem, so I don't have a good way to debug it. I need some more information:

  1. Is the code sample you provided exactly what you're using and experiencing issues with?
  2. What country/region are you in?
  3. Are you able to access TikTok and watch videos normally through a desktop browser?
Russell-Newton commented 1 year ago

Related to #24

alexempie commented 1 year ago

my code: with TikTokAPI(scroll_down_time=1, emulate_mobile=False) as api: challenge = api.challenge("Cat") for video in challenge.videos: response = requests.get(video.video.download_addr)

  1. In Georgia.
  2. I can watch videos in browser In response i have fully video info with url but i don't have permission to it image

download_addr https://v16-webapp-prime.tiktok.com/video/tos/alisg/tos-alisg-pve-0037c001/e7ead08247c446f8a3aaf5d164746e5c/?a=1988&ch=0&cr=0&dr=0&lr=tiktok_m&cd=0%7C0%7C1%7C0&cv=1&br=2038&bt=1019&cs=0&ds=3&ft=4fUEKMvx8Zmo0Mn2M64jVepUPpWrKsdm&mime_type=video_mp4&qs=0&rc=aWc6ZGVlZ2U5ZTZkOmg1O0BpanZ0cG9zOXdteTMzZTczM0BeXjEzMC0uNS0xMi1fNTEvYSNoMGA1amZoNmtfLS1hMTRzcw%3D%3D&btag=80000&expire=1677017561&l=2023022116123106BA019B6D2EB018D9A0&ply_type=2&policy=2&signature=116d6e7460b0a954743235d911dbd2d5&tk=tt_chain_token

play_addr: https://v16-webapp-prime.tiktok.com/video/tos/alisg/tos-alisg-pve-0037c001/e7ead08247c446f8a3aaf5d164746e5c/?a=1988&ch=0&cr=0&dr=0&lr=tiktok_m&cd=0%7C0%7C1%7C0&cv=1&br=2038&bt=1019&cs=0&ds=3&ft=4fUEKMvx8Zmo0Mn2M64jVepUPpWrKsdm&mime_type=video_mp4&qs=0&rc=aWc6ZGVlZ2U5ZTZkOmg1O0BpanZ0cG9zOXdteTMzZTczM0BeXjEzMC0uNS0xMi1fNTEvYSNoMGA1amZoNmtfLS1hMTRzcw%3D%3D&btag=80000&expire=1677017561&l=2023022116123106BA019B6D2EB018D9A0&ply_type=2&policy=2&signature=116d6e7460b0a954743235d911dbd2d5&tk=tt_chain_token

image

alejandroarmas commented 1 year ago

I am also getting a 403 error.

My code:

 async with AsyncTikTokAPI(emulate_mobile=False, navigation_retries=3, navigation_timeout=60) as api:
       async for video in user.videos:
              dwn_link = await self.save_video(video)

...

async def save_video(self, video: Video):
        async with aiohttp.ClientSession() as session:
            async with session.get(video.video.download_addr) as resp:
                print(f'{resp=}')
URL = https://tiktok.com/@kyliejenner/video/7202869446044650795
resp=<ClientResponse(https://v16-webapp-prime.us.tiktok.com/video/tos/useast5/tos-useast5-pve-0068-tx/7cccbd62e1f9451b8b14f8b10b280a99/?a=1988&ch=0&cr=0&dr=0&lr=tiktok_m&cd=0%7C0%7C1%7C0&cv=1&br=1474&bt=737&cs=0&ds=3&ft=4KJMyMzm8Zmo0FVD1A4jVwoWdpWrKsdm&mime_type=video_mp4&qs=0&rc=NTg7Ozw8ZWY5NTY5NjY0Z0BpM2VybTc6Zjt1aTMzZzczNEA2YF5iYDFfXzExYDAyYi9jYSNzc2g0cjRva25gLS1kMS9zcw%3D%3D&expire=1678277529&l=20230308061154A6172C27ED6C374EF78B&policy=2&signature=8491010f44ef385623181ab2cc6d805e&tk=tt_chain_token) [403 Forbidden]>
<CIMultiDictProxy('Server': 'AkamaiGHost', 'Mime-Version': '1.0', 'Content-Length': '408', 'Expires': 'Wed, 08 Mar 2023 06:11:56 GMT', 'Date': 'Wed, 08 Mar 2023 06:11:56 GMT', 'X-Cache': 'TCP_DENIED from a23-67-78-238.deploy.akamaitechnologies.com (AkamaiGHost/11.0.0-46340752) (-)', 'Connection': 'keep-alive', 'x-response-cache': '', 'x-tt-trace-tag': 'id=16;cdn-cache=miss;type=static', 'Server-Timing': 'cdn-cache; desc=MISS, edge; dur=0, origin; dur=0', 'Content-Type': 'video/mp4', 'Access-Control-Expose-Headers': 'Content-Length,Content-Range,content-type,expires,last-modified,via,X-Cache,x-response-cache,x-response-sinfo,x-response-cinfo', 'Accept-Ranges': 'bytes', 'Access-Control-Allow-Headers': 'range', 'Access-Control-Allow-Credentials': 'true', 'X-Akamai-Request-ID': '42391e2d')>
alejandroarmas commented 1 year ago

Is there any resolution? Thanks!

Russell-Newton commented 1 year ago

I haven't had a whole lot of time to try to debug this, but I can share my initial thoughts:

It's likely you may need to pass the cookies TikTokPy stores to aiohttp or requests.

@alejandroarmas can you try modifying your code to have this:

async with AsyncTikTokAPI(emulate_mobile=False, navigation_retries=3, navigation_timeout=60) as api:
       async for video in user.videos:
              dwn_link = await self.save_video(video, {cookie['name']: cookie['value'] for cookie in api.context.cookies()})

...

async def save_video(self, video: Video, cookies={}):
        async with aiohttp.ClientSession(cookies=cookies) as session:
            async with session.get(video.video.download_addr) as resp:
                print(f'{resp=}')

Note the changes here pass the api's cookies to the client session.

@alexempie can you try changing your code to use

response = requests.get(video.video.download_addr, cookies={cookie['name']: cookie['value'] for cookie in api.context.cookies()})

Note the change here passes the api's cookies to the requests.get call.


If these changes work, I can update the documentation to include the fix.

papayyg commented 1 year ago

@Russell-Newton I tried the code you sent. It gave the error

'method' object is not iterable

I modified it a little bit: cookies = {cookie['name']: cookie['value'] for cookie in await api.context.cookies()}

And then it worked. That is, the cookies that were used in the API I passed to the session. And the cookies themselves look like this: {'ttwid': '1%7CAVI7ulvcK00n-W6OElzY2emHneSrLvx4Slnjrnv_WJ0%7C1680351019%7Cb0453342efb89d920194f0eb407c408be94cecf3590b3161b44690ecdc527462', 'tt_csrf_token': 'CoTiG9bD-iWUO27U5OCm8oHNUzApu7tztK28', 'tt_chain_token': 'iM+Thjhn20W44pqVH04FLQ==', '__tea_cache_tokens_1988': '{%22_type_%22:%22default%22%2C%22user_unique_id%22:%227217052627791332865%22%2C%22timestamp%22:1680351030840}', 'tiktok_webapp_theme': 'light', 'csrf_session_id': 'cb3a4f15cb7bc696c753918d02bca160', 'msToken': 'ks6BGCynadZVeb0ztd_itBwQ2wOPJnCQp2-ldD539h1ddUxwhXpDkvIqWiqx2tbU0PQcWvyVdCQIue9pPp8FqE-FmN3dwspQ9tBG4-oDam7QFZ0jcTxOip8d-g=='}

But the answer was still: Access Denied. Then I checked how it worked on the TT site itself. And there I realized that the direct link to the video only opens if it is opened by the same cookies as the normal video link. And there I saw one cookie that does not appear in api.context.cookies(), namely s_v_web_id. And if you look at the cookies that are used when viewing the video via a direct link, you can only see those cookies: image

And if we look at the cookies I wrote, we see that everything is there except s_v_web_id.

I don't know why it's not issued when using api.context.cookies(). But I think sending this cookie in a session would solve the problem, or send the same cookie in both api and session initially.

Russell-Newton commented 1 year ago

Every time a video or user page is loaded, TikTokPy clears its cookies. It's possible that s_v_web_id doesn't consistently get populated every visit. I probably need to adjust the cookie management. I don't have a whole lot of time in the next couple of days, given I'm a full time masters student, but I'll try to address this soon.

Russell-Newton commented 1 year ago

After looking into it a bit, s_v_web_id seems to be a cookie that TikTok isn't really using anymore. It doesn't seem necessary for me when viewing or downloading videos on my desktop.

For whatever reason, I seem to be completely unable to recreate this issue consistently. On my end, looking at the cookies in my browser (I used a VPN connecting to an IP in Europe, Asia, and a different location in the Americas), and it looks like the only cookies that are required are msToken, ttwid, tt_chain_token and tt_csrf_token. I used incognito mode to get the effect of having no preloaded cookies, so only the necessary ones would be included.

I suspect these tokens are related to the download link, and any change in either makes the video not downloadable.

Could you try copying over only those 4 cookies and seeing if that makes any difference? Something like:

await self.save_video(video, {cookie["name"]: cookie["value"] for cookie in await api.context.cookies() if cookie["name"] in ("msToken", "ttwid", "tt_chain_token", "tt_csrf_token")})

If that doesn't work, you may need to use a VPN.

papayyg commented 1 year ago

It turned out to be much simpler and more obvious.

You follow the link to the video and do the following

  1. Take away the "src" attribute from the
  2. Taking the "tt_chain_token" value from the cookie
  3. Create header
    'referer': 'https://www.tiktok.com/'

    Send a GET request to the link from src with the cookie and header

It's roughly like this:

import requests

cookies = {
    'tt_chain_token': '8J8/JTDl922BTqv9SaYTpw==', #Get it out of cookies when you open the regular video page
}

headers = {
    'referer': 'https://www.tiktok.com/', #I don't know why, but it doesn't work without it.
}

response = requests.get(
    'https://v16-webapp-prime.tiktok.com/video/tos/useast2a/tos-useast2a-pve-0068/oUIa0yuCmAiBcoIhPzEAwfhgqVzDiGQcWNFvWk/?a=1988&ch=0&cr=0&dr=0&lr=tiktok_m&cd=0%7C0%7C1%7C0&cv=1&br=2626&bt=1313&cs=0&ds=3&ft=_RwJrB0rq8ZmokDNTc_vjdsT_AhLrus&mime_type=video_mp4&qs=0&rc=OGc3aGRoOTk4ZmVkPDxmM0BpajVoO2k6ZnE4ajMzNzczM0AxL2FeMjIvNS0xYDMuMF4uYSMvcWJfcjRnXm5gLS1kMTZzcw%3D%3D&btag=80000&expire=1681225744&l=20230411090855B646C0771A5183040DD5&ply_type=2&policy=2&signature=34ae9a57c2f8c8b8ca7ca0fff86e8fab&tk=tt_chain_token', #src link
    cookies=cookies,
    headers=headers,
)

with open('test.mp4', "wb") as f:
    f.write(response.content)
Russell-Newton commented 1 year ago

I'll add this to the documentation ASAP! Thank you for the help @papayyg. I'll reference this thread and your account when I add it.

Russell-Newton commented 1 year ago

This has been added to the documentation: https://tiktokpy.readthedocs.io/en/stable/users/usage.html#download-videos-and-slideshows

I'm closing the issue because it's been finished.

jinfu-leng commented 1 year ago

Hi, @Russell-Newton and @papayyg, I followed your example, but I still got the access denied error. Could you please help?

Here is the code:

async def fetch_video_bytes(video: Video, api: AsyncTikTokAPI):
    async with aiohttp.ClientSession(cookies={cookie["name"]: cookie["value"] for cookie in await api.context.cookies() if cookie["name"] == "tt_chain_token"}) as session:
        print("video.video.download_addr: " + video.video.download_addr)
        async with session.get(video.video.download_addr, headers={"referer": "https://www.tiktok.com/"}) as resp:
            return await resp.read()

async def download_tiktok_videos(user_handler, download_directory):
    video_download_directory = os.path.join(download_directory, user_handler, "video")
    os.makedirs(video_download_directory, exist_ok=True)

    async with AsyncTikTokAPI(navigation_timeout=60*1000) as api:
        user = await api.user(user_handler, video_limit=2000)

        async for video in user.videos:
            print(f"{video.id}, {video.desc}")
            if not video.image_post:
                video_bytes = await fetch_video_bytes(video, api)
                print("video_bytes:" + video_bytes.decode())
                async with aiofiles.open("output.mp4", 'wb') as file:
                    await file.write(video_bytes)
            return

Here is the error message:

video_bytes:<HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD><BODY>
<H1>Access Denied</H1>

You don't have permission to access "http&#58;&#47;&#47;v16&#45;webapp&#45;prime&#46;us&#46;tiktok&#46;com&#47;video&#47;tos&#47;useast5&#47;tos&#45;useast5&#45;ve&#45;0068c003&#45;tx&#47;oMCEchwzgYMBpmAft2kRDQAGIKBzngUYQfUajS&#47;&#63;" on this server.<P>
Reference&#32;&#35;18&#46;1e32c517&#46;1686182375&#46;8ffae9f
</BODY>
</HTML>
Russell-Newton commented 1 year ago

@jinfu-leng Have you tried using a proxy and/or VPN?

papayyg commented 1 year ago

They seem to have changed something. Now I can’t access the direct link in any way, it doesn’t work with vpn either

Russell-Newton commented 1 year ago

I'll see what I can find out. Once again, it's working fine on my end in the US, but I may be able to figure something out.

Russell-Newton commented 1 year ago

It looks like TikTok is now validating the X-Bogus parameter now, and from what I can tell this is generated right before fetching based on the query parameters. I'll keep looking into this.

ExpressGit commented 1 year ago

I have same problem

<HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD><BODY>
<H1>Access Denied</H1>

You don't have permission to access "http&#58;&#47;&#47;v16&#45;webapp&#45;prime&#46;us&#46;tiktok&#46;com&#47;video&#47;tos&#47;useast5&#47;tos&#45;useast5&#45;ve&#45;0068c003&#45;tx&#47;oMCEchwzgYMBpmAft2kRDQAGIKBzngUYQfUajS&#47;&#63;" on this server.<P>
Reference&#32;&#35;18&#46;1e32c517&#46;1686182375&#46;8ffae9f
</BODY>
</HTML>

code:

async def save_video(video: Video,api: AsyncTikTokAPI,cookies):
     async with aiohttp.ClientSession(cookies=cookies) as session:
        # Creating this header tricks TikTok into thinking it made the request itself
        async with session.get(video.video.download_addr, headers={"referer": "https://www.tiktok.com/"}) as resp:
            return await resp.read()

async def get_tiktok_trend_video():
    async with AsyncTikTokAPI() as api:
        user = await api.user("odapolf", video_limit=10)
        async for video in user.videos:
            print(video)
            video_bytes  = await save_video(video,api,{cookie['name']: cookie['value'] for cookie in api.context.cookies()})
            print("video_bytes:" + video_bytes.decode())
            file_path = os.path.join(directory,'{}.mp4'.format(video.desc))
            print(file_path)
            async with aiofiles.open(file_path, 'wb') as file:
                    await file.write(video_bytes)
        return "OK"
Russell-Newton commented 1 year ago

This may be fixed in version 0.2.4 with the video download function. It doesn't work on slideshows and requires the optional yt-dlp dependency. This can be installed with pip install yt-dlp or pip install tiktokapipy[download]. It may not work 100% depending on where you are, but it works well for me.