viniciusenari / twitch-highlights-bot

Python bot that creates video compilations of the most watched Twitch clips of the week and uploads it to Youtube
GNU General Public License v3.0
31 stars 7 forks source link

thumbnail_url no longer returns a valid url to grab the dowload link #9

Open armeniomp opened 2 months ago

armeniomp commented 2 months ago

The url format on recent clips for the thumbnail_url are as follows:

https://static-cdn.jtvnw.net/twitch-clips-thumbnails-prod/TiredPolitePlumageThunBeast-VkK6ArboLGiP8M3r/092b3865-3711-4031-ba9e-cfea02a858ae/preview-480x272.jpg

Twitch seems to no longer be using the https://clips-media-assets2.twitch.tv/

Using the example link I get from that example clip thumbnail we could use it to download it from: https://production.assets.clips.twitchcdn.net/v2/media/TiredPolitePlumageThunBeast-VkK6ArboLGiP8M3r/092b3865-3711-4031-ba9e-cfea02a858ae/video.mp4?sig=4d0f531f979ebf73818c2412e4dd756f093779eb&token=%7B%22authorization%22%3A%7B%22forbidden%22%3Afalse%2C%22reason%22%3A%22%22%7D%2C%22clip_uri%22%3A%22%22%2C%22clip_slug%22%3A%22TiredPolitePlumageThunBeast-VkK6ArboLGiP8M3r%22%2C%22device_id%22%3A%22cc49016415d8d503%22%2C%22expires%22%3A1725876517%2C%22user_id%22%3A%22%22%2C%22version%22%3A2%7D

The host URL changes but everything up to video.mp4 is still available but it doesn't work without the sig and token attributes and I don't know how to get them correctly without opening the actual clip https://clips.twitch.tv/TiredPolitePlumageThunBeast-VkK6ArboLGiP8M3r and get it from the video tag src

Xav-Deb commented 2 months ago

hi, same here

armeniomp commented 2 months ago

I've been messing around on the debugging and network tabs for the clip I mentioned and found where the sig and token are being fetched and it seems to come from a POST request in the p.js file

Request headers

POST /gql HTTP/3
Host: gql.twitch.tv
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:130.0) Gecko/20100101 Firefox/130.0
Accept: */*
Accept-Language: en-US
Accept-Encoding: gzip, deflate, br, zstd
Referer: https://clips.twitch.tv/
Client-Id: kimne78kx3ncx6brgo4mv6wki5h1ko
X-Device-Id: 252997540e7c16bf
Client-Version: 780f1908-7d4e-45ba-8134-90dcf9d433df
Client-Session-Id: 439b2195b16020b8
Client-Integrity: v4.local.JBgIDD6MQezLfZtsMIu2771hJpoXvT7nVZdy4b8ZPXg9iUe8iTK6Cbs14B3hKHzPfgjDXnV_BERC_pXFGs0pyHriUwW18xUs-Ut4qrrj71y_MvGEbXQwOBldMPyY-iHXkEEhc5W5Fd7JW4lEMXpwqp78RGNGADh602ej4gN2j8qttOT1ekOa_nZaadw1lL_vfbvjPtZpWXyTIkxHtxQwKZjVqNzWOOH1L6aYTqxtmfpYr7RRzBwSJ04ib0yc76LBpHXJCGVTAcrNpWujI_f1mo1eDIh9QGbCW0g1jck36Q_yaIYMosvrJxt3dLcN1YJzQe47TsKdjiweihSMz9P_woX4ditEabVMqmpn3Tg_a51EdYoP_UFjerUD6w1QzTkILF5zR84oc9S_dyy-1rV5MqdaVAWou5S4ZS6Pghh6SPKeZ5Q
Content-Type: text/plain;charset=UTF-8
Content-Length: 254
Origin: https://clips.twitch.tv
DNT: 1
Sec-GPC: 1
Connection: keep-alive
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-site
Priority: u=4
TE: trailers

Request Payload:

[{"operationName":"VideoAccessToken_Clip","variables":{"platform":"web","slug":"TiredPolitePlumageThunBeast-VkK6ArboLGiP8M3r"},"extensions":{"persistedQuery":{"version":1,"sha256Hash":"6fd3af2b22989506269b9ac02dd87eb4a6688392d67d94e41a6886f1e9f5c00f"}}}]

Response:

[{"data":{"clip":{"id":"4272315987","playbackAccessToken":{"signature":"170c944c9e0d5403f813e2a18d9d23e89729411c","value":"{\"authorization\":{\"forbidden\":false,\"reason\":\"\"},\"clip_uri\":\"\",\"clip_slug\":\"TiredPolitePlumageThunBeast-VkK6ArboLGiP8M3r\",\"device_id\":\"252997540e7c16bf\",\"expires\":1726051884,\"user_id\":\"\",\"version\":2}","__typename":"PlaybackAccessToken"},"videoQualities":[{"frameRate":59.99946975708008,"quality":"1080","sourceURL":"https://production.assets.clips.twitchcdn.net/v2/media/TiredPolitePlumageThunBeast-VkK6ArboLGiP8M3r/092b3865-3711-4031-ba9e-cfea02a858ae/video.mp4","__typename":"ClipVideoQuality"},{"frameRate":59.99625778198242,"quality":"720","sourceURL":"https://production.assets.clips.twitchcdn.net/v2/media/TiredPolitePlumageThunBeast-VkK6ArboLGiP8M3r/092b3865-3711-4031-ba9e-cfea02a858ae/video-720.mp4","__typename":"ClipVideoQuality"},{"frameRate":30.024330139160156,"quality":"480","sourceURL":"https://production.assets.clips.twitchcdn.net/v2/media/TiredPolitePlumageThunBeast-VkK6ArboLGiP8M3r/092b3865-3711-4031-ba9e-cfea02a858ae/video-480.mp4","__typename":"ClipVideoQuality"},{"frameRate":30.024330139160156,"quality":"360","sourceURL":"https://production.assets.clips.twitchcdn.net/v2/media/TiredPolitePlumageThunBeast-VkK6ArboLGiP8M3r/092b3865-3711-4031-ba9e-cfea02a858ae/video-360.mp4","__typename":"ClipVideoQuality"}],"__typename":"Clip"}},"extensions":{"durationMilliseconds":76,"operationName":"VideoAccessToken_Clip","requestID":"01J7E6NZ2X6BS1GYSSS1RSJ7E5"}}]

This response not only returns the sig and the token but also the complete urls for different desired resolutions.

I really like the simplicity this project has to just use a simple quick request to download the clips so I really didn't want to recourse to selenium or any more tools than necessary and I'm not that good at tracing these values to recreate this request. I'll keep this issue updated if I find a solution. Any help on how to get them or to find the solution?

armeniomp commented 2 months ago

Update I'm able to curl for the response

curl -H "Client-ID:kimne78kx3ncx6brgo4mv6wki5h1ko" -d "{\"operationName\":\"VideoAccessToken_Clip\",\"variables\":{\"platform\":\"web\",\"slug\":\"TiredPolitePlumageThunBeast-VkK6ArboLGiP8M3r\"},\"extensions\":{\"persistedQuery\":{\"version\":1,\"sha256Hash\":\"6fd3af2b22989506269b9ac02dd87eb4a6688392d67d94e41a6886f1e9f5c00f\"}}}" https://gql.twitch.tv/gql

The hash is a fixed parameter and needs to be passed, this is probably a dead end if it's hashed when the clip is created

eduardosantos1206 commented 2 months ago

Not the best, but, works!

imports added to clips.py

from selenium import webdriver from selenium.webdriver.firefox.options import Options import time

New def download_clip(sef,clip):

`

    option = Options()
    option.headless = False
    driver = webdriver.Firefox()
    driver.get(clip.url)
    time.sleep(5)

    clip_url = driver.find_element("xpath", "//div[@class='Layout-sc-1xcs6mc-0 video-ref']//video").get_property("src")

    driver.quit()
    r = requests.get(clip_url)

    if r.headers['Content-Type'] == 'binary/octet-stream' or r.headers['Content-Type'] == 'video/mp4':
        if not os.path.exists('files/clips'): os.makedirs('files/clips')
        with open(clip.path, 'wb') as f:
            f.write(r.content)
    else:
        print(f'Failed to download clip from thumb: {clip.thumbnail_url}')

`

viniciusenari commented 2 months ago

I conducted some investigation, and it seems that using Selenium is the only option for now. The older clips still maintain the same URL format for thumbnails, so I'll keep that option in the code as well. I will keep this issue open in case someone discovers an alternative.