PetterKraabol / Twitch-Chat-Downloader

Download chat messages from past broadcasts on Twitch
https://pypi.org/project/tcd
MIT License
643 stars 61 forks source link

Program has completely broke. #23

Closed TheSneakySniper closed 6 years ago

TheSneakySniper commented 7 years ago

Due to changes in twitch's API, the program has stopped working completely.

error log: https://hastebin.com/enurupoqon.sql os: osx sierra

TheSneakySniper commented 7 years ago

Now I have done some reconnaissance on the issue and have discovered that twitch now requires client ID to retrieve chat messages. They also have changed the domain from rechat.twitch.tv to api.twitch.tv.

If you curl the following URL with a twitch client ID, it gives you 50 of the comments in the standard JSON format. This will require a completely new edition of the application.

https://api.twitch.tv/v5/videos/179720371/comments?cursor=eyJpZCI6IjYzMWZmMTUzLTZlZGEtNDlhZi05N2JkLTdmOTYwNDA5NTIxOCIsImhrIjoiYnJvYWRjYXN0OjI2NDA0NDEwMzY4Iiwic2siOiJBQUFXLXV2djJ3QVU2bWt4T2Z4RUFBIn0f

The video this was extracted from was: https://www.twitch.tv/videos/179720371

Added note: the ending appears to be encoded in base64 decoded it reads: {"id":"631ff153-6eda-49af-97bd-7f9604095218","hk":"broadcast:26404410368","sk":"AAAW-uvv2wAU6mkxOfxEAA"}

Another added note: The ID can be found in the json that is outputted when you curl https://api.twitch.tv/v5/videos/179720371/comments by itself. This only prints out the first 50 lines of chat, though.

The documented API appears to have an abundance of information on how to use cursor. Cursor appears to abide by the rules stated there. https://dev.twitch.tv/docs/v5

Another added note: There appears to be an argument called "content_offset_seconds=11135"

it appends to the end of the URL as so: /comments?content_offset_seconds=11135 update: this is inconsistent, as it applies to how fast the chat is moving (it's weird)

Well, I am sorry I couldn't help you more, but I do hope you see this and fix this program!

jdpurcell commented 7 years ago

Yeah Twitch changed the rechat API apparently. The new API is actually simpler and better designed. Just like the rest of their API, it requires the "Accept" and "Client-ID" headers in the request. The first request is simply to:

https://api.twitch.tv/v5/videos/{videoId}/comments?content_offset_seconds=0

To get the first bunch of messages. The "_next" attribute of the response contains the cursor value that you pass to get the next bunch of messages:

https://api.twitch.tv/v5/videos/{videoId}/comments?cursor={nextCursor}

And so on, until finally the response is missing its "_next" attribute, indicating that the end has been reached.

Note that the structure of the message Json is also completely different, so any post-processing code needs to be redone as well.

I author a similar program and was able to get it working again, so you can take a peek at my repo if you need any hints.

TheSneakySniper commented 7 years ago

Thanks so much!

tezJR commented 7 years ago

Thank you both for the quick detective work! I've used Petter's program too and wondered if it was me. I have no idea how busy you are, but if you have the opportunity, I'd love to see this made compatible with the new API.

TheDrHax commented 7 years ago

I have started a total rewrite of this script. It does already support new Twitch API (thanks to @jdpurcell) and saving to .ass format.

Some of functions are not working yet:

Message reader implements Iterable interface, so you can just read messages one by one and write them to many files at the same time (issue #14).

Link: https://github.com/TheDrHax/Twitch-Chat-Downloader/blob/master/app.py

s1leviathan commented 7 years ago

Im sorry but since i have like no clue of what i just read :))) , my question might be 'stupid'... So that means that the app will be functual again in the future? Or is it already ready?

Gorrrg commented 7 years ago

The one here isn't functional, but people have already figured out the new API and are working on fixing the app in their own branch. You can either use their branch (which lacks some old features) or wait for Petter to adopt the changes. Oh, or wait for them to add the missing features.

s1leviathan commented 7 years ago

Thats great , so i asume that the new 'version' is on the link from TheDrHax 's reply?

Gorrrg commented 7 years ago

yes, it's the one closest to what TCD was