chat download error after twitch raid

[fllppi]

hi, after a twitch raid the chat download task crashes with theses errors:

container log:

{"level":"error","error":"exit status 1","time":"2023-01-15T02:08:23+01:00","message":"error waiting for chat_downloader for live chat download"}
{"level":"error","error":"exit status 1","time":"2023-01-15T02:08:23+01:00","message":"error downloading live chat"}

tasks log:

File "/usr/bin/chat_downloader", line 33, in 
    sys.exit(load_entry_point('chat-downloader==0.2.1', 'console_scripts', 'chat_downloader')())
  File "/usr/lib/python3.10/site-packages/chat_downloader-0.2.1-py3.10.egg/chat_downloader/cli.py", line 194, in main
    run(**args.__dict__)
  File "/usr/lib/python3.10/site-packages/chat_downloader-0.2.1-py3.10.egg/chat_downloader/chat_downloader.py", line 360, in run
    for message in chat:
  File "/usr/lib/python3.10/site-packages/chat_downloader-0.2.1-py3.10.egg/chat_downloader/sites/common.py", line 283, in __next__
    item = next(self.chat)
  File "/usr/lib/python3.10/site-packages/chat_downloader-0.2.1-py3.10.egg/chat_downloader/sites/twitch.py", line 1535, in _get_chat_messages_by_stream_id
    twitch_chat_irc = create_connection()
  File "/usr/lib/python3.10/site-packages/chat_downloader-0.2.1-py3.10.egg/chat_downloader/sites/twitch.py", line 1528, in create_connection
    irc = TwitchChatIRC()
  File "/usr/lib/python3.10/site-packages/chat_downloader-0.2.1-py3.10.egg/chat_downloader/sites/twitch.py", line 57, in __init__
    self.socket.connect(('irc.chat.twitch.tv', 6667))
TimeoutError: [Errno 110] Operation timed out

Is the an issue for this project or chat-downloader ?

[Zibbp]

A Twitch raid shouldn't affect the chat download as it's only watching that specific channel. The operation timeout error looks to me like a temporary connection issue either on your end or Twitch's. Has this happened more than once?

The chat json should still be there for the entire stream so you should be able to manually start the next task.

[fllppi]

happend multiple times (every time there is a raid) Connection should be fine since the video is captured just fine

restarting the the chat download didnt work (got "Task has been restarted" but instantly fails without log) since it was a livestream that isnt availabe anymore

update: starting the next tasks also fails, so i took a look at the tmp folder but couldnt find the chat file. It seems like restarting the download task deleted the original chat file. Maybe a check for old files, then incrementing a number in the file name and later reading all files for the rendering would be useful. That way, even connection issues wouldn't cause much trouble (besides a few lost messages)

[Zibbp]

Restarting the chat download task for a live archive will fail and also remove the previous chat json file. If it fails how it did originally without starting the chat download task again, you should be able to manually start the convert step.

I tested with a demo stream while archiving it and raiding another channel and could not reproduce the errors. I see in the log the version is 0.2.1. In the latest release that happened about an hour ago I migrated back to the original chat-downloader repositor which is now version 0.2.3. Can you try a live archive with the latest containers?

[Zibbp]

Also, are you live archiving for a particular reason? The VOD/video archiving is the preferred method. I know that is not always possible due to dual audio tracks, no VODs, etc.

[fllppi]

i just updated the containers because of the other issue, but will report back if the error occurs again.

Maybe automatically restarting download tasks if an error occurs would be great. That way even connection issues wouldn't cause much trouble (besides a few lost messages and small chunk of the stream)

[fllppi]

Also, are you live archiving for a particular reason? The VOD/video archiving is the preferred method. I know that is not always possible due to dual audio tracks, no VODs, etc.

Im using your tool to capture multiple music streamer that dont leave there vods up because of dmca problems

[Zibbp]

Maybe automatically restarting download tasks if an error occurs. That way even connection issues wouldn't cause much trouble (besides a few lost messages and small chunk of the stream)

Connection issues are already handled to a degree. If your connection is temporarily disabled it can hang out for a few seconds. If it's down for any longer the video download will detect that and kill the video and chat download tasks. Once connection is restored a new archive is created for the stream.

I've never seen the chat_downloader issue before and can't find anything in the repo for the program either so I think it may be something else.

[fllppi]

ok i just setup a second instance in another location and am now using the new image, maybe that fixes it. I'll report back if it happens again