search

Zibbp / ganymede

Twitch VOD and Live Stream archiving platform. Includes a rendered and real-time chat for each archive.
https://github.com/Zibbp/ganymede
GNU General Public License v3.0
396 stars 22 forks source link

chat download fails for one channel #437

Open NothingTV opened 1 month ago

NothingTV commented 1 month ago

Hi there,

I monitor few channels and one of them constantly fails to finish the chat download task, I can't see any relevant errors, besides of that, that seems to tell nothing about the cause itself:

ganymede-api  | {"level":"debug","time":"2024-05-28T16:41:32+02:00","message":"finished downloading live video for 42390743976"}
ganymede-api  | {"level":"debug","time":"2024-05-28T16:41:32+02:00","message":"getting video duration"}
ganymede-api  | {"level":"debug","time":"2024-05-28T16:41:32+02:00","message":"video duration: 9047"}
ganymede-api  | {"level":"debug","time":"2024-05-28T16:41:32+02:00","message":"getting twitch videos for user: 138907996 with type archive and cursor eyJiIjpudWxsLCJhIjp7Ik9mZnNldCI6NTJ9fQ"}
ganymede-api  | {"level":"info","time":"2024-05-28T16:41:32+02:00","message":"found vod id 2157350903 for livestream 42390743976, updating database"}
ganymede-api  | {"level":"debug","time":"2024-05-28T16:41:32+02:00","message":"stopping heartbeat download-livevideo-42390743976"}
ganymede-api  | {"level":"debug","service":"worker","ActivityID":"11","ActivityType":"KillTwitchLiveChatDownload","Attempt":1,"Namespace":"default","RunID":"d5abeedf-3a10-4601-ac0c-ad767b61d17c","TaskQueue":"archive","WorkerID":"7928c1a39b9b","WorkflowID":"8585e116-182d-4ecd-8e9b-6ba451ac7b46_5","WorkflowType":"DownloadTwitchLiveVideoWorkflow","time":"2024-05-28T16:41:32+02:00","message":"ExecuteActivity"}
ganymede-api  | {"level":"info","time":"2024-05-28T16:41:32+02:00","message":"killing chat downloader for channel silverline"}
ganymede-api  | {"level":"debug","time":"2024-05-28T16:41:32+02:00","message":"found pid 224\n245\n for chat_downloader"}
ganymede-api  | {"level":"error","service":"worker","ActivityType":"KillTwitchLiveChatDownload","Attempt":1,"Error":"exit status 1","Namespace":"default","RunID":"d5abeedf-3a10-4601-ac0c-ad767b61d17c","TaskQueue":"archive","WorkerID":"7928c1a39b9b","WorkflowID":"8585e116-182d-4ecd-8e9b-6ba451ac7b46_5","time":"2024-05-28T16:41:32+02:00","message":"Activity error."}
ganymede-api  | {"level":"debug","time":"2024-05-28T16:41:32+02:00","message":"Error notification is disabled"}

In the workflows, you can also see that it fails (none of the tasks have a valid time tho):

Terminated
    [87256fac-bf53-4acf-a0e7-0705e6c2a56b_34](https://XXXX/workflows/87256fac-bf53-4acf-a0e7-0705e6c2a56b_34/f70d19d2-b6f2-4911-99dc-7f4e58458f4c)  [f70d19d2-b6f2-4911-99dc-7f4e58458f4c](https://XXXX/workflows/87256fac-bf53-4acf-a0e7-0705e6c2a56b_34/f70d19d2-b6f2-4911-99dc-7f4e58458f4c) [ArchiveTwitchLiveChatWorkflow](https://XXXX/workflows/87256fac-bf53-4acf-a0e7-0705e6c2a56b_34/f70d19d2-b6f2-4911-99dc-7f4e58458f4c)    Invalid Date
Terminated
    [87256fac-bf53-4acf-a0e7-0705e6c2a56b_39](https://XXXX/workflows/87256fac-bf53-4acf-a0e7-0705e6c2a56b_39/50ac117d-f2d6-4317-8a10-f185806338ac)  [50ac117d-f2d6-4317-8a10-f185806338ac](https://XXXX/workflows/87256fac-bf53-4acf-a0e7-0705e6c2a56b_39/50ac117d-f2d6-4317-8a10-f185806338ac) [DownloadTwitchLiveChatWorkflow](https://XXXX/workflows/87256fac-bf53-4acf-a0e7-0705e6c2a56b_39/50ac117d-f2d6-4317-8a10-f185806338ac)   Invalid Date
Failed
    [4b4e4993-1ceb-11ef-9764-0242ac170007](https://XXXX/workflows/4b4e4993-1ceb-11ef-9764-0242ac170007/87256fac-bf53-4acf-a0e7-0705e6c2a56b)    [87256fac-bf53-4acf-a0e7-0705e6c2a56b](https://XXXX/workflows/4b4e4993-1ceb-11ef-9764-0242ac170007/87256fac-bf53-4acf-a0e7-0705e6c2a56b)    [ArchiveLiveVideoWorkflow](https://XXXX/workflows/4b4e4993-1ceb-11ef-9764-0242ac170007/87256fac-bf53-4acf-a0e7-0705e6c2a56b)    Invalid Date
Failed
    [87256fac-bf53-4acf-a0e7-0705e6c2a56b_44](https://XXX/workflows/87256fac-bf53-4acf-a0e7-0705e6c2a56b_44/8585e116-182d-4ecd-8e9b-6ba451ac7b46)   [8585e116-182d-4ecd-8e9b-6ba451ac7b46](https://XXXX/workflows/87256fac-bf53-4acf-a0e7-0705e6c2a56b_44/8585e116-182d-4ecd-8e9b-6ba451ac7b46) [ArchiveTwitchLiveVideoWorkflow](https://XXXX/workflows/87256fac-bf53-4acf-a0e7-0705e6c2a56b_44/8585e116-182d-4ecd-8e9b-6ba451ac7b46)   Invalid Date
Failed
    [8585e116-182d-4ecd-8e9b-6ba451ac7b46_5](https://XXXX/workflows/8585e116-182d-4ecd-8e9b-6ba451ac7b46_5/d5abeedf-3a10-4601-ac0c-ad767b61d17c)    [d5abeedf-3a10-4601-ac0c-ad767b61d17c](https://XXXX/workflows/8585e116-182d-4ecd-8e9b-6ba451ac7b46_5/d5abeedf-3a10-4601-ac0c-ad767b61d17c)  [DownloadTwitchLiveVideoWorkflow](https://XXXX/workflows/8585e116-182d-4ecd-8e9b-6ba451ac7b46_5/d5abeedf-3a10-4601-ac0c-ad767b61d17c)   Invalid Date

It's only for this one channel.

Zibbp commented 1 month ago

Looks like it's failing because there are two processes running downloading that chat. A couple of questions:

  1. Has this recently started happening?
  2. If you restart the containers (when nothing is archiving) does the next archive of the channel succeed? If not, can you post the logs for that failure?

I'll update the code to handle multiple processes if it gets stuck in a bad state like this again.

NothingTV commented 1 month ago

  1. I can't answer you that, since I recently revived the whole instance.

  2. Just restarted, no new errors, also same result (still pending)

  3. Okay, so it looks like it worked after the api restart, two vods could get recorded without issues, now the streamer had a network issue, because of that she had to restart the stream entirely and it's stuck again on the chat_downloader Task, but this time, it also doesn't start another task for the stream after her disconnect. Errors like this flood my api container:

    ganymede-api  | {"level":"warn","service":"worker","ActivityID":"9","ActivityType":"DownloadTwitchLiveChat","Attempt":1,"Error":"context canceled","Namespace":"default","RunID":"3ed0720a-0654-487c-a1fc-30447ab7d32e","TaskQueue":"archive","WorkerID":"7928c1a39b9b","WorkflowID":"c354a72c-8b5b-475a-be67-822256320a31_39","WorkflowType":"DownloadTwitchLiveChatWorkflow","time":"2024-05-31T18:04:49+02:00","message":"RecordActivityHeartbeat with error"}

    I think these are the only logs for this stream:

    ganymede-api  | {"level":"debug","time":"2024-05-31T17:06:22+02:00","message":"finished downloading live video for 42403585224"}
ganymede-api  | {"level":"debug","time":"2024-05-31T17:06:22+02:00","message":"getting video duration"}
ganymede-api  | {"level":"debug","time":"2024-05-31T17:06:23+02:00","message":"video duration: 2154"}
ganymede-api  | {"level":"debug","time":"2024-05-31T17:06:23+02:00","message":"getting twitch videos for user: 138907996 with type archive and cursor eyJiIjpudWxsLCJhIjp7Ik9mZnNldCI6NTJ9fQ"}
ganymede-api  | {"level":"info","time":"2024-05-31T17:06:23+02:00","message":"found vod id 2160007110 for livestream 42403585224, updating database"}
ganymede-api  | {"level":"debug","time":"2024-05-31T17:06:23+02:00","message":"stopping heartbeat download-livevideo-42403585224"}
ganymede-api  | {"level":"debug","service":"worker","ActivityID":"11","ActivityType":"KillTwitchLiveChatDownload","Attempt":1,"Namespace":"default","RunID":"e375084a-4f1e-4702-83b3-b6ebc867f531","TaskQueue":"archive","WorkerID":"7928c1a39b9b","WorkflowID":"f8333481-f9c5-4bcb-874b-0ccf30e78407_5","WorkflowType":"DownloadTwitchLiveVideoWorkflow","time":"2024-05-31T17:06:23+02:00","message":"ExecuteActivity"}
ganymede-api  | {"level":"info","time":"2024-05-31T17:06:23+02:00","message":"killing chat downloader for channel silverline"}
ganymede-api  | {"level":"debug","time":"2024-05-31T17:06:23+02:00","message":"found pid 172\n264\n for chat_downloader"}
ganymede-api  | {"level":"error","service":"worker","ActivityType":"KillTwitchLiveChatDownload","Attempt":1,"Error":"exit status 1","Namespace":"default","RunID":"e375084a-4f1e-4702-83b3-b6ebc867f531","TaskQueue":"archive","WorkerID":"7928c1a39b9b","WorkflowID":"f8333481-f9c5-4bcb-874b-0ccf30e78407_5","time":"2024-05-31T17:06:23+02:00","message":"Activity error."}
ganymede-api  | {"level":"debug","time":"2024-05-31T17:06:23+02:00","message":"Error notification is disabled"}
ganymede-api  | {"level":"warn","service":"worker","ActivityID":"9","ActivityType":"DownloadTwitchLiveChat","Attempt":1,"Error":"workflow execution already completed","Namespace":"default","RunID":"561516b2-f742-4e89-bd2c-87567b3f481f","TaskQueue":"archive","WorkerID":"7928c1a39b9b","WorkflowID":"4b1cba14-5e02-4cee-bcb7-f70c46290cea_39","WorkflowType":"DownloadTwitchLiveChatWorkflow","time":"2024-05-31T17:06:41+02:00","message":"RecordActivityHeartbeat with error"}

I also still have Invalid Date in every temporal task.

Zibbp commented 1 month ago

I made some changes in https://github.com/Zibbp/ganymede/pull/439 that should resolve the double process ID error that you are having. Can you use the :main tag for both the API and frontend container and try again? Before you switch the tags, please read https://github.com/Zibbp/ganymede/issues/441 first as you might encounter this issue when you go to pull the images.

NothingTV commented 1 month ago

Just switched to the :main-Tag, I will keep you posted.

I can't manually start tasks now tho, I get Request Failed Not Found after clicking the symbols.

Zibbp commented 1 month ago

Whoops, new :main image tag should resolve the requests failing.