search

Zibbp / ganymede

Twitch VOD and Live Stream archiving platform. Includes a rendered and real-time chat for each archive.
https://github.com/Zibbp/ganymede
GNU General Public License v3.0
488 stars 24 forks source link

Failing to fetch VOD ID after livestream ends causes queue to fail #358

Closed russelg closed 9 months ago

russelg commented 10 months ago

While testing the fix for #354, my internet went out :/. That caused the live video downloads to end, which is expected. But even though the video download task in the queue is marked complete, it's not moving on to video convert/move. And chat download is stuck in progress still.

Looks like the video download workflow failed due to the fetch VOD ID failing. Perhaps we can just ignore that failure? There's already handling for when the VOD ID isn't found, we can probably use the same logic in the case the fetch fails.

Most relevant line:

{"level":"error","ActivityType":"DownloadTwitchLiveVideo","Attempt":1,"Error":"failed to get twitch videos: Get \"https://api.twitch.tv/helix/videos?user_id=90937535&type=archive&first=100\": dial tcp: lookup api.twitch.tv on 127.0.0.11:53: read udp 127.0.0.1:41249->127.0.0.11:53: i/o timeout","Namespace":"default","RunID":"d9e1e6ad-08aa-4efe-b883-e76a30fa4d86","TaskQueue":"archive","WorkerID":"b183487d2851","WorkflowID":"6a89acb8-12eb-4b80-a595-8155f9fcde77_5","time":"2024-01-15T09:45:09+08:00","message":"Activity error."}

Full log:

{"level":"debug","time":"2024-01-15T09:37:55+08:00","message":"workflow id b45d1f90-b346-11ee-a650-0242ac12000f started for live stream 43418657787"}
{"level":"debug","Attempt":1,"ChildWorkflowID":"148902df-f5e8-4456-bcdd-3e78cf9936dc_5","Namespace":"default","RunID":"148902df-f5e8-4456-bcdd-3e78cf9936dc","TaskQueue":"archive","WorkerID":"b183487d2851","WorkflowID":"b45d1f90-b346-11ee-a650-0242ac12000f","WorkflowType":"CreateDirectoryWorkflow","time":"2024-01-15T09:37:55+08:00","message":"ExecuteChildWorkflow"}
{"level":"debug","ActivityID":"5","ActivityType":"CreateDirectory","Attempt":1,"Namespace":"default","RunID":"99d578de-03b1-41e4-8830-5a6d05017d66","TaskQueue":"archive","WorkerID":"b183487d2851","WorkflowID":"148902df-f5e8-4456-bcdd-3e78cf9936dc_5","WorkflowType":"CreateDirectoryWorkflow","time":"2024-01-15T09:37:55+08:00","message":"ExecuteActivity"}
{"level":"debug","time":"2024-01-15T09:37:55+08:00","message":"creating folder: lyricaldota/2024-01-15 - (LIVE)_Nouns_vs_BammySoy [43418657787][live][b45d1f90-b346-11ee-a650-0242ac12000f]"}
{"level":"debug","Attempt":1,"ChildWorkflowID":"148902df-f5e8-4456-bcdd-3e78cf9936dc_14","Namespace":"default","RunID":"148902df-f5e8-4456-bcdd-3e78cf9936dc","TaskQueue":"archive","WorkerID":"b183487d2851","WorkflowID":"b45d1f90-b346-11ee-a650-0242ac12000f","WorkflowType":"DownloadTwitchLiveThumbnailsWorkflow","time":"2024-01-15T09:37:55+08:00","message":"ExecuteChildWorkflow"}
{"level":"debug","ActivityID":"5","ActivityType":"DownloadTwitchLiveThumbnails","Attempt":1,"Namespace":"default","RunID":"3386fbbf-4186-42ec-8fd8-135da3bacf5e","TaskQueue":"archive","WorkerID":"b183487d2851","WorkflowID":"148902df-f5e8-4456-bcdd-3e78cf9936dc_14","WorkflowType":"DownloadTwitchLiveThumbnailsWorkflow","time":"2024-01-15T09:37:55+08:00","message":"ExecuteActivity"}
{"level":"debug","time":"2024-01-15T09:37:57+08:00","message":"checking if tasks are done for video 43418657787"}
{"level":"debug","Attempt":1,"ChildWorkflowID":"148902df-f5e8-4456-bcdd-3e78cf9936dc_23","Namespace":"default","RunID":"148902df-f5e8-4456-bcdd-3e78cf9936dc","TaskQueue":"archive","WorkerID":"b183487d2851","WorkflowID":"b45d1f90-b346-11ee-a650-0242ac12000f","WorkflowType":"SaveTwitchLiveVideoInfoWorkflow","time":"2024-01-15T09:37:57+08:00","message":"ExecuteChildWorkflow"}
{"level":"debug","ActivityID":"5","ActivityType":"SaveTwitchLiveVideoInfo","Attempt":1,"Namespace":"default","RunID":"b9b520dd-703d-40ab-9697-3751e87de7f2","TaskQueue":"archive","WorkerID":"b183487d2851","WorkflowID":"148902df-f5e8-4456-bcdd-3e78cf9936dc_23","WorkflowType":"SaveTwitchLiveVideoInfoWorkflow","time":"2024-01-15T09:37:57+08:00","message":"ExecuteActivity"}
{"level":"debug","time":"2024-01-15T09:37:57+08:00","message":"checking if tasks are done for video 43418657787"}
{"level":"debug","Attempt":1,"ChildWorkflowID":"148902df-f5e8-4456-bcdd-3e78cf9936dc_32","Namespace":"default","RunID":"148902df-f5e8-4456-bcdd-3e78cf9936dc","TaskQueue":"archive","WorkerID":"b183487d2851","WorkflowID":"b45d1f90-b346-11ee-a650-0242ac12000f","WorkflowType":"ArchiveTwitchLiveChatWorkflow","time":"2024-01-15T09:37:57+08:00","message":"ExecuteChildWorkflow"}
{"level":"debug","time":"2024-01-15T09:37:57+08:00","message":"Live chat archive workflow ID: 148902df-f5e8-4456-bcdd-3e78cf9936dc_32"}
{"level":"debug","Attempt":1,"ChildWorkflowID":"148902df-f5e8-4456-bcdd-3e78cf9936dc_37","Namespace":"default","RunID":"148902df-f5e8-4456-bcdd-3e78cf9936dc","TaskQueue":"archive","WorkerID":"b183487d2851","WorkflowID":"b45d1f90-b346-11ee-a650-0242ac12000f","WorkflowType":"DownloadTwitchLiveChatWorkflow","time":"2024-01-15T09:37:57+08:00","message":"ExecuteChildWorkflow"}
{"level":"debug","time":"2024-01-15T09:37:57+08:00","message":"Live chat workflow ID: 148902df-f5e8-4456-bcdd-3e78cf9936dc_37"}
{"level":"debug","Attempt":1,"ChildWorkflowID":"148902df-f5e8-4456-bcdd-3e78cf9936dc_42","Namespace":"default","RunID":"148902df-f5e8-4456-bcdd-3e78cf9936dc","TaskQueue":"archive","WorkerID":"b183487d2851","WorkflowID":"b45d1f90-b346-11ee-a650-0242ac12000f","WorkflowType":"ArchiveTwitchLiveVideoWorkflow","time":"2024-01-15T09:37:57+08:00","message":"ExecuteChildWorkflow"}
{"level":"debug","Attempt":1,"ChildWorkflowID":"6a89acb8-12eb-4b80-a595-8155f9fcde77_5","Namespace":"default","RunID":"6a89acb8-12eb-4b80-a595-8155f9fcde77","TaskQueue":"archive","WorkerID":"b183487d2851","WorkflowID":"148902df-f5e8-4456-bcdd-3e78cf9936dc_42","WorkflowType":"DownloadTwitchLiveVideoWorkflow","time":"2024-01-15T09:37:57+08:00","message":"ExecuteChildWorkflow"}
{"level":"debug","time":"2024-01-15T09:37:57+08:00","message":"starting heartbeat download-livevideo-43418657787"}
{"level":"debug","time":"2024-01-15T09:37:58+08:00","message":"streamlink live args: [--progress=force --force https://twitch.tv/lyricaldota best,best --http-header Authorization=OAuth ... --twitch-low-latency --twitch-disable-hosting -o /tmp/43418657787_b45d1f90-b346-11ee-a650-0242ac12000f-video.mp4]"}
{"level":"debug","time":"2024-01-15T09:37:58+08:00","message":"running: streamlink --progress=force --force https://twitch.tv/lyricaldota best,best --http-header Authorization=OAuth ... --twitch-low-latency --twitch-disable-hosting -o /tmp/43418657787_b45d1f90-b346-11ee-a650-0242ac12000f-video.mp4"}
{"level":"debug","ActivityID":"9","ActivityType":"DownloadTwitchLiveChat","Attempt":1,"Namespace":"default","RunID":"560112a8-0ea4-4357-9e0f-82c74d712328","TaskQueue":"archive","WorkerID":"b183487d2851","WorkflowID":"148902df-f5e8-4456-bcdd-3e78cf9936dc_37","WorkflowType":"DownloadTwitchLiveChatWorkflow","time":"2024-01-15T09:37:58+08:00","message":"ExecuteActivity"}
{"level":"debug","time":"2024-01-15T09:37:58+08:00","message":"starting heartbeat download-livechat-43418657787"}
{"level":"debug","time":"2024-01-15T09:44:59+08:00","message":"finished downloading live video for 43418657787"}
{"level":"error","ActivityType":"DownloadTwitchLiveVideo","Attempt":1,"Error":"failed to get twitch videos: Get \"https://api.twitch.tv/helix/videos?user_id=90937535&type=archive&first=100\": dial tcp: lookup api.twitch.tv on 127.0.0.11:53: read udp 127.0.0.1:41249->127.0.0.11:53: i/o timeout","Namespace":"default","RunID":"d9e1e6ad-08aa-4efe-b883-e76a30fa4d86","TaskQueue":"archive","WorkerID":"b183487d2851","WorkflowID":"6a89acb8-12eb-4b80-a595-8155f9fcde77_5","time":"2024-01-15T09:45:09+08:00","message":"Activity error."}
{"level":"debug","time":"2024-01-15T09:45:09+08:00","message":"stopping heartbeat download-livevideo-43418657787"}
{"level":"warn","ActivityID":"9","ActivityType":"DownloadTwitchLiveChat","Attempt":1,"Error":"workflow execution already completed","Namespace":"default","RunID":"560112a8-0ea4-4357-9e0f-82c74d712328","TaskQueue":"archive","WorkerID":"b183487d2851","WorkflowID":"148902df-f5e8-4456-bcdd-3e78cf9936dc_37","WorkflowType":"DownloadTwitchLiveChatWorkflow","time":"2024-01-15T09:45:19+08:00","message":"RecordActivityHeartbeat with error"}
{"level":"warn","ActivityID":"9","ActivityType":"DownloadTwitchLiveChat","Attempt":1,"Error":"context canceled","Namespace":"default","RunID":"560112a8-0ea4-4357-9e0f-82c74d712328","TaskQueue":"archive","WorkerID":"b183487d2851","WorkflowID":"148902df-f5e8-4456-bcdd-3e78cf9936dc_37","WorkflowType":"DownloadTwitchLiveChatWorkflow","time":"2024-01-15T09:46:19+08:00","message":"RecordActivityHeartbeat with error"}
{"level":"warn","ActivityID":"9","ActivityType":"DownloadTwitchLiveChat","Attempt":1,"Error":"context canceled","Namespace":"default","RunID":"560112a8-0ea4-4357-9e0f-82c74d712328","TaskQueue":"archive","WorkerID":"b183487d2851","WorkflowID":"148902df-f5e8-4456-bcdd-3e78cf9936dc_37","WorkflowType":"DownloadTwitchLiveChatWorkflow","time":"2024-01-15T09:47:19+08:00","message":"RecordActivityHeartbeat with error"}
<above repeated every minute>

Originally posted by @russelg in https://github.com/Zibbp/ganymede/issues/354#issuecomment-1891190355

Zibbp commented 9 months ago

Ah the recently added feature to update the stream ID with the VOD ID is causing this. Updated the functionality to log an error instead of returning an error as this isn't critical to the archive flow https://github.com/Zibbp/ganymede/pull/357/commits/afea0f59c6a110d629ad323830856a105b036ead in https://github.com/Zibbp/ganymede/pull/357.