Do not create a new live archive if external ID is already in queue

[soleny894]

Hi,

I have an issue with the queue system. When the stream have some lags and stops for some reason (ex: network issues coming from the streamer), it creates a duplicate for the next stream that downloads and renders the same thing. I noticed that both have the same live external ID but not the same UUID (the first part is different), so I guess it would be good to check if it's already in queue based on that ID and if that's the case, then don't add another archive.

[Zibbp]

Dealing with network interruptions or interruptions in the stream in general is not the easiest.

The current way it works is if Streamlink (what downloads the video stream) is unable to fetch new playlist segments after a few seconds, it will exit which in turn kills the chat download. Then the queue will post process the video, and render the chat. If the stream were to come back up again in a few minutes, a new queue item is created because the old stream was ended as far as Ganymede in concerned. A new queue item needs to be created because the chat download has no way of knowing the stream is down, it simply listens to the IRC channel. Restarting the video download and concatenating the two would lead to an offset of the video and chat due to the video being X minutes shorter (how ever long the stream was down for).

This is the easiest and I believe best way to handle it. If you have any suggestions or improvements let me know.

[soleny894]

In my case, this is not 2 different type of queue item, they actually download concurrently the same video and both render their part of the video they download when the stream end. It's difficult for me to find how exactly it happens, but it's always doing that when a streamer have a network issue. The stream external ID can change completely from the old one but it will still create a new queue item a few seconds later like the current one downloading doesn't exist. I use a 6 seconds timer to check if the channels are live streaming and It is more or less the difference in time between the two.

Here's an example I got from my queue history:

ID ending in 2332 is the original stream that started first, after sometime the stream ended because of network issues and rendered what was downloaded. A few minutes later, a new stream is started with a new ID ending in 0053, it starts downloading both video and chat. A few seconds later, a new queue item is added using the same external ID and also start downloading both video and chat but with a delay from the other one. When stream end, both stop recording and render the same thing.

[Zibbp]

Ah, got it. Do you have any debug logs around the time the stream ended and the two queue items got created? Does this happen every time a stream is abruptly ended?

[soleny894]

Hi again, sorry I was away for a few days.

It happened again today and here's my idea of why it's doing it (correct me if I'm wrong):

• First stream start and a queue item is added after 6 seconds
• Stream end and is restarted instantly
• New stream gets a new External ID
• A new queue item is added to download the new stream
• Old queue item can't download anymore and shows error like "Failed to reload playlist: Unable to open URL"
• Old queue item close download and set a "Currently Live" variable to "false"
• After 6 seconds channel is checked and it sees that the channel is not considered live when it actually is
• A new item is added and start downloading the stream
• Now 3 items are in queue:
  • The old queue from the first stream
  • The queue item from the new freshly started stream
  • The duplicated queue item that was created because the old queue said the channel was not live anymore when download ended

I'm not sure of it but that's most likely what's happening. I saw it happen in the queue after I was curious to know if it would do it after a stream restarted very quickly.

[Zibbp]

That sounds about right, but your second point, "Stream end and is restarted instantly". Streamlink needs to detect the stream is down which takes ~5-10 seconds then the channel gets marked as "not live". This means that in theory the channel should still "marked as live" for those 5-10 seconds until Streamlink exits. Your explanation is probably happening, but I need to find out where / how.

Debug logs around the time this happens will help as I need to see exactly what's being logged

[soleny894]

I turned debug to true in the settings, I'll reply here when I have logs of this happening again

[soleny894]

@Zibbp I'm back with debug logs. it happened again yesterday and I've be able to catch a part of the logs when it's doing it: https://pastebin.com/YzP9D58A

• 56ebbc5c - 468 is the original livestream that ended
• 3d983b38 - 676 is the new stream that started quickly after the previous one ended
• 4c1af018 - 676 is the duplicated queue item that appeared a few seconds later and recorded the same thing as "3d983b38"

[Zibbp]

I think I see what's happening. The live channel check is detecting the stream is down before Streamlink ends. The live check then re-adds the new live stream before Streamlink ends. Once streamlink ends, the channel is marked as "not live" once again, which gets picked up again by the live check function.

I think the best way to fix this is to get rid of one of the functions that declares the channel as offline. I'm thinking the Streamlink one is probably the best one to get rid of but it will need to be tested. One issue I know that may happen with removing that one if the Stream goes down (Streamlink ends) but Twitch doesn't update in time to mark the stream as offline. If the stream is never marked as offline and the stream comes back up the new stream will not be archived.

I can push a commit to the repository that removes the Streamlink "not live" update if you're willing to test it.

[soleny894]

Sure, let me know when it's available.

[Zibbp]

Change is available on the :main tag, switch the :latest tag to :main of the API container, pull, and bring up.

[soleny894]

It's up and running on my cluster now, I'll report back with results.

[soleny894]

@Zibbp I have some results, It made things worse than before. When a stream have a network issue for a few seconds and then return (without restarting the stream) it closes the live download and render what was downloaded while the stream continue to run. I reverted back to the last version until we can find another way, maybe it needs a timeout or something?

[Zibbp]

I reverted that change and added a function to check if there is a queue item with the same external ID that is downloading a video. If that is true, then the "live check" will not add the stream again. This has been merged in https://github.com/Zibbp/ganymede/pull/206 and is available in the :main tag if you want to try using that again.

[soleny894]

I'm back on the main branch, I'll report back once I have some results again.

[Zibbp]

I've released this fix in v1.3.0. Any updates on how well it's working?

[soleny894]

Sorry for the late answer. I think it's fixed, at least I didn't get any duplicate or issues since the last commit you did. I'll report here again if I find anything wrong but for now it seems stable.