Episode appears in feed during youtube premiere and has incorrect duration

[algot]

Episodes which are youtube premieres appear in the feed during the premiere and have very short duration (~6-9 minutes) and contains YouTube premiere countdown sound at the beginning.

Examples:

1. https://www.youtube.com/watch?v=Be3aFgnB4JM. Feed duration - 9:04. Youtube duration - 1:02:10
2. https://www.youtube.com/watch?v=z5vJdh9d3zI. Feed duration - 9:00. Youtube duration - 1:31:26

[umputun]

2022/11/06 08:39:08.558 [INFO]  new entry [1] Be3aFgnB4JM, ЖЕЛЕЗНЫЙ ЗАНАВЕС НАВСЕГДА? // Александр Генис и Ксения Ларина, @The Insider  04.11.22 20:00 мск, Ксения Ларина, {ChannelID:UCmt9BtDz7nSiau3iLWgIGBQ, VideoID:Be3aFgnB4JM, Title:"ЖЕЛЕЗНЫЙ ЗАНАВЕС НАВСЕГДА? // Александр Генис и Ксения Ларина, @The Insider  04.11.22 20:00 мск", Published:2022-11-05T15:07:05-05:00, Updated:2022-11-06T02:13:08-06:00, Author:Ксения Ларина, File:, Duration:0s}
2022/11/06 08:39:09.873 [INFO]  ERROR: [youtube] Be3aFgnB4JM: Premieres in 20 minutes
2022/11/06 08:39:09.930 [WARN]  failed to download Be3aFgnB4JM: failed to execute command: exit status 1
2022/11/06 08:54:08.478 [INFO]  new entry [1] Be3aFgnB4JM, ЖЕЛЕЗНЫЙ ЗАНАВЕС НАВСЕГДА? // Александр Генис и Ксения Ларина, @The Insider  04.11.22 20:00 мск, Ксения Ларина, {ChannelID:UCmt9BtDz7nSiau3iLWgIGBQ, VideoID:Be3aFgnB4JM, Title:"ЖЕЛЕЗНЫЙ ЗАНАВЕС НАВСЕГДА? // Александр Генис и Ксения Ларина, @The Insider  04.11.22 20:00 мск", Published:2022-11-05T15:07:05-05:00, Updated:2022-11-06T02:13:08-06:00, Author:Ксения Ларина, File:, Duration:0s}
2022/11/06 08:54:09.985 [INFO]  ERROR: [youtube] Be3aFgnB4JM: Premieres in 5 minutes
2022/11/06 08:54:10.047 [WARN]  failed to download Be3aFgnB4JM: failed to execute command: exit status 1
2022/11/06 09:09:08.488 [INFO]  new entry [1] Be3aFgnB4JM, ЖЕЛЕЗНЫЙ ЗАНАВЕС НАВСЕГДА? // Александр Генис и Ксения Ларина, @The Insider  04.11.22 20:00 мск, Ксения Ларина, {ChannelID:UCmt9BtDz7nSiau3iLWgIGBQ, VideoID:Be3aFgnB4JM, Title:"ЖЕЛЕЗНЫЙ ЗАНАВЕС НАВСЕГДА? // Александр Генис и Ксения Ларина, @The Insider  04.11.22 20:00 мск", Published:2022-11-06T09:00:07-06:00, Updated:2022-11-06T09:00:07-06:00, Author:Ксения Ларина, File:, Duration:0s}
2022/11/06 09:09:29.967 [INFO]  downloaded Be3aFgnB4JM (ЖЕЛЕЗНЫЙ ЗАНАВЕС НАВСЕГДА? // Александр Генис и Ксения Ларина, @The Insider  04.11.22 20:00 мск) to var/yt/4d3fd6714844e875d1a3f424efa2e53d379fe7bd.mp3, size: 10929826, channel: {Name:Ксения Ларина ID:UCmt9BtDz7nSiau3iLWgIGBQ Type:channel Keep:0 Language:ru-ru Filter:{Include: Exclude:}}
2022/11/06 09:09:30.003 [INFO]  save 1667747369-b00ea2394b07e1bb716ed6ef9398cf1d27a2550e - {ChannelID:UCmt9BtDz7nSiau3iLWgIGBQ, VideoID:Be3aFgnB4JM, Title:"ЖЕЛЕЗНЫЙ ЗАНАВЕС НАВСЕГДА? // Александр Генис и Ксения Ларина, @The Insider  04.11.22 20:00 мск", Published:2022-11-06T09:09:29-06:00, Updated:2022-11-06T09:00:07-06:00, Author:Ксения Ларина, File:var/yt/4d3fd6714844e875d1a3f424efa2e53d379fe7bd.mp3, Duration:545s}
2022/11/06 09:09:30.005 [INFO]  set processed 4d3fd6714844e875d1a3f424efa2e53d379fe7bd - {ChannelID:UCmt9BtDz7nSiau3iLWgIGBQ, VideoID:Be3aFgnB4JM, Title:"ЖЕЛЕЗНЫЙ ЗАНАВЕС НАВСЕГДА? // Александр Генис и Ксения Ларина, @The Insider  04.11.22 20:00 мск", Published:2022-11-06T09:09:29-06:00, Updated:2022-11-06T09:00:07-06:00, Author:Ксения Ларина, File:var/yt/4d3fd6714844e875d1a3f424efa2e53d379fe7bd.mp3, Duration:545s}
2022/11/06 09:09:30.006 [INFO]  saved Be3aFgnB4JM (ЖЕЛЕЗНЫЙ ЗАНАВЕС НАВСЕГДА? // Александр Генис и Ксения Ларина, @The Insider  04.11.22 20:00 мск) to var/yt/4d3fd6714844e875d1a3f424efa2e53d379fe7bd.mp3, channel: {Name:Ксения Ларина ID:UCmt9BtDz7nSiau3iLWgIGBQ Type:channel Keep:0 Language:ru-ru Filter:{Include: Exclude:}}

this part of logs shows what it does. As soon as yt api feed shows the new episode feed-master requests download via yt-dlp. Initiall attempts to download failed on yt-dlp level with "Premieres in 20 minutes", "Premieres in 5 minutes" and so on, but at some point yt-dlp accepts the request and downloads it. Looks like it downloads a partial file in this case.

I'm not sure if such behavior is a bug on yt-dlp side, or maybe this is what it does by design. To me it looks like a bug. Another possibility - maybe yt-dlp has a flag to reject partial (live) downloads, but I don't know if this the case and what the flag

[umputun]

if anyone can reproduce the issue with yt-dlp and figure how to avoid it, pls make a pr or at least describe the fix you found

[umputun]

I think --match-filter "!is_live should do the trick. @algot -pls let me know if the issue fixed for you

[umputun]

apparently this is a little bit more complicated. The filter is working however it doesn't communicate back such a filtered entries as an error, and feed-master has no clue what happened. It detects such files as successfully downloaded but the actual file is absent.

I have altered the logic of youtube.feed.Downloader to check if file was actually dowloaded by yt-dlp and if not - this will be treated as any other temporary error and the next loop will try it again

[algot]

@umputun no episodes from Живой Гвоздь appears in the feed on the evening today https://www.youtube.com/channel/UCWAIvx2yYLK_xTYD4F2mUNw/streams

e.g. this one https://www.youtube.com/watch?v=8jsveJkBvyk

[umputun]

yeah, figured this one some time ago and the fix already in place

[umputun]

according to my logs it works fine now, this is an example:

new entry reported by yt and detected at 11:46, however it was "live" and was not downloaded, to eliminate partial file

2022/11/09 11:46:00.252 [INFO]  new entry [7] jR8M18WdCuQ, Цена вопроса / Сергей Алексашенко и Лиза Аникина // 09.11.2022, Живой Гвоздь, {ChannelID:UCWAIvx2yYLK_xTYD4F2mUNw, VideoID:jR8M18WdCuQ, Title:"Цена вопроса / Сергей Алексашенко и Лиза Аникина // 09.11.2022", Published:2022-11-08T11:26:03-06:00, Updated:2022-11-09T01:56:01-06:00, Author:Живой Гвоздь, File:, Duration:0s}
2022/11/09 11:46:01.873 [INFO]  skipping {ChannelID:UCWAIvx2yYLK_xTYD4F2mUNw, VideoID:jR8M18WdCuQ, Title:"Цена вопроса / Сергей Алексашенко и Лиза Аникина // 09.11.2022", Published:2022-11-08T11:26:03-06:00, Updated:2022-11-09T01:56:01-06:00, Author:Живой Гвоздь, File:, Duration:0s}

the same thing was detected at 12:01 again, this time it didn't have "is_live" and was fully downloaded and publshed

2022/11/09 12:01:00.233 [INFO]  new entry [7] jR8M18WdCuQ, Цена вопроса / Сергей Алексашенко и Лиза Аникина // 09.11.2022, Живой Гвоздь, {ChannelID:UCWAIvx2yYLK_xTYD4F2mUNw, VideoID:jR8M18WdCuQ, Title:"Цена вопроса / Сергей Алексашенко и Лиза Аникина // 09.11.2022", Published:2022-11-08T11:26:03-06:00, Updated:2022-11-09T12:00:24-06:00, Author:Живой Гвоздь, File:, Duration:0s}
2022/11/09 12:02:18.249 [INFO]  downloaded jR8M18WdCuQ (Цена вопроса / Сергей Алексашенко и Лиза Аникина // 09.11.2022) to var/yt/169000bd06c8c129ecc5afe0ccf03cca4b8bea9a.mp3, size: 71665500, channel: {Name:Живой Гвоздь ID:UCWAIvx2yYLK_xTYD4F2mUNw Type: Keep:8 Language:ru-ru Filter:{Include: Exclude:}}
2022/11/09 12:02:18.474 [INFO]  save 1667928363-b249486deea7d9baadccb2cf1520eeb10351a963 - {ChannelID:UCWAIvx2yYLK_xTYD4F2mUNw, VideoID:jR8M18WdCuQ, Title:"Живой Гвоздь: Цена вопроса / Сергей Алексашенко и Лиза Аникина // 09.11.2022", Published:2022-11-08T11:26:03-06:00, Updated:2022-11-09T12:00:24-06:00, Author:Живой Гвоздь, File:var/yt/169000bd06c8c129ecc5afe0ccf03cca4b8bea9a.mp3, Duration:3330s}
2022/11/09 12:02:18.477 [INFO]  set processed 169000bd06c8c129ecc5afe0ccf03cca4b8bea9a - {ChannelID:UCWAIvx2yYLK_xTYD4F2mUNw, VideoID:jR8M18WdCuQ, Title:"Живой Гвоздь: Цена вопроса / Сергей Алексашенко и Лиза Аникина // 09.11.2022", Published:2022-11-08T11:26:03-06:00, Updated:2022-11-09T12:00:24-06:00, Author:Живой Гвоздь, File:var/yt/169000bd06c8c129ecc5afe0ccf03cca4b8bea9a.mp3, Duration:3330s}
2022/11/09 12:02:18.478 [INFO]  saved jR8M18WdCuQ (Живой Гвоздь: Цена вопроса / Сергей Алексашенко и Лиза Аникина // 09.11.2022) to var/yt/169000bd06c8c129ecc5afe0ccf03cca4b8bea9a.mp3, channel: {Name:Живой Гвоздь ID:UCWAIvx2yYLK_xTYD4F2mUNw Type: Keep:8 Language:ru-ru Filter:{Include: Exclude:}}

feel free to close if the fix works as expected

[algot]

Here we go again This episode is reproduced the bug https://www.youtube.com/watch?v=CpEN96GhcH0

[umputun]

hmm, found a similar report in yt-dlp repo. Someone suggested to extend condition to --match-filter "!is_live & live_status!=is_upcoming & availability=public". I'm not sure what it supposed to resolve but changed, will see.

pls report if you see truncated files again

[orthur]

It seems only live videos have been processed now. E.g. two latest videos first and second from Майкл Наки channel not in feed

[umputun]

i have dropped live_status!=is_upcoming, it seems to bring those missing episodes back

[umputun]

I'm not sure if it solved the original problem; however, it solved another problem – the long (very long, 2+ hours) episodes are not truncated anymore.

[algot]

Original problem is solved I believe. But I will keep watching whether everything is alright

[algot]

Is reproduced again on this video

https://www.youtube.com/watch?v=7z-zNzeKxnw

[umputun]

Yeah, I have also seen partial downloads for some enormous episodes. Frankly, I have no clue what can be done here and am not even sure if smth can be done on the feed-master side. I'll mark this ticket as "help needed"; maybe someone will figure it out.

Btw, I still think the issue is in yt-dlp. You can try to reproduce it by running the same command feed-master running and observing what it downloaded. Maybe you will see some pattern and will be able to report it to yt-dlp. This is the command (template, replace {{.FileName}} and {{.ID}} parts):

yt-dlp --extract-audio --audio-format=mp3 --audio-quality=0 -f m4a/bestaudio "https://www.youtube.com/watch?v={{.ID}}" --no-progress -o {{.FileName}}.tmp --match-filter "!is_live & availability=public

[umputun]

Not much else we can do here