Open RedPenguin2 opened 3 years ago
RedPenguin2
commented
3 years ago I may have made some progress but I have to watch more episodes downloaded now to verify.
I was using the ffmpeg versions from johnvansickle but temporarily removed them and went back to the default one from Ubuntu Repos and so far the 5min 30sec error is gone in that video.
Mr-Jake
commented
3 years ago I have also experienced this, but always thought it to be a CBS problem since it only occurs a small percentage of the time.
Did you try using HLS native instead of ffmpeg?
You could download each video twice and compare the CRCs of the files. If the CRC is a match, you have a good download. If the CRCs are different, download the video again until you have a match. I find this method faster and easier than having to watch each video where you could easily miss something.
RedPenguin2
commented
3 years ago Did you try using HLS native instead of ffmpeg?
You could download each video twice and compare the CRCs of the files. If the CRC is a match, you have a good download. If the CRCs are different, download the video again until you have a match. I find this method faster and easier than having to watch each video where you could easily miss something.
I have just now tried the native option, forgot about it from the past.
I did also try downloading a video twice and got the exact same md5sum.
I will have to see how it goes as I watch episodes but so far it looks promising. It seems like Kodi is "liking" or playing the videos somewhat better using native. It appears to seek easier and resume from pause a lot smoother.
RedPenguin2
commented
3 years ago Well it looks like so far for CBS, at least on my machine, - - hls-prefer-native is required.
I've watched at least 5, probably more episodes now and have not seen any audio issues like I mentioned in the OP.
I had some moments where I thought they returned but it was due to temporary player/network issues and not the videos during those times.
I guess for whatever reason there is something just not 100% compatible with CBS and ffmpeg. I know I've had that with other servers in the past but I never found out exactly why.
ryandesign
commented
3 years ago I've also been experiencing what I believe is this same issue with downloads from CBS for the past several months. I've hesitated to report it because of the intermittent nature of the problem. I encountered it today when downloading Monday's and Tuesday's episodes of The Late Show with Stephen Colbert. I then tried --hls-prefer-native for Wednesday's episode and did not encounter the problem, but since the problem is intermittent, I will continue to use --hls-prefer-native for future downloads and will report back if the problem occurs when using that flag.
As far as I understand it, the video is stored on the server as hundreds of ten-second chunks which are downloaded one after another and then combined. The way that I would describe the problem is that from time to time, the video portion of a chunk is mistakenly paired with the audio portion of the previous chunk. (You hear the same ten seconds of audio twice, but the video doesn't match it the second time it plays.) Then, the video freezes for ten seconds while the correct audio portion of the chunk plays. Then everything plays normally again until the next such error. The length of the complete file is thus too long by the number of errors times the length of a chunk. Seemingly at random, a file might download with no errors, or I might see dozens of errors in a single file.
If downloads normally happen at 20x, then each ten-second chunk downloads in half a second, so each line of download output is printed half a second apart. But sometimes there will be a noticeable pause of a second or five seconds or ten seconds, and I believe that such pauses coincide with the occurrence of the errors though I have not proved this and would not know whether it is a symptom or a cause.
I have experienced the issue on two different Macs, with macOS High Sierra 10.13.6 and macOS Big Sur 11.0.1, with different versions of Python 3.7/3.8/3.9.
ryandesign
commented
3 years ago Happened again with the latest Colbert episode from 3/4/2021, with youtube-dl 2021.03.03 running under Python 3.9.2. I was using --hls-prefer-native but it was apparently ignored because the output contained:
WARNING: hlsnative has detected features it does not support, extraction will be delegated to ffmpegAs suggested elsewhere, I installed the pycrypto module and then hlsnative downloads were possible again.
RedPenguin2
commented
3 years ago Happened again with the latest Colbert episode from 3/4/2021, with youtube-dl 2021.03.03 running under Python 3.9.2. I was using
--hls-prefer-nativebut it was apparently ignored because the output contained:WARNING: hlsnative has detected features it does not support, extraction will be delegated to ffmpegAs suggested elsewhere, I installed the pycrypto module and then hlsnative downloads were possible again.
Thanks for the info!
I know on my Ubuntu PC, it should be 1-2-3.
I know it will probably be more fun on Windows. I did find pre-built binaries for Windows but I have to see how you tell streamlink where it is.
I like how so many say Windows is more user friendly but for stuff like this, I find Windows more work....
RedPenguin2
commented
3 years ago I am having a strange issue that I do not know how to explain.
I have youtube-dl latest, on both my Ubuntu and Win 10 PCs and they have both always worked, no problem at all.
At first I thought it was the --hls-prefer-native but even using ffmpeg produces the exact same result.
For some reason, the Win 10 youtube-dl appears to download a CBS episode perfectly but the Ubuntu version will claim it downloaded 100% but end up with some strange 15-100MB or so file that is a blob with broken video and no audio.
ryandesign
commented
3 years ago I will continue to use
--hls-prefer-nativefor future downloads and will report back if the problem occurs when using that flag.
I have now observed the problem occur more than once when using --hls-prefer-native so using that flag is not a solution or workaround.
sometimes there will be a noticeable pause of a second or five seconds or ten seconds, and I believe that such pauses coincide with the occurrence of the errors though I have not proved this and would not know whether it is a symptom or a cause.
I can now confirm this observation. Here is a log I made of a download, piping the output of youtube-dl through ts -i %.T to show how long each line took. Corruptions seem to coincide with the lines that start with "00:00:1"—lines that took longer than 10 seconds.
putnam
commented
3 years ago I am able to repro this as well. It is non-deterministic in nature so there is no one video URL we can share that will repro, only that it will eventually do so. It's also really tricky to detect whether it happens, until you watch an affected file. Excellent work narrowing it down, @ryandesign
I also attempted to download through an alternative method and experienced similar random audio hiccups and haven't had time to dig deeper to understand why these don't affect playback in CBS/Paramount's own web player.
Jessecar96
commented
3 years ago I'm also having this issue when downloading from cbs/paramount+. I'm able to find out if one failed by looking at the file and comparing the subtitles to the audio later in the video. If they don't match then there's a glitch somewhere in it. I've noticed this only happens sometimes after downloading a bunch of episodes. I'm thinking it's some kind of weird rate limiting but I haven't been able to stay within it with anything I tried.
ryandesign
commented
3 years ago I don't believe it's rate-limiting since it can happen on the first video I download. It's just random. For all I know, maybe it's just that one of the Paramount Plus content delivery servers is misbehaving and it's just random chance which server you connect to.
I've been trying to remember to use the --keep-fragments option so that when I encounter the issue I can examine the fragments to see if the problem is already present in the fragments or is just a result of combining. I now have two downloads of the same episode: one with many occurrences of the problem and one with no problems. But the individual fragments don't appear to be valid mp4 files by themselves that VLC will recognize.
I used diff -rq to compare the "good" and "bad" set of fragments, but it reports that all fragments differ, so that's no help.
I got more insight by comparing the fragment sizes. In both the "good" and "bad" directories I made a text file of file sizes (BSD stat (e.g. macOS): stat -f '%z %N' *Frag*; GNU stat (e.g. Linux): stat -c '%s %n' *Frag*) Diffing the "good" and "bad" file size files showed 25 fragments differ in size, with the "bad" fragments slightly larger than the "good" ones, which is consistent with my previous observations that the "bad" versions contain extra data in the form of repeated audio. Looking at my timing log from downloading the "bad" version, there were 37 lines that took longer than 10 seconds. (There are many more [download] log lines than there are fragments—244 fragments, 3411 [download] log lines—so some of these lines must be for the same fragment).
RedPenguin2
commented
3 years ago I'm also having this issue when downloading from cbs/paramount+. I'm able to find out if one failed by looking at the file and comparing the subtitles to the audio later in the video. If they don't match then there's a glitch somewhere in it. I've noticed this only happens sometimes after downloading a bunch of episodes. I'm thinking it's some kind of weird rate limiting but I haven't been able to stay within it with anything I tried.
I just wanted to say TY again for this!
I wanted to get some shows but was leery of downloading & converting corrupted files. I knew there would be ways to check it but hated to do time-consuming checks but this method seems to work 1-2-3.
My first episode was perfect but my second one was unbelievably off, as the subtitles were not even close to matching.
RedPenguin2
commented
3 years ago I am not 100% sure what this has to do with the issue but I noticed for some reason, if an episode says download in like 10 minutes, it normally will be good. If it says it will take like 20 minutes+ to download then it will be broken.
Sipherdrakon
commented
3 years ago I am able to repro this as well. It is non-deterministic in nature so there is no one video URL we can share that will repro, only that it will eventually do so. It's also really tricky to detect whether it happens, until you watch an affected file. Excellent work narrowing it down, @ryandesign
I also attempted to download through an alternative method and experienced similar random audio hiccups and haven't had time to dig deeper to understand why these don't affect playback in CBS/Paramount's own web player.
Still having this same issue. I tried with the subtitle trick, but it still means I have to open each file. But I thought maybe with the audio loop it would effect the duration. Which it seems to. Looking at the duration from a specific video on the website vs. the duration of the downloaded file there was a 30 second discrepancy which would account for the audio loopback and subsequent video freeze. So testing a batch script (only thing I am relatively familiar with) to get the duration from the json printed from yt-dlp and duration from mediainfo to delete videos outside of a 2 second difference in the durations. Also experimenting with socket timeouts for anything longer than 5 seconds as I was looking through the comments and a long delay seems to potentially be the culprit in the audio loopback and video freeze.
UPDATE Don't use the socket-timeout and continue options together. They make a giant mess of the resulting video and audio. It was a nice thought to have it timeout when the download lags but picking up from that and continuing the download leads to a very horrible looking video that is completely unwatchable.
RedPenguin2
commented
3 years ago I know this isn't a fix for the actual issue but it's a nice workaround.
I found this script on the youtube-dl subreddit:
https://github.com/Grandfather-Paradox/paramount-dl
It's a bash (also Cygwin) script that batch downloads from paramount+ or CBS and compares the duration after and if it's not a match, it redownloads until good.
Sometimes you have to redownload like 3 times but eventually you end up with good files.
Sipherdrakon
commented
3 years ago I know this isn't a fix for the actual issue but it's a nice workaround.
I found this script on the youtube-dl subreddit:
https://github.com/Grandfather-Paradox/paramount-dl
It's a bash (also Cygwin) script that batch downloads from paramount+ or CBS and compares the duration after and if it's not a match, it redownloads until good.
Sometimes you have to redownload like 3 times but eventually you end up with good files.
Thanks for the link. I will use it for further testing, but for now try downloading using yt-dlp and specifically their aria2c downloader. Jump to the discussion over on the link I provided. It's mostly just me working through the problem since everyone seems willing to add their criticism without helping to combat the issue. But I have done a test of ffmpeg,native,and aria2c downloaders with verification on increased duration of downloaded files and their relevant ratio of good vs corrupt videos. I have come to the localized conclusion that aria2c might be the best working downloader to deal with the audio loopback and video freeze we have all been experiencing. Will try to migrate the bash script to something a little more windows friendly and test it on a much larger set of videos.
nothing2obvi
commented
2 years ago I know this isn't a fix for the actual issue but it's a nice workaround.
I found this script on the youtube-dl subreddit:
https://github.com/Grandfather-Paradox/paramount-dl
It's a bash (also Cygwin) script that batch downloads from paramount+ or CBS and compares the duration after and if it's not a match, it redownloads until good.
Sometimes you have to redownload like 3 times but eventually you end up with good files.
Wow, thanks for this. I've been going crazy trying to find a solution to this problem. Cheers!
benfugate
commented
2 years ago https://github.com/benfugate/paramountdl-dockerized I sped up Grandfather-Paradox's solution script, if anybody is interested. I changed up some things so seasons download concurrently.
I also 'dockerized' his script, so there's no need to install dependencies anymore. Although, I did add instructions for how to run without docker for those who are into that sort of thing.
The episodes are also now named based on their season and episode number (s1e1.mkv) instead of their paramount names which I had to go through and manually rename for my organization tools.
benfugate
commented
2 years ago If anyone can provide example example URL with the problem, I think I have a fix for this
I have had it on almost anything I haven't already downloaded at least once. Big Brother had many issues (per episode) throughout season 6 for me. One example I can see was redownloaded in yesterday's log was Picard, S02E02.
Commenter in which I am replying too has removed all their comments from this thread
ryandesign
commented
2 years ago There is no specific URL guaranteed to exhibit the problem. It occurs randomly with seemingly any CBS download URL.
putnam
commented
2 years ago As has been discussed numerous times in this thread, it is non-deterministic and doesn't affect any one single video. It's an artifact of using the service itself. I suggest reading the issue again, and if you need a repro you can simply download the same episode of a random show on the site repeatedly until it occurs. It won't take long.
If you want to contribute, maybe fork the repo, commit your change, or just discuss here what you think it is so we can help get to a solution together. It's not a race.
On Sat, Mar 12, 2022 at 9:11 PM Zombo @.***> wrote:
Maybe one of us can help you publish it
I dont need help publishing, I need help getting a test case
— Reply to this email directly, view it on GitHub https://github.com/ytdl-org/youtube-dl/issues/27972#issuecomment-1066012205, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAASBT4EHUYGWZPKXJQI3ZDU7VMFPANCNFSM4WS7NOQA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you are subscribed to this thread.Message ID: @.***>
Checklist
Verbose log
Description
It downloads the episodes just fine from a link grabbing standpoint and dumping to hard drive.
Yet, I keep having some sort of random audio drift on various episodes. In that episode, right around 00:05:30, the audio randomly looses sync and starts to repeat itself once or twice then maybe 30 seconds later it's back to normal.
I have tried upgrading my ffmpeg and using different audio players including VLC, Media Player Classic, and Kodi.
Not all episodes seem to do this but a majority do but I'd say it started only in October and affected more and more over time.