Open deant opened 6 years ago
deant
commented
6 years ago R.e. the "account-needed" thing: You need an account to play the videos in-browser, but you DON'T need one either to simply search for video titles/URLs on their website, nor to download videos via youtube-dl (unless something has just changed, and that's the problem with the videos above...?) Also, accounts are free.
deant
commented
6 years ago Well, I tried logging in and then using youtube-dl with cookies exported from my browser. Made no difference. Also tried the -u and -p parameters (username/password). Also made no difference.
philacid
commented
6 years ago Find the real mp4 file-link being played through browser
deant
commented
6 years ago Philacid, it's not an mp4 file as such, it's a stream, delivered in segments. Using the Network tab of Chrome's developer tools, I see that for the first video above, it gets its initial chunk of data from:
and then further segments follow as the video plays:
https://sbsvodns-vh.akamaihd.net/i/svod/SBS/managed/2018/06/06/2018-06-06_625550_,1000,1500,512,128,K.mp4.csmil/segment2_1_av.ts?set-akamai-hls-revision=5&id=AgDL1iFHlmWhBWpULVtIGTMRaiOcmjpCGuRzfFXBbF1g+t9aO5jx1Sh2LaI08dSkJPIhmqzy1D8I6Q%3d%3d&hdntl=exp=1529783786~acl=/i/svod/SBS/managed/2018/06/06/2018-06-06_625550_*~data=hdntl~hmac=96495c6649c53acfb3543e43a97a57facedfc3b197186f6a4a8fff5555064bd2 https://sbsvodns-vh.akamaihd.net/i/svod/SBS/managed/2018/06/06/2018-06-06_625550_,1000,1500,512,128,K.mp4.csmil/segment3_1_av.ts?set-akamai-hls-revision=5&id=AgDL1iFHlmWhBWpULVtIGTMRaiOcmjpCGuRzfFXBbF1g+t9aO5jx1Sh2LaI08dSkJPIhmqzy1D8I6Q%3d%3d&hdntl=exp=1529783786~acl=/i/svod/SBS/managed/2018/06/06/2018-06-06_625550_*~data=hdntl~hmac=96495c6649c53acfb3543e43a97a57facedfc3b197186f6a4a8fff5555064bd2 https://sbsvodns-vh.akamaihd.net/i/svod/SBS/managed/2018/06/06/2018-06-06_625550_,1000,1500,512,128,K.mp4.csmil/segment4_1_av.ts?set-akamai-hls-revision=5&id=AgDL1iFHlmWhBWpULVtIGTMRaiOcmjpCGuRzfFXBbF1g+t9aO5jx1Sh2LaI08dSkJPIhmqzy1D8I6Q%3d%3d&hdntl=exp=1529783786~acl=/i/svod/SBS/managed/2018/06/06/2018-06-06_625550_*~data=hdntl~hmac=96495c6649c53acfb3543e43a97a57facedfc3b197186f6a4a8fff5555064bd2
... etc.
(Note that the above 3 URLs are identical except for the segment2_1_av.ts, segment3_1_av.ts, segment4_1_av.ts, ... parts)
Does that help? Do you need the info for the other two videos also?
philacid
commented
6 years ago You are in the right area hypothetically. Perhaps playing around with the various links seen in the network analysis might work hypothetically. Not that I have done exactly that myself. Certainly I didn't do it with Firefox network analysis using the built in filters to show only media types in the network stream, and I didn't then use youtube-dl with what looked like the most likely file-link served on the akamaihd server. Saying any more could trigger an unwanted change to this hypothetical hole.
philacid
commented
6 years ago future updates to youtube-dl might allow passing username and password as I understand works already with other sites. I don't use OBS Studio when all else fails.
deant
commented
6 years ago I don't know OBS Studio, but actually at the moment "Flash Video Downloader" and (I think?) "Video DownloadHelper" (available for both Firefox and Chrome) will grab SBS Ondemand videos, once they've started playing in the browser, and it takes essentially just 2 mouseclicks to do it. That's what I try if youtube-dl doesn't work... though this week I had no further problems downloading a bunch of things with youtube-dl, so it must have been something very specifically different about those three videos.
@philacid, all the things you definitely don't do or use when all else fails are all very interesting, but this is a bug report posted with the view towards youtube-dl being able to download the video from just the page URL that it's on... so if the youtube-dl developers are going to fix this, they need the actual information about what works to download the video, vs whatever youtube-dl does currently, and not just vague hints. If you're one of those developers, and you've found the solution, then great - I look forward to the next update fixing this problem. If you're not, then you're being somewhat obtuse. I already posted a bunch of explicit links; I don't think another will make a difference. If you'd like me to follow your hinted directions and post the results, please tell me so explicitly: I'm happy to help if specific steps are required from someone in the right geo-location, but I don't have the time and initiative right now to play around and investigate off my own bat. I posted this bug because the youtube-dl error message told me to do so!
Also I'm not clear: Is your last post saying you've definitely confirmed that the existing youtube-dl URL is correct but that these 3 videos required user authentication on the video server itself (while most others obviously do not)? Or is that just a possible theory? When you [didn't] use youtube-dl on the most likely file-link served on the akamaihd server, what was[n't] the result: success, or failure? And could you be any clearer about what URL was[n't] that, exactly? Hypothetically speaking, of course? (i.e. was it just the 1st URL I posted above, or does it require alteration somehow?).
TommoCrabb
commented
6 years ago @deant, I came here because I was having a similar issue to you. I tried to download about 20 videos; most of them worked, but two or three failed with the same sort of output you posted. After reading this thread, I signed up for an account with SBS. I then retried downloading the videos that had failed earlier, only this time, I used the -u (username) and -p (password) flags with youtube-dl. This time, the downloads worked perfectly. If I had to guess, I'd say SBS has started requiring verification for certain videos. Maybe it's a gradual roll out?
Having said that, I tested the "John McCain, Maverick" link you posted above, and it downloaded for me just fine, even without the username and password flags. So, I dunno what the full story is.
deant
commented
6 years ago @TommoCrabb that's interesting, because I tried using the -u and -p options with youtube-dl, and it didn't help me. So maybe it's a different issue, or maybe the SBS server is undergoing ongoing changes. Alternatively, the way youtube-dl passes -u and -p options to the SBS server might have changed slightly, if there's been a new version release in the last 9 days... hard to say!
deant
commented
6 years ago OK, so this is interesting: All three videos I originally posted above ARE NOW WORKING, with exactly the same settings & youtube-dl version as I originally used (I didn't update). Something on the SBS server has evidently changed, for/about those videos.
@TommoCrabb, if you can reproduce your failure-to-download on your own problem videos, perhaps you could post the links here so that someone from youtube-dl will still have a non-working example video to diagnose?
TommoCrabb
commented
6 years ago @deant, just spit-balling here, but have you tried downloading those videos without all the extra options (--restrict-filenames, etc, etc)? Also, you know that the username is the full email address you signed up to SBS with, right?
Not trying to insult your intelligence. It just seemed worth asking.
TommoCrabb
commented
6 years ago Okay, here's one that I couldn't download without using the username and password options.
Stacey Dooley: Face to Face with Isis
@deant, I assume that's what you meant?
deant
commented
6 years ago @TommoCrabb Actually I wasn't sure if the username parameter was my displayed username or my email, but if memory serves correctly I tried both, just to be safe. I didn't try changing the other options, no; they didn't look like they had much to do with the error I was getting, and they're added by the frontend I usually use to invoke youtube-dl (a little paste-your-links floating box called YT-DL Dropper), which has until now always worked fine for me.
Interestingly, the same link you just posted, I also downloaded a few days ago, and it worked without any problems for me (and without any username/password parameters in the commandline options).
It may be this is just some kind of intermittent server problem affecting certain videos somehow, although it seems that the "effect" persists for 1-2 days at least once it happens (based on my experience with the the first 3 videos I posted).
Hopefully someone from youtube-dl will investigate while we have a non-working link to demonstrate to them...?
philacid
commented
6 years ago @deant Obtuse! Prize for stating the bleedin obvious even if you don't have the correct sense of obtuse; but I do understand what you meant to write.
Hmm, now why would someone living in Australia not post the URI's of the actual streams served from a TV provider, thus providing the exact means of defeating their file protection mechanism?
"I don't know OBS Studio" - now that's obtuse, like saying 'I don't know frickin google'. Seems to me from reading the thread so far, the devs have got enough info to make things simple again for the entitled among us, shame they are volunteers or you could crack the whip a little.
For the rest of us with no developer training but can use a search engine to learn a tiny bit about media serving, none of the files from this particular provider are inaccessible to youtube-dl because after starting to play the file in the browser, the network traffic monitor in Firefox very soon reveals only one link that makes sense to pass to youtube-dl.
deant
commented
6 years ago @philacid Well, checking some online dictionaries, most give the primary meaning of obtuse as "dull". But Oxford dictionary also allows "Difficult to understand, especially deliberately so". What I was going for was that, more-or-less, although "deliberately unhelpful" would probably be closer, and AFAIK obtuse is often used in exactly that sense, especially when you describe someone as "being obtuse". At least one person on English.stackexchange agrees with me on that definition.
Hmm, well, I live in Australia, and I'm fairly sure saving video streams for personal viewing is not illegal here, any more than recording a TV show on VCR would be. (Unless you then distribute the recording, of course). In any case there's zero chance SBS would bother coming after you for it. At worst they might cancel your free SBS user account, assuming there's any way they could connect your github username to it (and you could always just make another). Or possibly you think they're trawling github issue trackers for suspicious URLs to see if they have cause to improve their servers' security mechanisms? I assure you, they aren't. Although it's been discontinued now (I believe because the author didn't want to maintain it anymore), for years there was a program called SBSRipper, designed specifically for downloading SBS streaming content. It had it's own website and everything, and SBS didn't care. The well-known Australian internet forum Whirlpool maintains an ongoing, public, 9-year-old and much-used discussion thread specifically about how to download SBS OnDemand programs (part 1, part 2, part 3), as well as a page including some helpful technical information about the service. Which all goes to say, I think you're being overly paranoid. Waaaay overly. (Also, [sarcasm] if someone did want to come after you, I'm pretty sure that saying it's just what you "hypothetically didn't do" isn't a bulletproof legal defence strategy [/sarcasm]).
I dunno why you think I'm obtuse if I don't know OBS Studio. Just about everybody with a computer or smartphone searches the internet, so they know "frickin" Google. But not everyone does video recording or mixing on their computer - I don't, except to save some TV streams, and sometimes a YouTube or Vimeo video that I just use a browser add-on to get - so there's no reason to assume I'd have heard of some popular software package(s) for doing that. I never heard of OBS Studio before, and the fact is I'll probably never have reason to use it. And your usage of "obtuse" was rude.
Oh, I am a professional software developer, just BTW. (Not for youtube-dl, though, obviously). I think I'm not going to "crack the whip" on the youtube-dl bug team... as you point out, they are volunteers, and for now at least I'm going to assume that they know best where their valuable and freely-donated time most needs to be spent, without my screaming "Fix my bug!", "Fix my bug!" at them ...especially since there's an easy workaround anyway. But you know, you can go right ahead and do that yourself, if you're that-way inclined, it's a free country.
Make sure you are using the latest version: run
youtube-dl --versionand ensure your version is 2018.06.14. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.Before submitting an issue make sure you have:
What is the purpose of your issue?
I just tried to download 22 videos from https://sbs.com.au/ondemand 19 worked perfectly. 3 repeatedly gave errors. I have verified that I can get those 3 to play inside my browser (Google Chrome) although oddly I had to refresh the browser page a couple of times on each video before it played successfully. Note that these videos are probably region-locked to Australian users only, though I haven't confirmed that.
EDIT (9 days later): The three specific videos below now DO download correctly, using exactly the same settings & version of youtube-dl as was originally tried, so something on the server has changed. However, another user is having quite similar problems with some other SBS videos; see below.
The 3 --verbose mode logs are below:
(1)
(2)
(3)