Open johnhawkinson opened 7 years ago
It looks like this is an issue with CSPAN livestreams, rather than recorded ones. Both of these streams seem to work now, using the cspan player:
...
[CSpan] 427706: Downloading webpage
[CSpan] 477121: Downloading JSON metadata
[CSpan] 477121: Downloading XML
[download] Downloading playlist: [International Refugee Assistance Project v. Trump]
[CSpan] playlist [International Refugee Assistance Project v. Trump]: Collected 12 video ids (downloading 12 of them)
[download] Downloading video 1 of 12
Let me know if anyone needs the full dump of the HTML pages during the livestream. I took a quick look and I'm not sure that all the necessary information to construct a brightcove URL is in the above brightcove-player-embed
div, but I haven't had a chance to spend much time on it.
after modifying BrightcoveNewIE
(need more work later for better AdobePass support):
python __main__.py http://players.brightcove.net/3162030207001/default_default/index.html?videoId=5250952122001
[brightcove:new] 5250952122001: Downloading webpage
[brightcove:new] 5250952122001: Downloading JSON metadata
[brightcove:new] 5250952122001: Retrieving Authorization Token
[brightcove:new] 5250952122001: Retrieving Media Token
[brightcove:new] 5250952122001: Downloading JSON metadata
[brightcove:new] 5250952122001: Downloading m3u8 information
[info] Available formats for 5250952122001:
format code extension resolution note
hls-64-0 mp4 audio only 64k , mp4a.40.2
hls-64-1 mp4 audio only 64k , mp4a.40.2
hls-264-0 mp4 480x270 264k , avc1.77.30, mp4a.40.2
hls-264-1 mp4 480x270 264k , avc1.77.30, mp4a.40.2
hls-464-0 mp4 640x360 464k , avc1.77.30, mp4a.40.2
hls-464-1 mp4 640x360 464k , avc1.77.30, mp4a.40.2
hls-664-0 mp4 960x540 664k , avc1.77.30, mp4a.40.2
hls-664-1 mp4 960x540 664k , avc1.77.30, mp4a.40.2
hls-1064-0 mp4 1024x576 1064k , avc1.77.30, mp4a.40.2
hls-1064-1 mp4 1024x576 1064k , avc1.77.30, mp4a.40.2 (best)
Yeah, it sounds like there are two issues:
BrightcoveNewIE
is apparently required to support the "obvious" URL. I wasn't sure that default_default
was right, but I had tried earlier with the current code, and got:$ ythls -v 'http://players.brightcove.net/3162030207001/default_default/index.html?videoId=5250952122001'
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--no-part', u'--hls-use-mpegts', u'-v', u'http://players.brightcove.net/3162030207001/default_default/index.html?videoId=5250952122001']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.05.07
[debug] Python version 2.7.10 - Darwin-14.5.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg git-2017-02-28-7f62368, ffprobe git-2017-02-28-7f62368, rtmpdump 2.4
[debug] Proxy map: {}
[brightcove:new] 5250952122001: Downloading webpage
[brightcove:new] 5250952122001: Downloading JSON metadata
ERROR: Video sources are restricted by access policy.
Traceback (most recent call last):
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 760, in extract_info
ie_result = ie.extract(url)
File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 433, in extract
ie_result = self._real_extract(url)
File "/usr/local/bin/youtube-dl/youtube_dl/extractor/brightcove.py", line 674, in _real_extract
error.get('message') or error.get('error_subcode') or error['error_code'], expected=True)
ExtractorError: Video sources are restricted by access policy.
Thanks.
@remitamine: Does not work for me:
pb3:youtube-dl jhawk$ python -m youtube_dl 'http://players.brightcove.net/3162030207001/default_default/index.html?videoId=5250952122001' -s -v
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'http://players.brightcove.net/3162030207001/default_default/index.html?videoId=5250952122001', u'-s', u'-v']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.05.07
[debug] Git HEAD: 5996d21ae
[debug] Python version 2.7.10 - Darwin-14.5.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg git-2017-02-28-7f62368, ffprobe git-2017-02-28-7f62368, rtmpdump 2.4
[debug] Proxy map: {}
[brightcove:new] 5250952122001: Downloading webpage
[brightcove:new] 5250952122001: Downloading JSON metadata
ERROR: An extractor error has occurred. (caused by KeyError(u'source_url',)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
File "youtube_dl/extractor/common.py", line 433, in extract
ie_result = self._real_extract(url)
File "youtube_dl/extractor/brightcove.py", line 610, in _real_extract
smuggled_data['source_url'], video_id,
KeyError: u'source_url'
Traceback (most recent call last):
File "youtube_dl/YoutubeDL.py", line 760, in extract_info
ie_result = ie.extract(url)
File "youtube_dl/extractor/common.py", line 446, in extract
raise ExtractorError('An extractor error has occurred.', cause=e)
ExtractorError: An extractor error has occurred. (caused by KeyError(u'source_url',)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Serving the previously-saved HTML locally, and with a small tweak, I see instead:
class CSpanIE(InfoExtractor):
- _VALID_URL = r'https?://(?:www\.)?c-span\.org/video/\?(?P<id>[0-9a-f]+)'
+ _VALID_URL = r'https?://(?:www\.|localhost\.)?c-span\.org/video/\?(?P<id>[0-9a-f]+)'
python -m youtube_dl -vs 'http://localhost.c-span.org/video/?427706-1/fourth-circuit-hears-oral-argument-travel-ban'
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-vs', u'http://localhost.c-span.org/video/?427706-1/fourth-circuit-hears-oral-argument-travel-ban']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.05.07
[debug] Git HEAD: 5996d21ae
[debug] Python version 2.7.10 - Darwin-14.5.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg git-2017-02-28-7f62368, ffprobe git-2017-02-28-7f62368, rtmpdump 2.4
[debug] Proxy map: {}
[CSpan] 427706: Downloading webpage
[brightcove:new] 5250952122001: Downloading webpage
[brightcove:new] 5250952122001: Downloading JSON metadata
ERROR: This video is only available for users of participating TV providers. Use --ap-mso to specify Adobe Pass Multiple-system operator Identifier and --ap-username and --ap-password or --netrc to provide account credentials.
Traceback (most recent call last):
File "youtube_dl/YoutubeDL.py", line 760, in extract_info
ie_result = ie.extract(url)
File "youtube_dl/extractor/common.py", line 433, in extract
ie_result = self._real_extract(url)
File "youtube_dl/extractor/brightcove.py", line 612, in _real_extract
custom_fields['bcadobepassresourceid'])
File "youtube_dl/extractor/adobepass.py", line 1377, in _extract_mvpd_auth
raise_mvpd_required()
File "youtube_dl/extractor/adobepass.py", line 1357, in raise_mvpd_required
'and --ap-username and --ap-password or --netrc to provide account credentials.', expected=True)
ExtractorError: This video is only available for users of participating TV providers. Use --ap-mso to specify Adobe Pass Multiple-system operator Identifier and --ap-username and --ap-password or --netrc to provide account credentials.
Is this correct? C-SPAN shouldn't require provider credentials to view their content at SD resolution, only at HD, I think? I am not exactly sure how this works, but it certainly works in a browser without credentials.
i don't think that the restriction is related to the quality, it is related to weather the served stream is live or replay. live streams are served using brightcove and they require provider credentials, while replays are free.
I watched the livestream via brightcove in my browser without provider credentials this afternoon. Here's a screenshot of the control-click information from the brightcove browser player:
C-SPAN documents their quality restriction here: https://www.c-span.org/about/TVeverywhere/
Online access to our three television channels (C‑SPAN, C‑SPAN2, and C‑SPAN3) is reserved for cable or satellite TV customers.
Sign-in to watch highest quality video here
Anyone can access all federal government events here.
so the SD quality is served using data-alt*
attributes:
youtube-dl -F http://players.brightcove.net/3617315736001/default_default/index.html?videoId=5258309162001
[brightcove:new] 5258309162001: Downloading webpage
[brightcove:new] 5258309162001: Downloading JSON metadata
[brightcove:new] 5258309162001: Downloading m3u8 information
[info] Available formats for 5258309162001:
format code extension resolution note
hls-64-0 mp4 audio only 64k , mp4a.40.2
hls-64-1 mp4 audio only 64k , mp4a.40.2
hls-264-0 mp4 480x270 264k , avc1.77.30, mp4a.40.2
hls-264-1 mp4 480x270 264k , avc1.77.30, mp4a.40.2
hls-464-0 mp4 640x360 464k , avc1.77.30, mp4a.40.2
hls-464-1 mp4 640x360 464k , avc1.77.30, mp4a.40.2 (best)
youtube-dl -F http://players.brightcove.net/3617315736001/default_default/index.html?videoId=5258323065001
[brightcove:new] 5258323065001: Downloading webpage
[brightcove:new] 5258323065001: Downloading JSON metadata
[brightcove:new] 5258323065001: Downloading m3u8 information
[info] Available formats for 5258323065001:
format code extension resolution note
hls-64-0 mp4 audio only 64k , mp4a.40.2
hls-64-1 mp4 audio only 64k , mp4a.40.2
hls-264-0 mp4 480x270 264k , avc1.77.30, mp4a.40.2
hls-264-1 mp4 480x270 264k , avc1.77.30, mp4a.40.2
hls-464-0 mp4 640x360 464k , avc1.77.30, mp4a.40.2
hls-464-1 mp4 640x360 464k , avc1.77.30, mp4a.40.2 (best)
I think I am a little confused how you ended up with accountid 3617315736001 instead of 3162030207001, but I agree that works. Edit: Oh, I see, that's from my screenshot.
Also, it seemed to me that 5996d21 hardcoded a bunch of stuff (like the '7001 id) without being very clear about where it came from, and I wasn't sure why if '&vod' not in url
was necessary. It seemed like this is fragile enough to want a test, and I guess it would have to be a faked-up test in tests/ rather than in the TESTS array in brightcove.py
.
I wasn't sure why if '&vod' not in url was necessary.
when i enter one of the live pages, it defaults to the live stream but if you click on play from the beginning
it adds &vod
to the url and switch to the c-span built-in service(non brightcove video), so i added it to follow what is happing in the browser.
related js code:
($(document).on({
click: function(h) {
h.preventDefault();
void 0 != $(this).data("progid") ? (hideBrightcovePlayer(), $("a.legacy").length && (h = $("a.legacy").attr("href"), h += "&vod", $("a.legacy").attr("href", h)), $("#video-embed").removeClass("hidden"), $("#video-embed").hasClass("player-not-init") && ($("#video-embed").removeClass("player-not-init"),
$("#video-embed").VLplayer(a)), h = window.location.href, -1 == h.indexOf("&vod") && (h += "&vod"), history.replaceState({}, document.title, h)) : void 0 != $(this).data("live") && ($("a.legacy").length && (h = $("a.legacy").attr("href"), h = h.replace("&vod", ""), $("a.legacy").attr("href", h)), showBrightcovePlayer(), $("#video-embed").replaceWith("<div id='video-embed' class='hidden player-not-init'></div>"), h = window.location.href, h = h.replace("&vod", ""), history.replaceState({}, document.title, h));
$(this).addClass("current").siblings("a.live-vod-toggle-link").removeClass("current")
}
},
"a.live-vod-toggle-link"), 0 <= window.location.href.indexOf("&vod") && $("a.live-vod-toggle-vod").click(), $(document).on({
click: function(a) {
return !0
}
}, "a.video-airing-toggle-link"))
hardcoded a bunch of stuff (like the '7001 id) without being very clear about where it came from
they represent the current values used by website for now, and it better to provide them as fallback rather than failing.
For what it's worth, looking at https://www.c-span.org/video/?428302-1/senate-democrats-lose-confidence-rod-rosenstein&live right now, current HEAD tries this:
pb3:youtube-dl jhawk$ python -m youtube_dl -v 'https://www.c-span.org/video/?428302-1/senate-democrats-lose-confidence-rod-rosenstein&live'
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'https://www.c-span.org/video/?428302-1/senate-democrats-lose-confidence-rod-rosenstein&live']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.05.09
[debug] Git HEAD: 3166b1f0a
[debug] Python version 2.7.10 - Darwin-14.5.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg git-2017-02-28-7f62368, ffprobe git-2017-02-28-7f62368, rtmpdump 2.4
[debug] Proxy map: {}
[CSpan] 428302: Downloading webpage
[brightcove:new] 5250955207001: Downloading webpage
[brightcove:new] 5250955207001: Downloading JSON metadata
ERROR: This video is only available for users of participating TV providers. Use --ap-mso to specify Adobe Pass Multiple-system operator Identifier and --ap-username and --ap-password or --netrc to provide account credentials.
but it should be the '6001 account ID instead of the '7001. Changing the code to override that and print a URL, we get this failure:
[CSpan] 428302: Downloading webpage
bc url http://players.brightcove.net/3617315736001/SyGGpuJy3g_default/index.html?videoId=5250955207001
[brightcove:new] 5250955207001: Downloading webpage
ERROR: Unable to download webpage: HTTP Error 404: Not Found (caused by HTTPError()); please repo
And apparently the embed code is the issue. Either using default_default
or S1eEqdyJ2e_default
works:
pb3:youtube-dl jhawk$ python -m youtube_dl -v 'http://players.brightcove.net/3617315736001/S1eEqdyJ2e_default/index.html?videoId=5258319040001'
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'http://players.brightcove.net/3617315736001/S1eEqdyJ2e_default/index.html?videoId=5258319040001']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.05.09
[debug] Git HEAD: 3166b1f0a
[debug] Python version 2.7.10 - Darwin-14.5.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg git-2017-02-28-7f62368, ffprobe git-2017-02-28-7f62368, rtmpdump 2.4
[debug] Proxy map: {}
[brightcove:new] 5258319040001: Downloading webpage
[brightcove:new] 5258319040001: Downloading JSON metadata
[brightcove:new] 5258319040001: Downloading m3u8 information
[debug] Invoking downloader on u'https://cspan2nontve-lh.akamaihd.net/i/CSpan2NonTVE_1@312669/index_400_av-b.m3u8?sd=10&rebase=on'
or
pb3:x1 jhawk$ PYTHONPATH=~/src/youtube-dl python -m youtube_dl -v 'http://players.brightcove.net/3617315736001/default_default/index.html?videoId=5258319040001'
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'http://players.brightcove.net/3617315736001/default_default/index.html?videoId=5258319040001']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.05.09
[debug] Git HEAD: 3166b1f0a
[debug] Python version 2.7.10 - Darwin-14.5.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg git-2017-02-28-7f62368, ffprobe git-2017-02-28-7f62368, rtmpdump 2.4
[debug] Proxy map: {}
[brightcove:new] 5258319040001: Downloading webpage
[brightcove:new] 5258319040001: Downloading JSON metadata
[brightcove:new] 5258319040001: Downloading m3u8 information
[debug] Invoking downloader on u'https://cspan2nontve-lh.akamaihd.net/i/CSpan2NonTVE_1@312669/index_400_av-b.m3u8?sd=10&rebase=on'
so fyi.
this patch will enable the free SD streams instead of HD streams:
diff --git a/youtube_dl/extractor/cspan.py b/youtube_dl/extractor/cspan.py
index 171820e27..14602e0fb 100644
--- a/youtube_dl/extractor/cspan.py
+++ b/youtube_dl/extractor/cspan.py
@@ -87,10 +87,10 @@ class CSpanIE(InfoExtractor):
if bc:
bc_attr = extract_attributes(bc)
bc_url = self.BRIGHTCOVE_URL_TEMPLATE % (
- bc_attr.get('data-bcaccountid', '3162030207001'),
- bc_attr.get('data-noprebcplayerid', 'SyGGpuJy3g'),
- bc_attr.get('data-newbcplayerid', 'default'),
- bc_attr['data-bcid'])
+ bc_attr.get('data-altbcaccountid', '3617315736001'),
+ bc_attr.get('data-altnoprebcplayerid', 'S1eEqdyJ2e'),
+ bc_attr.get('data-altnewbcplayerid', 'default'),
+ bc_attr['data-altbcid'])
return self.url_result(smuggle_url(bc_url, {'source_url': url}))
# We first look for clipid, because clipprog always appears before
python __main__.py -F 'https://www.c-span.org/video/?428302-1/senate-democrats-lose-confidence-rod-rosenstein&live'
[CSpan] 428302: Downloading webpage
[brightcove:new] 5258319040001: Downloading webpage
[brightcove:new] 5258319040001: Downloading JSON metadata
[brightcove:new] 5258319040001: Downloading m3u8 information
[info] Available formats for 5258319040001:
format code extension resolution note
hls-64-0 mp4 audio only 64k , mp4a.40.2
hls-64-1 mp4 audio only 64k , mp4a.40.2
hls-264-0 mp4 480x270 264k , avc1.77.30, mp4a.40.2
hls-264-1 mp4 480x270 264k , avc1.77.30, mp4a.40.2
hls-464-0 mp4 640x360 464k , avc1.77.30, mp4a.40.2
hls-464-1 mp4 640x360 464k , avc1.77.30, mp4a.40.2 (best)
Is anything holding up @remitamine's patch? https://github.com/rg3/youtube-dl/issues/13028#issuecomment-300591376
Is anything holding up @remitamine's patch?
making the change in https://github.com/rg3/youtube-dl/issues/13028#issuecomment-300591376 will make it impossible for users to get the HD stream, so the prefered way is to try to use the HD streams and fallback to SD stream in case of an error. i have other priorities for now, so i don't have much time to spend on the project this days.
(Sorry, this Issue was submitted prematurely. Now done editing. Sorry for any confusion.)
Description of your issue, suggested solution and other information
It appears that current live CSPAN videos are now Brightcove videos that don't work either with the cspan extractor or with the generic extractor. Two examples:
https://www.c-span.org/video/?427577-1/sally-yates-james-clapper-testify-russian-interference-2016-election&live and https://www.c-span.org/video/?427706-1/fourth-circuit-hears-oral-argument-travel-ban&live
Looks like the HTML has:
or
This also fails with
--force-generic
, i.e. with the Brightcove embed search in the generic extractor: