Add support for iTunes U (was: ERROR: Unsupported URL: https://itunes.apple.com/...)

vi commented 10 years ago

[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['--verbose', 'https://itunes.apple.com/us/itunes-u/uc-davis-symphony-orchestra/id403834767']
[debug] Encodings: locale 'UTF-8', fs 'UTF-8', out 'UTF-8', pref: 'UTF-8'
[debug] youtube-dl version 2014.01.03
[debug] Python version 2.6.6 - Linux-2.6.32-5-xen-686-i686-with-debian-6.0.4
[debug] Proxy map: {}
[generic] id403834767: Requesting header
WARNING: Falling back on generic information extractor.
[generic] id403834767: Downloading webpage
[generic] id403834767: Extracting information
ERROR: Unsupported URL: https://itunes.apple.com/us/itunes-u/uc-davis-symphony-orchestra/id403834767; please report this issue on https://yt-dl.org/bug . Be sure to call youtube-dl with the --verbose flag and include its complete output. Make sure you are using the latest version; type  youtube-dl -U  to update.
Traceback (most recent call last):
  File "/home/vi/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 487, in extract_info
    ie_result = ie.extract(url)
  File "/home/vi/bin/youtube-dl/youtube_dl/extractor/common.py", line 150, in extract
    return self._real_extract(url)
  File "/home/vi/bin/youtube-dl/youtube_dl/extractor/generic.py", line 332, in _real_extract
    raise ExtractorError(u'Unsupported URL: %s' % url)
ExtractorError: Unsupported URL: https://itunes.apple.com/us/itunes-u/uc-davis-symphony-orchestra/id403834767; please report this issue on https://yt-dl.org/bug . Be sure to call youtube-dl with the --verbose flag and include its complete output. Make sure you are using the latest version; type  youtube-dl -U  to update.

anovicecodemonkey commented 10 years ago

It will be a complex IE but not an impossible one it seems, just based on my basic research.

I am not sure how you got the URL you passed to youtube-dl as I don't use iTunes but I am going to assume that the URL for various courses can be sourced easily from within iTunes (or hopefully elsewhere) and is the same format.

The basic process appears to be:

Spoof the user-agent as iTunes.
Grab the attribute of the <feedURL> key from the xml file that results from the following URL:

https://buy.itunes.apple.com/WebObjects/MZFinance.woa/wa/com.apple.jingle.app.finance.DirectAction/subscribePodcast?id=403834767&wasWarnedAboutPodcasts=true

The only aspect of the URL that changes is the id passed to subscribePodcast. Each course/lecture has its own id.

Download the .xht feed at feedURL.
Do the usual with the downloaded .xht file to extract the video/audio URL and metadata information. The media file is always named enclosure.[file-extension-goes-here] so the attributes of the other XML keys will need to be used to properly name and provide metadata to the media file(s). Though, the metadata appears to be rich so that shouldn't be much of a problem.

Hopefully this helps somebody create a usable IE for iTunesU.

florianjacob commented 6 years ago

Might be helpful for implementing this or as an intermediary solution: https://github.com/rbrito/tunesviewer is an iTunesU Browse-GUI which can play the videos even in an external player. It seems unmaintained, but the url extraction and video player playback still works.

NewsGuyTor commented 6 years ago

A lot of progress has been done here: https://github.com/rg3/youtube-dl/pull/9590

ytdl-org / youtube-dl

Add support for iTunes U (was: ERROR: Unsupported URL: https://itunes.apple.com/...) #2097