Open Shithappenz opened 2 years ago
I can see the same issues.
In the first case, from the UK I get "video unavailable" but using --geo-bypass-country DE
gives a log similar to yours. I can't see what's happening, except that the API request to 'http://vas.sim-technik.de/vas/live/v2/videos/%s/sources/url' % clip_id
isn't working. Since no data is retrieved for any of the three failing sources, there are no formats. Why that might be is unclear: no such API calls occur when I fail to be able to play the show in the browser (presumably, proxy needed).
In the second case, which doesn't need --geo-bypass-country DE
, it's initially straightforward. Here is the data returned from the "videos JSON":
{u'duration': 0, u'sources': [], u'is_protected': True, u'id': u'6237106'}
Because video['is_protected']
is True
, the DRM error is raised. But if it hadn't been, there are no sources (video['sources']
) and so, as before, there would have been no formats, without the trouble of even asking for non-existent sources.
The extractor isn't completely broken as its tests 0, 7, 8, which are the only ones expected to pass, do succeed.
The errors you've reported also occur with yt-dlp, which is expected because the extractors are almost identical. Maybe the issue should be raised in that tracker as I see there are contributors there who have worked on related extractors.
Hi @dirkf !
Thank you for your fast response! I forgot to say, that I am from germany and therefore I don't use the geo-bypass-function. I hope there are some contributors, that can fix the broken extractor. I am sorry, that I can't do more than to report the bug.
The issue (at least the 404 one) still exists with other URLs. One example would be the currently running My Hero Academia Series (https://www.prosiebenmaxx.de/anime/my-hero-academia)
Here's an example.
R:\>youtube-dl` -v https://www.prosiebenmaxx.de/anime/my-hero-academia/video/415-schwelende-flammen-ganze-folge
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'https://www.prosiebenmaxx.de/anime/my-hero-academia/video/415-schwelende-flammen-ganze-folge']
[debug] Encodings: locale cp1252, fs mbcs, out cp850, pref cp1252
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.4.4 (CPython) - Windows-7-6.1.7601-SP1
[debug] exe versions: ffmpeg git-2020-03-24-e5d25d1, ffprobe git-2020-03-24-e5d25d1
[debug] Proxy map: {}
[prosiebensat1] anime/my-hero-academia/video/415-schwelende-flammen-ganze-folge: Downloading webpage
[prosiebensat1] 6237444: Downloading videos JSON
[prosiebensat1] 6237444: Downloading protocols JSON
WARNING: Unable to download JSON metadata: HTTP Error 404: Not Found
[prosiebensat1] 6237444: Downloading sources JSON
[prosiebensat1] 6237444: Downloading urls JSON
WARNING: Unable to download JSON metadata: HTTP Error 404: Not found
[prosiebensat1] 6237444: Downloading urls JSON
WARNING: Unable to download JSON metadata: HTTP Error 404: Not found
[prosiebensat1] 6237444: Downloading urls JSON
WARNING: Unable to download JSON metadata: HTTP Error 404: Not found
ERROR: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to
update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\YoutubeDL.py", line 815, in wrapper
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\YoutubeDL.py", line 836, in __extract_info
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\extractor\common.py", line 534, in extract
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\extractor\prosiebensat1.py", line 495, in _real_ex
tract
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\extractor\prosiebensat1.py", line 440, in _extract
_clip
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\extractor\prosiebensat1.py", line 162, in _extract
_video_info
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\extractor\common.py", line 1374, in _sort_formats
youtube_dl.utils.ExtractorError: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest versio
n; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
I'll give more insight once I figure out what and where my browser does to get the video.
Apparently something changed in the underlying system.
In my test I used https://www.prosiebenmaxx.de/anime/my-hero-academia/video/414-eine-strahlende-zukunft-ganze-folge
My browser currently calls
https://vas-v4.p7s1video.net/4.0/getsources?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6InhfcHJvc2llYmVubWF4eC1kZSJ9.eyJjb250ZW50X2lkcyI6eyJ2X2toZml5YzlqYmFmZyI6eyJzdWJjbGlwX3N0YXJ0IjowLCJzdWJjbGlwX2VuZCI6MTM3MX19LCJzZWN1cmVfZGVsaXZlcnkiOnRydWUsImlhdCI6MTY1MDgyNjExMSwibmJmIjoxNjUwODI1ODExLCJleHAiOjE2NTA4MjY0MTF9.Zj817nYUWonbYcmQS0BoYErkHYWzdmf-qbm9SvekFAA
to get all necessary data. (I don't see any interesting headers there.) Then it calls the linked dash URL while hls and progressive are available as well.
The plugin completely ignores that URL and first tries to get self._V4_BASE_URL + 'protocols'
resulting in a 404 and as fallback tries 'http://vas.sim-technik.de/vas/live/v2/videos/%s/sources/url' % clip_id
instead which now also results in the 404 for urls JSON.
The correct URL is built with self._V4_BASE_URL + 'sources?token=' + some_token
but I currently have no idea how that token is built. It is probably built somehow in source-fetcher.ts (some webpack code? Don't know where it gets its source from... source URI reads webpack:///source-fetcher/src/source-fetcher.ts
)
Here's some snippets to get a general grasp about where to search
const url = encodeURI(
`${getUrl(contentType, groupConfig)}?token=${getJWT(groupConfig, payload)}${
config.qaHeaders ? '&p7s1_debug=true' : ''
}`
);
const getJWT = ({ encryption_key, access_id }: Config, payload: VasSourcesPayload): string =>
jwt.sign(payload, encryption_key, {
algorithm: 'HS256',
header: { kid: access_id },
expiresIn: '5m',
notBefore: '-5m',
});
and lots more code to get the input data
[edit] found a link to that file in a referenced issue: https://gist.github.com/dirkf/35ffc6c1d1f3adf30fabacf11619f8b8
The issue (at least the 404 one) still exists with other URLs. One example would be the currently running My Hero Academia Series (https://www.prosiebenmaxx.de/anime/my-hero-academia) [...] I'll give more insight once I figure out what and where my browser does to get the video.
I am so sorry. Usually all clips or videos on prosiebenmaxx has a notice, when the clip or video will be deleted (e.g. „noch 3 Tage“ or „only 3 days“ in english). But the videos didn't show a notice, when I posted this issue. Yes, the problem is similar to the other videos of prosiebenmaxx. I found a little workaround. The direct url doesn't work:
c:\Download>youtube-dl.exe -U
youtube-dl is up-to-date (2021.12.17)
c:\Download>youtube-dl.exe "https://www.prosiebenmaxx.de/anime/my-hero-academia/video/415-schwelende-flammen-ganze-folge" -F --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['https://www.prosiebenmaxx.de/anime/my-hero-academia/video/415-schwelende-flammen-ganze-folge', '-F', '--verbose']
[debug] Encodings: locale cp1252, fs mbcs, out cp850, pref cp1252
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.19041
[debug] exe versions: ffmpeg 4.2.2, ffprobe 4.2.2
[debug] Proxy map: {}
[prosiebensat1] anime/my-hero-academia/video/415-schwelende-flammen-ganze-folge: Downloading webpage
[prosiebensat1] 6237444: Downloading videos JSON
[prosiebensat1] 6237444: Downloading protocols JSON
WARNING: Unable to download JSON metadata: HTTP Error 404: Not Found
[prosiebensat1] 6237444: Downloading sources JSON
[prosiebensat1] 6237444: Downloading urls JSON
WARNING: Unable to download JSON metadata: HTTP Error 404: Not found
[prosiebensat1] 6237444: Downloading urls JSON
WARNING: Unable to download JSON metadata: HTTP Error 404: Not found
[prosiebensat1] 6237444: Downloading urls JSON
WARNING: Unable to download JSON metadata: HTTP Error 404: Not found
ERROR: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\YoutubeDL.py", line 815, in wrapper
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\YoutubeDL.py", line 836, in __extract_info
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\extractor\common.py", line 534, in extract
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\extractor\prosiebensat1.py", line 495, in _real_extract
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\extractor\prosiebensat1.py", line 440, in _extract_clip
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\extractor\prosiebensat1.py", line 162, in _extract_video_info
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\extractor\common.py", line 1374, in _sort_formats
youtube_dl.utils.ExtractorError: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
But if I log in with the browser, take the playlist-url for dash, log out and put the dash-playlist-url in youtube-dl, it's still working. Very important: I didn’t log in via youtube-dl.exe and I didn’t use cookies!
c:\Download>youtube-dl.exe "https://vas-v4.p7s1video.net/4.0/playlist.mpd?x=01&y=s92O_P99T61PIOU6Pwsl3HUDoTRGlPMlN3jnsSotbAQCMWuval9hFuDH28kgw706HxSqYOIDMe6oHfsRb2uAqDXRGsnoHqm3PoypMe4lurkR4sz8eFAHV8GykL6qHqi2ezn0-_8sceKlsbGj4_eXT5_qgcUv2tGzeWyZPfG_uAAhDvpA_fQbnjt8Jl8_Gvv2GHgINVb_n-I_JI4GvKUbCUxIokmf5Lgbux5HAHBpbhkxLC15qjbbYj6bWkl1rn6-yBlBzZTpZncLlK8tyIA_5HQIWMXdmvMx6R0mE_q9elzIakzUEhT4BzGC6Gccyg8G6o2f6jZACWX2DxW2ITGUpe1TmjV7BfvjuHaiFGF8k54" -F --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['https://vas-v4.p7s1video.net/4.0/playlist.mpd?x=01&y=s92O_P99T61PIOU6Pwsl3HUDoTRGlPMlN3jnsSotbAQCMWuval9hFuDH28kgw706HxSqYOIDMe6oHfsRb2uAqDXRGsnoHqm3PoypMe4lurkR4sz8eFAHV8GykL6qHqi2ezn0-_8sceKlsbGj4_eXT5_qgcUv2tGzeWyZPfG_uAAhDvpA_fQbnjt8Jl8_Gvv2GHgINVb_n-I_JI4GvKUbCUxIokmf5Lgbux5HAHBpbhkxLC15qjbbYj6bWkl1rn6-yBlBzZTpZncLlK8tyIA_5HQIWMXdmvMx6R0mE_q9elzIakzUEhT4BzGC6Gccyg8G6o2f6jZACWX2DxW2ITGUpe1TmjV7BfvjuHaiFGF8k54', '-F', '--verbose']
[debug] Encodings: locale cp1252, fs mbcs, out cp850, pref cp1252
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.19041
[debug] exe versions: ffmpeg 4.2.2, ffprobe 4.2.2
[debug] Proxy map: {}
[generic] playlist: Requesting header
WARNING: Falling back on generic information extractor.
[generic] playlist: Downloading webpage
[generic] playlist: Extracting information
[info] Available formats for playlist:
format code extension resolution note
f1-a1-x3 m4a audio only DASH audio 64k , m4a_dash container, mp4a.40.2 (48000Hz)
f2-a1-x3 m4a audio only DASH audio 96k , m4a_dash container, mp4a.40.2 (48000Hz)
f3-a1-x3 m4a audio only DASH audio 128k , m4a_dash container, mp4a.40.2 (48000Hz)
f4-v1-x3 mp4 640x360 DASH video 344k , mp4_dash container, avc1.4d401e, 25fps, video only
f5-v1-x3 mp4 640x360 DASH video 493k , mp4_dash container, avc1.4d401e, 25fps, video only
f6-v1-x3 mp4 768x432 DASH video 839k , mp4_dash container, avc1.4d401e, 25fps, video only
f7-v1-x3 mp4 768x432 DASH video 1187k , mp4_dash container, avc1.4d401e, 25fps, video only
f8-v1-x3 mp4 1024x576 DASH video 2582k , mp4_dash container, avc1.4d401f, 25fps, video only
f9-v1-x3 mp4 1280x720 DASH video 3567k , mp4_dash container, avc1.4d401f, 25fps, video only (best)
Therefore the problem is to get the playlist-url or getsources-url like you think. I found an old entry of youtube-dlp, where one user (NeroBurner) tried to implement "getJWT" (JSON Web Token): https://github.com/yt-dlp/yt-dlp/issues/988 Maybe this could be helpful.
Are those streams downloadable under the same conditions? 403 from UK, even with --geo-bypass-country DE
.
The URLs seem to expire pretty fast. Seems like something between 5 and 10 minutes. It worked in my try as well, though. The payload encrypted in the token looks like this (exp probably meaning expiring)
header = {
"alg": "HS256",
"typ": "JWT",
"kid": "x_prosiebenmaxx-de"
}
payload = {
"content_ids": {
"v_khfiyc9jbafg": {
"subclip_start": 0,
"subclip_end": 1371
}
},
"secure_delivery": true,
"iat": 1650907516,
"nbf": 1650907216,
"exp": 1650907816
}
In multipart files (movies) it looks similar
payload = {
"content_ids": {
"v_c5zwq9l5bzui": [
{
"subclip_start": 0,
"subclip_end": 1431.75,
"id": 0
},
{
"subclip_start": 1421.75,
"subclip_end": 2863.5,
"id": 1
},
{
"subclip_start": 2853.5,
"subclip_end": 4295.25,
"id": 2
},
{
"subclip_start": 4285.25,
"subclip_end": 5727,
"id": 3
}
]
},
"secure_delivery": true,
"iat": 1650908517,
"nbf": 1650908217,
"exp": 1650908817
}
subclip_start
seems to be the last duration -10 seconds and subclip_end
just adds up the duration found in the page source for each subclip.
For reference I used this link and the video id to look for is v_c5zwq9l5bzui
I've looked through the changes to yt-dlp mentioned a few posts earlier. Seems like I got everything necessary figured out. The encryption key is already stored correctly in _ENCRYPTION_KEY
, so only the update to new behavior is missing.
I'll try to fix it once I find time again.
That sounds great. I am sorry, that I am not a bigger help. I am from germany. If I should try some new code for prosiebenmaxx, then you can ask me (you just have to explain, how I can run the new code).
I did some testing and copy pasting of what was done in commit https://github.com/yt-dlp/yt-dlp/pull/1012/commits/c008b59df863d0ec50cdd402e39d584254d7352d and downloading anime seems to work again.
Since my code is still full of print() functions for debugging and other untested stuff I won't create a pull request to the official repo just yet.
I pushed my code to https://github.com/Yavos/youtube-dl-1/tree/prosiebensat1-dev which currently completely circumvents the old code for downloading clips. (Bonus: season_number and episode_number now work.) No idea if this breaks anything (e.g. with prosieben or sat1) or if playlists work or not. I didn't test it yet.
@Shithappenz If you want to use that code you need: python3 installed and a local copy of that branch I linked above. In my tests I start the plugin with path_to_local_copy\youtube-dl\__main__.py <parameters>
Here's some example how it looks when I use it R:\>d:\Portable\youtube-dl\python\youtube_dl\__main__.py -r 2M -o "%(series)s\S%(season_number)02dE%(episode_number)02d %(title)s-%(id)s.%(ext)s" https://www.prosiebenmaxx.de/anime/date-a-live/video/41-keine-panik-das-ist-nur-die-falle-eines-geistes-ganze-folge
If you get an error for opening the py file you probably need to add python to your path or add a python
or path_to_python.exe
in front of the whole other stuff (plus a space).
I kind of got playlists working now. But I still run into 2 problems which I could need some help with:
'_type': 'multi_video'
entries to the playlist. Trying to do so results in the following error
Traceback (most recent call last):
File "D:\Portable\youtube-dl\python\youtube_dl\__main__.py", line 19, in <module>
youtube_dl.main()
File "D:\Portable\youtube-dl\python\youtube_dl\__init__.py", line 475, in main
_real_main(argv)
File "D:\Portable\youtube-dl\python\youtube_dl\__init__.py", line 465, in _real_main
retcode = ydl.download(all_urls)
File "D:\Portable\youtube-dl\python\youtube_dl\YoutubeDL.py", line 2078, in download
url, force_generic_extractor=self.params.get('force_generic_extractor', False))
File "D:\Portable\youtube-dl\python\youtube_dl\YoutubeDL.py", line 808, in extract_info
return self.__extract_info(url, ie, download, extra_info, process)
File "D:\Portable\youtube-dl\python\youtube_dl\YoutubeDL.py", line 815, in wrapper
return func(self, *args, **kwargs)
File "D:\Portable\youtube-dl\python\youtube_dl\YoutubeDL.py", line 847, in __extract_info
return self.process_ie_result(ie_result, download, extra_info)
File "D:\Portable\youtube-dl\python\youtube_dl\YoutubeDL.py", line 933, in process_ie_result
return self.__process_playlist(ie_result, download)
File "D:\Portable\youtube-dl\python\youtube_dl\YoutubeDL.py", line 1067, in __process_playlist
entry_result = self.__process_iterable_entry(entry, download, extra)
File "D:\Portable\youtube-dl\python\youtube_dl\YoutubeDL.py", line 815, in wrapper
return func(self, *args, **kwargs)
File "D:\Portable\youtube-dl\python\youtube_dl\YoutubeDL.py", line 1077, in __process_iterable_entry
entry, download=download, extra_info=extra_info)
File "D:\Portable\youtube-dl\python\youtube_dl\YoutubeDL.py", line 923, in process_ie_result
webpage_url = ie_result['webpage_url']
KeyError: 'webpage_url'
Hey, any news on this? I'd love to see your changes merged.
@pannal @Yavos Yavos did an amazing work. I am using his code since two months and it woks great. @Yavos: Thank you so much for fixxing the download of videos from the platform prosiebenmaxx. The download of movies or a playlist doesn't work, but I don't know why. I solved it with a little workaround: I downloaded the parts of the movie and concatenate them via FFmpeg. But this workaround is not perfect. In the playlists there are the first ten seconds of a video the last 10 seconds of the video before. E.g. the first 10 seconds of the 2. video are the last 10 seconds of the 1. video in a playlist. Therefore my workaround has repititions.
Weird. I thought I had that movie problem solved if you don't download from playlists. But apparently the '_type': 'multi_video'
doesn't work at all. I'll look into this.
Maybe I need to switch to yt-dlp or some other variant that's more maintained to find a suitable workaround...
[edit] as a reminder for myself, the times for concatenation (they usually overlap by 10 seconds - see https://github.com/ytdl-org/youtube-dl/issues/30598#issuecomment-1108865434) are already present in _extract_clip
. Just need to somehow use this info for later processing:
# extract content ids
contentResource = video_page['contentResource']
content_id = contentResource[0]['id']
series = contentResource[0]['tvShowTitle']
content_ids = [({'id': id, 'subclip_start': content['start'], 'subclip_end': content['end']} if content['contentType'] else {})
for id, content in enumerate(contentResource)]
@Yavos
Weird. I thought I had that movie problem solved if you don't download from playlists.
Sorry, I forgot to mention, that all big movies (length around 1 hour or more) on prosiebenmaxx are saved as playlists. The video of a series (lenght around 20 minutes) is not saved as a playlist. The download of videos works perfect!
Yeah, I know that movies are saved as playlists. But that's exactly what the multi_video type is supposed to be for. It should automatically concatenate them into a single file ignoring that it's actually a playlist. Don't know why it won't work.
@Yavos I tried it (https://www.prosiebenmaxx.de/anime/anime-filme/videos/i-want-to-eat-your-pancreas-ganze-folge2) and I got the following error. Maybe there is a problem in naming the parts of the playlist?
C:\youtube-dl-test\youtube_dl>__main__.py "https://www.prosiebenmaxx.de/anime/anime-filme/videos/i-want-to-eat-your-pancreas-ganze-folge2" -o "%(series)s\S%(season_number)02dE%(episode_number)02d %(title)s-%(id)s.%(ext)s" -F --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['https://www.prosiebenmaxx.de/anime/anime-filme/videos/i-want-to-eat-your-pancreas-ganze-folge2', '-o', '%(series)s\\S%(season_number)02dE%(episode_number)02d %(title)s-%(id)s.%(ext)s', '-F', '--verbose']
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.9.1 (CPython) - Windows-10-10.0.19041-SP0
[debug] exe versions: ffmpeg 4.2.2, ffprobe 4.2.2
[debug] Proxy map: {}
[prosiebensat1] anime/anime-filme/videos/i-want-to-eat-your-pancreas-ganze-folge2: Downloading webpage
[prosiebensat1] v_cpk1o4o25imp: Downloading videos JSON
[prosiebensat1] v_cpk1o4o25imp: Downloading MPD manifest
[prosiebensat1] v_cpk1o4o25imp: Downloading m3u8 information
[prosiebensat1] v_cpk1o4o25imp: Downloading MPD manifest
[prosiebensat1] v_cpk1o4o25imp: Downloading m3u8 information
[prosiebensat1] v_cpk1o4o25imp: Downloading MPD manifest
[prosiebensat1] v_cpk1o4o25imp: Downloading m3u8 information
[prosiebensat1] v_cpk1o4o25imp: Downloading MPD manifest
[prosiebensat1] v_cpk1o4o25imp: Downloading m3u8 information
[download] Downloading playlist: I Want to Eat Your Pancreas
[prosiebensat1] playlist I Want to Eat Your Pancreas: Collected 4 video ids (downloading 4 of them)
[download] Downloading video 1 of 4
[info] Available formats for v_cpk1o4o25imp:
format code extension resolution note
hls-audio0-default mp4 audio only
dash-f1-a1-x3 m4a audio only DASH audio 64k , m4a_dash container, mp4a.40.2 (48000Hz)
dash-f2-a1-x3 m4a audio only DASH audio 96k , m4a_dash container, mp4a.40.2 (48000Hz)
dash-f3-a1-x3 m4a audio only DASH audio 128k , m4a_dash container, mp4a.40.2 (48000Hz)
dash-f4-v1-x3 mp4 640x360 DASH video 346k , mp4_dash container, avc1.4d401e, 25fps, video only
hls-410 mp4 640x360 410k , avc1.4d401e, 25.0fps, video only
dash-f5-v1-x3 mp4 640x360 DASH video 496k , mp4_dash container, avc1.4d401e, 25fps, video only
hls-560 mp4 640x360 560k , avc1.4d401e, 25.0fps, video only
dash-f6-v1-x3 mp4 768x432 DASH video 845k , mp4_dash container, avc1.4d401e, 25fps, video only
hls-909 mp4 768x432 909k , avc1.4d401e, 25.0fps, video only
dash-f7-v1-x3 mp4 768x432 DASH video 1195k , mp4_dash container, avc1.4d401e, 25fps, video only
hls-1259 mp4 768x432 1259k , avc1.4d401e, 25.0fps, video only
dash-f8-v1-x3 mp4 1024x576 DASH video 2591k , mp4_dash container, avc1.4d401f, 25fps, video only
hls-2655 mp4 1024x576 2655k , avc1.4d401f, 25.0fps, video only
dash-f9-v1-x3 mp4 1280x720 DASH video 3584k , mp4_dash container, avc1.4d401f, 25fps, video only
hls-3648 mp4 1280x720 3648k , avc1.4d401f, 25.0fps, video only
progressive mp4 unknown (best)
[download] Downloading video 2 of 4
[info] Available formats for v_cpk1o4o25imp:
format code extension resolution note
hls-audio0-default mp4 audio only
dash-f1-a1-x3 m4a audio only DASH audio 64k , m4a_dash container, mp4a.40.2 (48000Hz)
dash-f2-a1-x3 m4a audio only DASH audio 96k , m4a_dash container, mp4a.40.2 (48000Hz)
dash-f3-a1-x3 m4a audio only DASH audio 128k , m4a_dash container, mp4a.40.2 (48000Hz)
dash-f4-v1-x3 mp4 640x360 DASH video 346k , mp4_dash container, avc1.4d401e, 25fps, video only
hls-410 mp4 640x360 410k , avc1.4d401e, 25.0fps, video only
dash-f5-v1-x3 mp4 640x360 DASH video 496k , mp4_dash container, avc1.4d401e, 25fps, video only
hls-560 mp4 640x360 560k , avc1.4d401e, 25.0fps, video only
dash-f6-v1-x3 mp4 768x432 DASH video 845k , mp4_dash container, avc1.4d401e, 25fps, video only
hls-909 mp4 768x432 909k , avc1.4d401e, 25.0fps, video only
dash-f7-v1-x3 mp4 768x432 DASH video 1195k , mp4_dash container, avc1.4d401e, 25fps, video only
hls-1259 mp4 768x432 1259k , avc1.4d401e, 25.0fps, video only
dash-f8-v1-x3 mp4 1024x576 DASH video 2591k , mp4_dash container, avc1.4d401f, 25fps, video only
hls-2655 mp4 1024x576 2655k , avc1.4d401f, 25.0fps, video only
dash-f9-v1-x3 mp4 1280x720 DASH video 3584k , mp4_dash container, avc1.4d401f, 25fps, video only
hls-3648 mp4 1280x720 3648k , avc1.4d401f, 25.0fps, video only
progressive mp4 unknown (best)
[download] Downloading video 3 of 4
[info] Available formats for v_cpk1o4o25imp:
format code extension resolution note
hls-audio0-default mp4 audio only
dash-f1-a1-x3 m4a audio only DASH audio 64k , m4a_dash container, mp4a.40.2 (48000Hz)
dash-f2-a1-x3 m4a audio only DASH audio 96k , m4a_dash container, mp4a.40.2 (48000Hz)
dash-f3-a1-x3 m4a audio only DASH audio 128k , m4a_dash container, mp4a.40.2 (48000Hz)
dash-f4-v1-x3 mp4 640x360 DASH video 346k , mp4_dash container, avc1.4d401e, 25fps, video only
hls-410 mp4 640x360 410k , avc1.4d401e, 25.0fps, video only
dash-f5-v1-x3 mp4 640x360 DASH video 496k , mp4_dash container, avc1.4d401e, 25fps, video only
hls-560 mp4 640x360 560k , avc1.4d401e, 25.0fps, video only
dash-f6-v1-x3 mp4 768x432 DASH video 845k , mp4_dash container, avc1.4d401e, 25fps, video only
hls-909 mp4 768x432 909k , avc1.4d401e, 25.0fps, video only
dash-f7-v1-x3 mp4 768x432 DASH video 1195k , mp4_dash container, avc1.4d401e, 25fps, video only
hls-1259 mp4 768x432 1259k , avc1.4d401e, 25.0fps, video only
dash-f8-v1-x3 mp4 1024x576 DASH video 2591k , mp4_dash container, avc1.4d401f, 25fps, video only
hls-2655 mp4 1024x576 2655k , avc1.4d401f, 25.0fps, video only
dash-f9-v1-x3 mp4 1280x720 DASH video 3584k , mp4_dash container, avc1.4d401f, 25fps, video only
hls-3648 mp4 1280x720 3648k , avc1.4d401f, 25.0fps, video only
progressive mp4 unknown (best)
[download] Downloading video 4 of 4
[info] Available formats for v_cpk1o4o25imp:
format code extension resolution note
hls-audio0-default mp4 audio only
dash-f1-a1-x3 m4a audio only DASH audio 64k , m4a_dash container, mp4a.40.2 (48000Hz)
dash-f2-a1-x3 m4a audio only DASH audio 96k , m4a_dash container, mp4a.40.2 (48000Hz)
dash-f3-a1-x3 m4a audio only DASH audio 128k , m4a_dash container, mp4a.40.2 (48000Hz)
dash-f4-v1-x3 mp4 640x360 DASH video 346k , mp4_dash container, avc1.4d401e, 25fps, video only
hls-410 mp4 640x360 410k , avc1.4d401e, 25.0fps, video only
dash-f5-v1-x3 mp4 640x360 DASH video 496k , mp4_dash container, avc1.4d401e, 25fps, video only
hls-560 mp4 640x360 560k , avc1.4d401e, 25.0fps, video only
dash-f6-v1-x3 mp4 768x432 DASH video 845k , mp4_dash container, avc1.4d401e, 25fps, video only
hls-909 mp4 768x432 909k , avc1.4d401e, 25.0fps, video only
dash-f7-v1-x3 mp4 768x432 DASH video 1195k , mp4_dash container, avc1.4d401e, 25fps, video only
hls-1259 mp4 768x432 1259k , avc1.4d401e, 25.0fps, video only
dash-f8-v1-x3 mp4 1024x576 DASH video 2591k , mp4_dash container, avc1.4d401f, 25fps, video only
hls-2655 mp4 1024x576 2655k , avc1.4d401f, 25.0fps, video only
dash-f9-v1-x3 mp4 1280x720 DASH video 3584k , mp4_dash container, avc1.4d401f, 25fps, video only
hls-3648 mp4 1280x720 3648k , avc1.4d401f, 25.0fps, video only
progressive mp4 unknown (best)
[download] Finished downloading playlist: I Want to Eat Your Pancreas
C:\youtube-dl-test\youtube_dl>__main__.py "https://www.prosiebenmaxx.de/anime/anime-filme/videos/i-want-to-eat-your-pancreas-ganze-folge2" -o "%(series)s\S%(season_number)02dE%(episode_number)02d %(title)s-%(id)s.%(ext)s" --verbose -f "dash-f9-v1-x3"+"dash-f3-a1-x3"
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['https://www.prosiebenmaxx.de/anime/anime-filme/videos/i-want-to-eat-your-pancreas-ganze-folge2', '-o', '%(series)s\\S%(season_number)02dE%(episode_number)02d %(title)s-%(id)s.%(ext)s', '--verbose', '-f', 'dash-f9-v1-x3+dash-f3-a1-x3']
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.9.1 (CPython) - Windows-10-10.0.19041-SP0
[debug] exe versions: ffmpeg 4.2.2, ffprobe 4.2.2
[debug] Proxy map: {}
[prosiebensat1] anime/anime-filme/videos/i-want-to-eat-your-pancreas-ganze-folge2: Downloading webpage
[prosiebensat1] v_cpk1o4o25imp: Downloading videos JSON
[prosiebensat1] v_cpk1o4o25imp: Downloading MPD manifest
[prosiebensat1] v_cpk1o4o25imp: Downloading m3u8 information
[prosiebensat1] v_cpk1o4o25imp: Downloading MPD manifest
[prosiebensat1] v_cpk1o4o25imp: Downloading m3u8 information
[prosiebensat1] v_cpk1o4o25imp: Downloading MPD manifest
[prosiebensat1] v_cpk1o4o25imp: Downloading m3u8 information
[prosiebensat1] v_cpk1o4o25imp: Downloading MPD manifest
[prosiebensat1] v_cpk1o4o25imp: Downloading m3u8 information
[download] Downloading playlist: I Want to Eat Your Pancreas
[prosiebensat1] playlist I Want to Eat Your Pancreas: Collected 4 video ids (downloading 4 of them)
[download] Downloading video 1 of 4
[debug] Invoking downloader on 'https://vas-v4.p7s1video.net/4.0/playlist.mpd?x=01&y=V8haLzOXLT35zd2OsFzmhX58AgZWiQXaeuA1JaEimezUU0BoL8sDkSCXru5b9feBXx-ptOzLbI9803DfLCtPpaUNezKp1SCO3-F2AqdjB5tddHYST4wmTWg00IRUJSDIwlMCryAGx-Rhe904kuiioWm2bWkxTfj5ZdUh-Pg9-G9-fz__3vW3_nXIzH871tYDjlR00g9--OsK1OKkc-M_YGPFy28Tl2CBaUkD-h9cckiUTWP0W7HrvIUlg0tiG7SCetehtdIHp2OX9B-jpSZlSetP0GwVc1267h40w_EAHL0aD1OsL1yLoYzI8BT1y75AcXc5eNmS7KiROuOEj1X0zpGdQasEtIuwUAL4_4TTU14fR4WwQDgEVBwklK-ZdTId'
[dashsegments] Total fragments: 260
[download] Destination: I Want to Eat Your Pancreas\SNAENA I Want to Eat Your Pancreas-v_cpk1o4o25imp.fdash-f9-v1-x3.mp4
[download] 100% of 665.76MiB in 02:23
[debug] Invoking downloader on 'https://vas-v4.p7s1video.net/4.0/playlist.mpd?x=01&y=V8haLzOXLT35zd2OsFzmhX58AgZWiQXaeuA1JaEimezUU0BoL8sDkSCXru5b9feBXx-ptOzLbI9803DfLCtPpaUNezKp1SCO3-F2AqdjB5tddHYST4wmTWg00IRUJSDIwlMCryAGx-Rhe904kuiioWm2bWkxTfj5ZdUh-Pg9-G9-fz__3vW3_nXIzH871tYDjlR00g9--OsK1OKkc-M_YGPFy28Tl2CBaUkD-h9cckiUTWP0W7HrvIUlg0tiG7SCetehtdIHp2OX9B-jpSZlSetP0GwVc1267h40w_EAHL0aD1OsL1yLoYzI8BT1y75AcXc5eNmS7KiROuOEj1X0zpGdQasEtIuwUAL4_4TTU14fR4WwQDgEVBwklK-ZdTId'
[dashsegments] Total fragments: 260
[download] Destination: I Want to Eat Your Pancreas\SNAENA I Want to Eat Your Pancreas-v_cpk1o4o25imp.fdash-f3-a1-x3.m4a
[download] 100% of 24.28MiB in 00:38
[ffmpeg] Merging formats into "I Want to Eat Your Pancreas\SNAENA I Want to Eat Your Pancreas-v_cpk1o4o25imp.mp4"
[debug] ffmpeg command line: ffmpeg -y -loglevel "repeat+info" -i "file:I Want to Eat Your Pancreas\SNAENA I Want to Eat Your Pancreas-v_cpk1o4o25imp.fdash-f9-v1-x3.mp4" -i "file:I Want to Eat Your Pancreas\SNAENA I Want to Eat Your Pancreas-v_cpk1o4o25imp.fdash-f3-a1-x3.m4a" -c copy -map "0:v:0" -map "1:a:0" "file:I Want to Eat Your Pancreas\SNAENA I Want to Eat Your Pancreas-v_cpk1o4o25imp.temp.mp4"
Deleting original file I Want to Eat Your Pancreas\SNAENA I Want to Eat Your Pancreas-v_cpk1o4o25imp.fdash-f9-v1-x3.mp4 (pass -k to keep)
Deleting original file I Want to Eat Your Pancreas\SNAENA I Want to Eat Your Pancreas-v_cpk1o4o25imp.fdash-f3-a1-x3.m4a (pass -k to keep)
[download] Downloading video 2 of 4
[download] I Want to Eat Your Pancreas\SNAENA I Want to Eat Your Pancreas-v_cpk1o4o25imp.mp4 has already been downloaded and merged
[download] Downloading video 3 of 4
[download] I Want to Eat Your Pancreas\SNAENA I Want to Eat Your Pancreas-v_cpk1o4o25imp.mp4 has already been downloaded and merged
[download] Downloading video 4 of 4
[download] I Want to Eat Your Pancreas\SNAENA I Want to Eat Your Pancreas-v_cpk1o4o25imp.mp4 has already been downloaded and merged
[download] Finished downloading playlist: I Want to Eat Your Pancreas
Simply that the actual concatenation isn't implemented in yt-dl: it is in yt-dlp. I would expect a multi_video
playlist to be treated just like an ordinary playlist in yt-dl at present.
Is any metadata offered by the site to specify the mapping between programme timecode and video-part timecode?
Please feel free to make a PR when you're ready.
I just looked into the code for downloading multi_video
entries. It's exactly the same as for playlists so the supposed automated concatenation won't happen.
Now I'll take a look into post processing options.
If somebody's got some idea how to get rid of the time overlap, please tell me. ;)
@Shithappenz add %(playlist_index)d
somewhere in your output template to download all parts at once. (I just tested it for myself.) There's still time overlap in the videos, though.
Is any metadata offered by the site to specify the mapping between programme timecode and video-part timecode?
We get the "subclip_start"
and "subclip_end"
tags for each clip/part of the video as seen in https://github.com/ytdl-org/youtube-dl/issues/30598#issuecomment-1108865434
@Shithappenz add %(playlist_index)d somewhere in your output template to download all parts at once. (I just tested it for myself.) There's still time overlap in the videos, though.
Thank you for your input! Sorry! I don't have so much programming experience. I can only help to find errors and test the code...
Ah. Well, we ought to implement the concatenation function and then the clip timing data can be passed to ffmpeg. In due course.
If the parts are just clipped, at an I-frame boundary, out of an original single encode, just trimming the last part of part n-1 and concatenating part n to it should be fine. If the I-frame timing is different in each part, one I-frame group at each boundary might have to be re-encoded, so that would be a problem: currently we generally do no video re-encoding, despite the name of the --recode-video
(should be --remux-video
).
I did a few experiments with ffmpeg and apparently movies aren't cut at keyframes (which according to LosslessCut are every 2 seconds). So there's no easy way without reencoding.
I'll look into adding the playlist-index to the video-id for movies so we don't need that workaround with the output filename anymore anymore since single parts all have the exact same id.
The concatenating won't be part of youtube-dl since it's not as easy and needs a solution with other tools anyway.
I did a few experiments with ffmpeg and apparently movies aren't cut at keyframes (which according to LosslessCut are every 2 seconds). So there's no easy way without reencoding.
Scrap that. Using some better tool (avidemux) I figured out the movies are properly cut at keyframes (with 10 second overlap - but I wouldn't count on that to be always like that). The times might differ a bit from what we get, though. I'm still trying to figure that out properly, since there seem to be differences between different information sources, although I can definitely say it always cuts at 2 second intervals and values like 1542.5 appear to be always rounded up to the next 2-second-interval (1544 in this case).
Theoretically it should be possible to automatically merge the downloaded files including proper cutting of the video files.
Hey @Yavos, with your current dev branch I get the following error:
python37 __main__.py -r 2M -o "%(series)s\S%(season_number)02dE%(episode_number)02d %(title)s-%(id)s.%(ext)s" "https://www.prosieben.de/tv/das-duell-um-die-welt-joko-gegen-klaas/videos/10-jahre-duell-um-die-welt-joko-und-klaas-blicken-zurueck-ganze-folge"
[prosiebensat1] tv/das-duell-um-die-welt-joko-gegen-klaas/videos/10-jahre-duell-um-die-welt-joko-und-klaas-blicken-zurueck-ganze-folge: Downloading webpage
[info] >>> extracting information for cutting from site
[info] [video segment 1] duration: 2177.8, start: 0, end: 2177.8
[info] [video segment 2] duration: 2187.8, start: 2167.8, end: 4355.6
[info] video segment 2 is overlapping for 10.0 seconds with previous segment.
[info] [video segment 3] duration: 2187.8, start: 4345.6, end: 6533.4
[info] video segment 3 is overlapping for 10.0 seconds with previous segment.
[info] [video segment 4] duration: 2187.8, start: 6523.4, end: 8711.2
[info] video segment 4 is overlapping for 10.0 seconds with previous segment.
[info] [video segment 5] duration: 2187.8, start: 8701.2, end: 10889
[info] video segment 5 is overlapping for 10.0 seconds with previous segment.
[prosiebensat1] v_rq6ejgftfjrx: Downloading videos JSON
Traceback (most recent call last):
File "__main__.py", line 19, in <module>
youtube_dl.main()
File "F:\temp data\__YT_DLP\youtube-dl-1\youtube_dl\__init__.py", line 475, in main
_real_main(argv)
File "F:\temp data\__YT_DLP\youtube-dl-1\youtube_dl\__init__.py", line 465, in _real_main
retcode = ydl.download(all_urls)
File "F:\temp data\__YT_DLP\youtube-dl-1\youtube_dl\YoutubeDL.py", line 2078, in download
url, force_generic_extractor=self.params.get('force_generic_extractor', False))
File "F:\temp data\__YT_DLP\youtube-dl-1\youtube_dl\YoutubeDL.py", line 808, in extract_info
return self.__extract_info(url, ie, download, extra_info, process)
File "F:\temp data\__YT_DLP\youtube-dl-1\youtube_dl\YoutubeDL.py", line 815, in wrapper
return func(self, *args, **kwargs)
File "F:\temp data\__YT_DLP\youtube-dl-1\youtube_dl\YoutubeDL.py", line 836, in __extract_info
ie_result = ie.extract(url)
File "F:\temp data\__YT_DLP\youtube-dl-1\youtube_dl\extractor\common.py", line 534, in extract
ie_result = self._real_extract(url)
File "F:\temp data\__YT_DLP\youtube-dl-1\youtube_dl\extractor\prosiebensat1.py", line 700, in _real_extract
return self._extract_clip(url, webpage)
File "F:\temp data\__YT_DLP\youtube-dl-1\youtube_dl\extractor\prosiebensat1.py", line 538, in _extract_clip
for video in videos_data]
File "F:\temp data\__YT_DLP\youtube-dl-1\youtube_dl\extractor\prosiebensat1.py", line 538, in <listcomp>
for video in videos_data]
AttributeError: 'str' object has no attribute 'get'
@Yavos any news on your work on this issue? Thanks!
I didn't really have motivation to look into it since the relevant part for me (downloading anime) works. A quick check back then with the unmodified plugin gave me a drm error.
In all current versions I get either a json_data error or a extract clip_id error. I'll look into it if I find time. But it seems something changed at prosieben. The urls don't match current videos anymore either.
@pannal I tried looking into it but didn't succeed. "Joko gegen Klaas" seems to be drm protected anyway. So even if I got it to work, you apparently can't use it.
In general my request (tried with S5E1) fails with the following response:
{'meta': {'countrycode_platform': 'de', 'countrycode_resolved': 'de', 'netspeed_resolved': 'cable-dsl', 'lifecycle_id': 'ARdOMDO1eLlFNYhE'}, 'data': {'v_e8kgje877njt': {'error': {'code': 2032, 'title': 'no matching visibility rule'}}}}
The last part should not contain an error. I tried tinkering with the payload but couldn't figure it out. Maybe it's a different key used here. Maybe it's something about login/cookies. Don't get your hopes up to get a solution.
Checklist
Verbose log
Description
I founded two bugs / errors on prosiebenmaxx.de: There is a Error 404 and a mafunction in the drm-protection-check. I posted both logs above. There are some clips, where I get the 404 Error (e.g. the link above). I tryed to download it serveral times, but youtube-dl answered the same 404 Error. Maybe in the extractor there are some lines, that are not working anymore? Futhermore the drm-protection check is not working. For example youtube-dl answered, that the clip is drm protected. In the header of the side, one can read, that there is no drm protection: meta data-react-helmet="true" property="video:drm" content="0" I downloded the clip manually and vlc can play that clip. I am sorry for my english. I know it is not quiet good, but I hope, I posted a helpful log.