Open betterthanever2 opened 1 year ago
There is no specific support for smotrim.ru in either the released yt-dl or the latest git master. Some time ago we discovered that the rutube.ru extractor would also handle the site.
Maybe you are using yt-dlp, which does have an extractor that now times out? See #30839.
smotrim.ru is handled by the RUTV
extractor. A few months back I submitted an issue here about it not being supported explicitly, and eventually a patch was suggested, and it still works for me.
I changed the extractor script to handle the above issue as well, so downloads work fine for me right now. I just thought, I'd let you know about this change.
I changed the extractor script to handle the above issue as well
... Perhaps, then, you'd be kind and willing 😉 to share this newest patch of yours here or, even kinder, create a PR with the necessary changes to rutv.py
that restore smotrim.ru
support in youtube-dl
?
Thanks 😃
Sure, if you're using a patched version, that's understandable, but you could have noted that against "verified that I'm running youtube-dl version 2021.12.17".
Perhaps submit your additional patch so that we can roll it in due course, or into yt-dlp?
OP is located inside Ukraine...
Living in the EU, I find I'm unable to access:
https://smotrim.ru/video/2568723
[Unable to connect
error in my browser]
so I assume this applies...
However,
http://player.rutv.ru/iframe/datavideo/id/2568723
DOES load OK here (the opposite to OP), while
https://player.smotrim.ru/iframe/video/id/2568723
and
https://player.smotrim.ru/iframe/datavideo/id/2568924/sid/smotrim
DO NOT (to be expected, due to EU-wide block of smotrium.ru
)
OTOH, When I VPN to Russia, https://smotrim.ru/video/2568723 DOES load now, as well as https://player.smotrim.ru/iframe/video/id/2568723 and https://player.smotrim.ru/iframe/datavideo/id/2568924/sid/smotrim but http://player.rutv.ru/iframe/datavideo/id/2568723 DOES NOT (it times out, consistent with OP's report) ...
At the current state of world affairs, it seems any eventual support for smotrim.ru
(via rutv.py
) would be IP-dependent... 😞
I changed the extractor script to handle the above issue as well
... Perhaps, then, you'd be kind and willing 😉 to share this newest patch of yours here or, even kinder, create a PR with the necessary changes to
rutv.py
that restoresmotrim.ru
support inyoutube-dl
?Thanks 😃
The patch is replacing line 155 'http://player.rutv.ru/iframe/data%s/id/%s' % ('live' if is_live else 'video', video_id)
with f'http://player.smotrim.ru/iframe/data{"live" if is_live else "video"}/id/{video_id}/sid/smotrim'
. I am wary about submitting this as a PR, because I'm pretty sure this won't work universally.
OP is located inside the Ukraine... not that big of a deal, but Ukraine goes without an article.
The patch is replacing line 155
Which "line 155" ?
If one patches rutv.py (current state in git master
) according to this patch by dirkf, one can't find L155
with cited content 😞 ... There's now L160
that appears to be relevant:
159 json_data = self._download_json(
160 '%s/iframe/data%s/id/%s' % (player, 'live' if is_live else 'video', video_id),
161 video_id, 'Downloading JSON')
so, perhaps, this one needs to be patched? IOW, what's the file you applied your posted patch on?
not that big of a deal, but Ukraine goes without an article.
Noted and corrected 😉 ; I must have been (subconsciously) carried away by
... inside the UK
,
that I often write myself :smile: ...
Regards.
The patch is replacing line 155
so, perhaps, this one needs to be patched? IOW, what's the file you applied your posted patch on?
Yes, the line you reference is the one. The patch by dirkf is the one I applied, I may have done that manually, and that may have resulted in a different number of lines in the file. Quite honestly, I don't remember, I just made it work and moved on.
... Right... It was a case of finding a working and whitelisted RU HTTPS proxy 😉 ...
As I posted above, I simply patched the git master
edition of rutv.py
with the linked patch by dirkf - I did NOT patch anything else in RUTVIE
...
yt-dl --proxy "localhost:8080" -vF "https://smotrim.ru/video/2568723" =>
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--ffmpeg-location', '.\\FFmpeg', '--external-downloader-args', '-v 8 -stats', '--proxy', 'localhost:8080', '-vF', 'https://smotrim.ru/video/2568723']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.02.23.114514
[debug] Python version 3.4.4 (CPython) - Windows-Vista-6.0.6003-SP2
[debug] exe versions: ffmpeg n5.2-dev-2245-N-109649-gab8cde6, ffprobe n5.2-dev-2245-N-109649-gab8cde6, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {'https': 'localhost:8080', 'http': 'localhost:8080'}
[RUTV] 2568723: Downloading JSON
[RUTV] 2568723: Downloading m3u8 information
[info] Available formats for 2568723:
format code extension resolution note
hls-400 mp4 unknown 400k
hls-800 mp4 unknown 800k
hls-1200 mp4 unknown 1200k
hls-1800 mp4 unknown 1800k
hls-4050 mp4 unknown 4050k
http-1080 mp4 1920x1080
http-234 mp4 1920x1080
http-360 mp4 1920x1080
http-540 mp4 1920x1080
http-720 mp4 1920x1080 (best)
The extractor has to be amended, so that resolutions are:
a) calculated for the HLS formats (currently unknown
)
b) corrected for the HTTP formats (all showing as 1920x1080
;hint: width
is derived from the format code, e.g. http-360
=> 640x360
)
Additionally, the (best) label should be assigned to http-1080
rather than to http-720
; and, though advertised as available, format http-234
always returns a 404
...
The media files AREN'T BLOCKED per se, at least not here...
yt-dl --proxy "localhost:8080" -f http-360 "https://smotrim.ru/video/2568723" -g =>
https://cdn-v.rtr-vesti.ru/_cdn_auth/secure/v/vh/mp4/medium-wide/002/740/306.mp4?auth=mh&vid=2740306
and then one can perform a DIRECT download:
yt-dl "https://cdn-v.rtr-vesti.ru/_cdn_auth/secure/v/vh/mp4/medium-wide/002/740/306.mp4?auth=mh&vid=2740306" -o "Кто против Одно из самых масштабных посланий президента Федеральному Собранию. Эфир от 21.02.2023-2568723.mp4" =>
[generic] 306: Requesting header
[download] Destination: Кто против Одно из самых масштабных посланий президента Федеральному Собранию. Эфир от 21.02.2023-2568723.mp4
[download] 100% of 271.86MiB in 07:39
If the HLS media URLs are like the mp4 ones, a descriptive resolution can be extracted from the URL itself.
OT: historically the word "Ukraine", which is etymologically equivalent to Borders or Marches, was used in English for the region of the Russian Tsarist/Soviet empire where the modern independent country Ukraine is situated; that is the significance of "the Ukraine". A place might be inside the RF and not in Ukraine, but still be in "the Ukraine".
A place might be inside the RF and not in Ukraine, but still be in "the Ukraine".
Sorry, what?
Anyway, word "Ukraine" has an etymology, that is true, the same goes for almost any country name. What you mentioned about "borders or marches" is one hypothesis in this case (not an established fact, mind you), but more importantly, contemporary usage of the words is very rarely tied to their etymology. The world would be a weird place, if it was. So why don't we just agree to call the countries whatever their respective people have implicitly agreed upon, and leave it at that?
This is hardly a place for such discussions.
This is hardly a place for such discussions.
I agree 👍 ; be that as it may, "I" did not "invent" the "term" "the Ukraine" out of the blue, nor did I use it consciously to denote something demeaning about Ukraine's state of independence and/or citizens 😉 ...
After these two latest comments, I "searched" and Wikipedia (despite, at times, not being 100% credible) has an entry just for that, under "Etymology and orthography":
In the English-speaking world during most of the 20th century, Ukraine (whether independent or not) was referred to as "the Ukraine". This is because the word ukraina means "borderland", so the definite article would be natural in the English language; this is similar to "Nederlanden", which means "low lands" and is rendered in English as "the Netherlands".
However, since Ukraine's declaration of independence in 1991, this usage has become politicised and is now rarer, and style guides advise against its use. US ambassador William Taylor said that using "the Ukraine" implies disregard for Ukrainian sovereignty. The official Ukrainian position is that "the Ukraine" is both grammatically and politically incorrect.
Thus, I now understand why someone inside Ukraine today would feel "sensitive" about such an inclusion of the "definite article" (be it purely inadvertent 😄 ) ...
That is, the "modern independent country whose name is Ukraine", glory to it.
The established usage and distinction in English that I described (more accurately than Wikipedia IMO) isn't a hypothesis. The use of "the Ukraine" is becoming rarer, if only and horrifically because the invasion has led English speakers to focus on the nation rather than the general area. Use of "the" doesn't apply in the main languages of the region, so the distinction might appear strange. An answer to "what?" could be Belgorod which has fallen under both Ukrainian and Russian sovereignty or administration within a few generations.
Although historical usage had "the Sudan" and "the Yemen", "the" is not used with a country name unless the actual name is some sort of phrase: the United States of America, the People's Republic of China, even the Nether-lands (nether being a quaint synonym for lower, as in be-neath); but actually we call it Holland, as an American might say England for Britain.
no
The established usage and distinction in English that I described (more accurately than Wikipedia IMO) isn't a hypothesis.
Well, it is a hypothesis, albeit a commonly accepted one. I just don't think, there's such thing as a 100% fact with these things. It's not a matter of record, after all.
The use of "the Ukraine" is becoming rarer, if only and horrifically because the invasion has led English speakers to focus on the nation rather than the general area.
Quite honestly, I have never heard anybody referring to the 'general area' that way, not in English, not in Russian, and not in Ukrainian, so this "becoming rarer" feels... 🤨 How old are you, exactly? 😄
From the UK, all the variants player.rutv.ru, together with player.smotrim.ru and player.vgtrk.com (same IP), resolve but respond to pings only occasionally, if at all, at least until the forthcoming Untergang.
The patched code can be modified at the 4th line of the _real_extract()
method:
player = 'http://player.smotrim.ru'
Possibly --geo-verification-proxy ...
works?
Possibly
--geo-verification-proxy
... works?
It doesn't:
yt-dl -vF --geo-verification-proxy "localhost:8080" "https://smotrim.ru/video/2568723" =>
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--ffmpeg-location', '.\\FFmpeg', '--external-downloader-args', '-v 8 -stats', '-vF', '--geo-verification-proxy', 'localhost:8080','https://smotrim.ru/video/2568723']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.02.25.334
[debug] Python version 3.4.4 (CPython) - Windows-Vista-6.0.6003-SP2
[debug] exe versions: ffmpeg n5.2-dev-2245-N-109649-gab8cde6, ffprobe n5.2-dev-2245-N-109649-gab8cde6, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {}
[RUTV] 2568723: Downloading JSON
ERROR: Unable to download JSON metadata: <urlopen error [WinError 10061] No connection could be made because the target machine actively refused it> (caused by URLError(ConnectionRefusedError(10061, 'No connection could be made because the target machine actively refused it', None, 10061, None),))
File "common.py", line 635, in _request_webpage
File "YoutubeDL.py", line 2300, in urlopen
File "C:\hostedtoolcache\windows\Python\3.4.4\x86\lib\urllib\request.py", line 464, in open
File "C:\hostedtoolcache\windows\Python\3.4.4\x86\lib\urllib\request.py", line 482, in _open
File "C:\hostedtoolcache\windows\Python\3.4.4\x86\lib\urllib\request.py", line 442, in _call_chain
File "D:\a\youtube-dl\youtube-dl\youtube_dl\utils.py", line 2634, in http_open
File "C:\hostedtoolcache\windows\Python\3.4.4\x86\lib\urllib\request.py", line 1185, in do_open
... and the reason it doesn't work I have explained in the past (in the context of a French TV InfoExtractor I'm now lazy to dig up): the InfoExtractor it's used with has to explicitly support that switch... Supporting it inside the IE means passing once (or multiple times), in the "right" place(s), an additional request header in the form of
headers=self.geo_verification_headers()
E.g. theplatformIE
, a "framework" used in many other (mainly American) IEs, has this part of code:
--geo-verification-proxy
is particular to IEs where:
a) the playlist/manifest generating API is geo-blocked and can't be fooled by an x-Forwarded-For
header (this includes the --geo-bypass
and --geo-bypass-country
switches)
b) the stream CDN(s) do not geo-block.
In this case, the webpage itself is being blocked for me, so I expect the "extra" code has to be applied in the webpage fetch itself, not just for the "manifest" API... 😞
OK, based on my analysis above, and a lot of trial-and-error, I concocted a version of rutv.py
that works for me with the --geo-verification-proxy
switch and my HTTPS-only Russian proxy:
142 def _real_extract(self, url):
143 mobj = re.match(self._VALID_URL, url)
144 video_id = mobj.group('id')
145 video_path = mobj.group('path')
146- player = 'http://player.smotrim.ru'
146+ player = 'https://player.smotrim.ru'
147
148 if video_path.startswith('iframe'):
149 video_type = mobj.group('type')
150 if video_type == 'swf':
151 video_type = 'video'
152 elif video_path.startswith('index/iframe/cast_id'):
153 video_type = 'live'
154 else:
155 video_type = 'video'
156
157 is_live = video_type == 'live'
158
159 json_data = self._download_json(
160 '%s/iframe/data%s/id/%s' % (player, 'live' if is_live else 'video', video_id),
161- video_id, 'Downloading JSON')
161+ video_id, 'Downloading JSON',
162+ headers=self.geo_verification_headers())
I had to explicitly set the HTTPS version of the player, else my proxy couldn't handle the redirection from HTTP to HTTPS (???):
yt-dl -vF --geo-verification-proxy "https://localhost:8080" "https://smotrim.ru/video/2568723" =>
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--ffmpeg-location', '.\\FFmpeg', '--external-downloader-args', '-v 8 -stats', '-vF', '--geo-verification-proxy', 'https://localhost:8080', 'https://smotrim.ru/video/2568723']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.02.25.334
[debug] Python version 3.4.4 (CPython) - Windows-Vista-6.0.6003-SP2
[debug] exe versions: ffmpeg n5.2-dev-2245-N-109649-gab8cde6, ffprobe n5.2-dev-2245-N-109649-gab8cde6, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {}
[RUTV] 2568723: Downloading JSON
[RUTV] 2568723: Downloading m3u8 information
[info] Available formats for 2568723:
format code extension resolution note
hls-400 mp4 unknown 400k
hls-800 mp4 unknown 800k
hls-1200 mp4 unknown 1200k
hls-1800 mp4 unknown 1800k
hls-4050 mp4 unknown 4050k
http-1080 mp4 1920x1080
http-234 mp4 1920x1080
http-360 mp4 1920x1080
http-540 mp4 1920x1080
http-720 mp4 1920x1080 (best)
and then:
yt-dl --geo-verification-proxy "https://localhost:8080" -f http-540 "https://smotrim.ru/video/2568723" =>
[RUTV] 2568723: Downloading JSON
[RUTV] 2568723: Downloading m3u8 information
[download] Destination: Кто против Одно из самых масштабных посланий президента Федеральному Собранию. Эфир от 21.02.2023-2568723.mp4
[download] 4.7% of 528.15MiB at 693.60KiB/s ETA 12:22
NB that a dl speed of ca. 700KiB/s is due to the DIRECT connection 😜 ...
Checklist
Description
Right now, trying to download a video from an address like
https://smotrim.ru/video/2568723
results in a timeout error due to failure to resolve theplayer
address. This is likely because resolved video address used to have this format:http://player.rutv.ru/iframe/datavideo/id/2568723
, but seems to have been changed recently tohttps://player.smotrim.ru/iframe/video/id/2568723
.Correction: the issue is actually with the JSON data file URL, which is now
https://player.smotrim.ru/iframe/datavideo/id/2568924/sid/smotrim
, i.e. featuressmotrim.ru
instead ofrutv.ru
, and has the/sid/smotrim
part at the end.