search

spaam / svtplay-dl

Small command-line program to download videos from some streaming sites.
https://svtplay-dl.se
MIT License
714 stars 118 forks source link

Download hangs on svtplay.se #1305

Open 6b6561 opened 3 years ago

6b6561 commented 3 years ago

Versions:

Issue: Download of movies from svtplay.se end's up hanging with no progress after downloading most of the movie, same behavior seen with numerous movies. Basically it look's like the script is waiting forever for a response to the last get.

A wget test for https://svt-vod-5a.akamaized.net:443/d0/se/20201205/896a09fb-7ad9-4c04-b641-8e328245c42a/cmaf-video-avc-4/cmaf-video-avc-4-1683.mp4 succeded so the problem isn't on the server nor network side.

Command run: ./svtplay-dl -v -S https://www.svtplay.se/video/24912518/pulp-fiction

DEBUG [1610114835.286998] /usr/lib/python3/dist-packages/urllib3/connectionpool.py/_make_request: https://svt-vod-5a.akamaized.net:443 "GET /d0/se/20201205/896a09fb-7ad9-4c04-b641-8e328245c42a/cmaf-video-avc-4/cmaf-video-avc-4-1680.mp4 HTTP/1.1" 200 2210959
[1682/2316][============================================================================================================.........................................] ETA: 0:14:17DEBUG [1610114837.1990762] ./svtplay-dl/svtplay_dl/utils/http.py/request: HTTP getting 'https://svt-vod-5a.akamaized.net/d0/se/20201205/896a09fb-7ad9-4c04-b641-8e328245c42a/cmaf-video-avc-4/cmaf-video-avc-4-1681.mp4'
DEBUG [1610114837.7429664] /usr/lib/python3/dist-packages/urllib3/connectionpool.py/_make_request: https://svt-vod-5a.akamaized.net:443 "GET /d0/se/20201205/896a09fb-7ad9-4c04-b641-8e328245c42a/cmaf-video-avc-4/cmaf-video-avc-4-1681.mp4 HTTP/1.1" 200 1811772
[1683/2316][============================================================================================================.........................................] ETA: 0:14:16DEBUG [1610114839.07769] ./svtplay-dl/svtplay_dl/utils/http.py/request: HTTP getting 'https://svt-vod-5a.akamaized.net/d0/se/20201205/896a09fb-7ad9-4c04-b641-8e328245c42a/cmaf-video-avc-4/cmaf-video-avc-4-1682.mp4'
DEBUG [1610114839.178305] /usr/lib/python3/dist-packages/urllib3/connectionpool.py/_make_request: https://svt-vod-5a.akamaized.net:443 "GET /d0/se/20201205/896a09fb-7ad9-4c04-b641-8e328245c42a/cmaf-video-avc-4/cmaf-video-avc-4-1682.mp4 HTTP/1.1" 200 1227132
[1684/2316][============================================================================================================.........................................] ETA: 0:14:15DEBUG [1610114840.2337902] ./svtplay-dl/svtplay_dl/utils/http.py/request: HTTP getting 'https://svt-vod-5a.akamaized.net/d0/se/20201205/896a09fb-7ad9-4c04-b641-8e328245c42a/cmaf-video-avc-4/cmaf-video-avc-4-1683.mp4'
DEBUG [1610114840.310824] /usr/lib/python3/dist-packages/urllib3/connectionpool.py/_make_request: https://svt-vod-5a.akamaized.net:443 "GET /d0/se/20201205/896a09fb-7ad9-4c04-b641-8e328245c42a/cmaf-video-avc-4/cmaf-video-avc-4-1683.mp4 HTTP/1.1" 200 1430964
pythonuser3856 commented 3 years ago

When using svtplay-dl in a different configuration but still the python code variant, I'm able to successfully download:

svtplay-dl_sshot_20201231_141922 --subtitle --format-preferred hevc-51 https://www.svtplay.se/video/24912518/pulp-fiction

INFO: Selected to download dash, bitrate: 6263 format: hevc-51 INFO: Outfile: pulp-fiction-e4db251-svtplay.m4a [2316/2316][========================================] ETA: 0:00:00 INFO: Outfile: pulp-fiction-e4db251-svtplay.mp4 [2316/2316][========================================] ETA: 0:00:00 INFO: Merge audio and video into pulp-fiction-e4db251-svtplay.mp4 INFO: Merging done, removing old files.

6b6561 commented 3 years ago

Thanks, I will test that.

I did try DASH and HLS and had the same freeze on both. I have also tried to switch between to different 4G providers on different 4G routers and same result on both.

Unfortunately I'm starting to fear that my issue might be connectivity related, as I every once and now see a similar thing casting to my chromecast, ~30 minutes into the program I get a freeze, and I'm forced to stop the cast and resume from the point where it freezes.

I haven't looked at your code, but do you have any timeout/retry implemented on the gets? I guess it would help in issues like this if you don't.

pythonuser3856 commented 3 years ago

I'm not a developer but rather a user of this application so I'm unfortunately not able to comment your question.

One observation though; You say that you make use of 4G and by that you are sharing bandwidth with other users of 4G. In addition, if a 4G connection is dropped it may become restarted in a not so seemless manner and by that cause a communication break. With small chunks of data such disturbances are not really noticed but if the download is time consuming, then you're in a snag. If you don't have an option to use a leased line connection, and, if the root cause to your issue turns out to be the inherent qualities of a 4G connection (or rather any connection that by its nature is unreliable), one option would be to make svtplay-dl to support functionality similar to that of the old ZMODEM transfer protocol, i.e. to be able to resume a previous but interrupted download. To be able to let svtplay-dl make use of previously incomplete downloaded data and start from there. But that would be an svtplay-dl feature request and not an issue report. (see Wikipedia for info about ZMODEM)

6b6561 commented 3 years ago

Same issue, ends up hanging forever on one segment.

If I remember correctly svtplay-dl used to download all the segments one by one and at the end concatenate all of the segments into a single file, or it might have been yle-dl the software to download finnish public broadcast companies videos.

The general recommendation for request is to use timeout and retry, but then you can't directly append the segment to the already downloaded segments, best would be a temp file that is appended once the segment has been downloaded.

It's a quite philosophical question if it's an issue or feature. The issue is that download's hangs but the question is then if the root cause is how it's coded or should be seen as an enhancement.

Basically I think that issues like this could be avoided with a timeout on the segment and a retry.

pythonuser3856 commented 3 years ago

I'm using a connection with leased line optical fibre and have not at any time experienced the type of hanging that you describe. Based on that it would be interesting to get more details of the event of hanging and what's changed in the connection when you retry.

6b6561 commented 3 years ago

The 4G connection stays up all the time, so no problem there and the IP remains unchanged.

I just finished one more test where I ran svtplay-dl -P hls ... to get URL of the akamai files, then ran a wget loop to get all the .ts segments.

What I noted was that I got a couple of segments "hanging" in connected state, and once the wget read-timeout timer expired wget did gracefully a retry and got the file.

--2021-01-09 17:07:21--  https://svt-vod-5a.akamaized.net/d0/se/20201205/896a09fb-7ad9-4c04-b641-8e328245c42a/hls-video-avc-4/hls-video-avc-4-1816.ts
Resolving svt-vod-5a.akamaized.net (svt-vod-5a.akamaized.net)... 79.136.65.143, 79.136.65.144, 79.136.65.136, ...
Connecting to svt-vod-5a.akamaized.net (svt-vod-5a.akamaized.net)|79.136.65.143|:443... connected.

So my current workaround is to:

  1. Run svtplay-dl -P hls ... to get URL of the akamai files
  2. Run a wget loop to get all the individual .ts files
  3. Run a loop to concatenate all the .ts files into a single file
  4. Run ffmpeg to combine the audio and video .ts files into a single file
SYSophie commented 3 years ago

Versions:

  • svtplay-dl: 2.8, installed by "curl"
  • OS: Debian Linux 4.9.0-14-amd64 #1 SMP Debian 4.9.246-2 (2020-12-17) x86_64
  • Python: Python 3.5.3
  • pyyaml, cryptography, pyyaml & pysocks upgraded by pip3 install --upgrade.

Issue: Download of movies from svtplay.se end's up hanging with no progress after downloading most of the movie, same behavior seen with numerous movies. Basically it look's like the script is waiting forever for a response to the last get.

A wget test for https://svt-vod-5a.akamaized.net:443/d0/se/20201205/896a09fb-7ad9-4c04-b641-8e328245c42a/cmaf-video-avc-4/cmaf-video-avc-4-1683.mp4 succeded so the problem isn't on the server nor network side.

Command run: ./svtplay-dl -v -S https://www.svtplay.se/video/24912518/pulp-fiction

DEBUG [1610114835.286998] /usr/lib/python3/dist-packages/urllib3/connectionpool.py/_make_request: https://svt-vod-5a.akamaized.net:443 "GET /d0/se/20201205/896a09fb-7ad9-4c04-b641-8e328245c42a/cmaf-video-avc-4/cmaf-video-avc-4-1680.mp4 HTTP/1.1" 200 2210959
[1682/2316][============================================================================================================.........................................] ETA: 0:14:17DEBUG [1610114837.1990762] ./svtplay-dl/svtplay_dl/utils/http.py/request: HTTP getting 'https://svt-vod-5a.akamaized.net/d0/se/20201205/896a09fb-7ad9-4c04-b641-8e328245c42a/cmaf-video-avc-4/cmaf-video-avc-4-1681.mp4'
DEBUG [1610114837.7429664] /usr/lib/python3/dist-packages/urllib3/connectionpool.py/_make_request: https://svt-vod-5a.akamaized.net:443 "GET /d0/se/20201205/896a09fb-7ad9-4c04-b641-8e328245c42a/cmaf-video-avc-4/cmaf-video-avc-4-1681.mp4 HTTP/1.1" 200 1811772
[1683/2316][============================================================================================================.........................................] ETA: 0:14:16DEBUG [1610114839.07769] ./svtplay-dl/svtplay_dl/utils/http.py/request: HTTP getting 'https://svt-vod-5a.akamaized.net/d0/se/20201205/896a09fb-7ad9-4c04-b641-8e328245c42a/cmaf-video-avc-4/cmaf-video-avc-4-1682.mp4'
DEBUG [1610114839.178305] /usr/lib/python3/dist-packages/urllib3/connectionpool.py/_make_request: https://svt-vod-5a.akamaized.net:443 "GET /d0/se/20201205/896a09fb-7ad9-4c04-b641-8e328245c42a/cmaf-video-avc-4/cmaf-video-avc-4-1682.mp4 HTTP/1.1" 200 1227132
[1684/2316][============================================================================================================.........................................] ETA: 0:14:15DEBUG [1610114840.2337902] ./svtplay-dl/svtplay_dl/utils/http.py/request: HTTP getting 'https://svt-vod-5a.akamaized.net/d0/se/20201205/896a09fb-7ad9-4c04-b641-8e328245c42a/cmaf-video-avc-4/cmaf-video-avc-4-1683.mp4'
DEBUG [1610114840.310824] /usr/lib/python3/dist-packages/urllib3/connectionpool.py/_make_request: https://svt-vod-5a.akamaized.net:443 "GET /d0/se/20201205/896a09fb-7ad9-4c04-b641-8e328245c42a/cmaf-video-avc-4/cmaf-video-avc-4-1683.mp4 HTTP/1.1" 200 1430964

@6b6561 yt-dl supports resume download (and svtplay), maybe this is something that you would benefit from using when using a cellular connection?

6b6561 commented 3 years ago

Thanks, but the -r resume of svtplay-dl is a old legacy thing for RTMP streams so this will not help in this case.

spaam commented 3 years ago

this seems a bit weird, because we added stuff that should timeout after X seconds and retry again 🤔

yeah. -r was for old RTMP streams. i could fix the resuming thing. but i have not decided how to do it. if i want to have the files separately then joining them at the end to a big file. it would be an easy way to do it but it will eat more space for people. another way to have to to only one file then try to figure out where it failed. its a bit harder.. 🤔

6b6561 commented 3 years ago

I must start by apologizing to you spaam, normally I don't cut corners or make lazy assumptions when I deal with developers. The developers time is too valuable too have to deal with crap feedback. I must admit that I where a bit lazy with my wget the other day when I said "and once the wget read-timeout timer expired wget did gracefully a retry and got the file.". Actually I didn't get a timeout and retry at that point... my wget where just hanging untiI I did a ctrl-c. I then read the man page added a timeout and assumed it would behave... wrote the post... and later on I realized that wget didn't timeout...

Here comes a revised entry.

My wget loop runs NOW with a single argument to wget --timeout=60, and re-runing the script now caused wget to fail with "Unable to establish SSL connection." on one segment.

--2021-01-10 22:09:35--  https://svt-vod-7t.akamaized.net/d0/se/20201207/66b10fad-c5f5-4ab0-a4eb-ea95d7a23580/hls-video-avc-4/hls-video-avc-4-213.ts
Resolving svt-vod-7t.akamaized.net (svt-vod-7t.akamaized.net)... 79.136.65.139, 79.136.65.140, 79.136.65.141, ...
Connecting to svt-vod-7t.akamaized.net (svt-vod-7t.akamaized.net)|79.136.65.139|:443... connected.
Unable to establish SSL connection.

Earlier and sub-sequent wget's to the same host succeded, so for some reason one of the wget's failed on SSL connection setup.

I will run more wget loops tomorrow and see if I get more failures due to the same reason.

SYSophie commented 3 years ago

Thanks, but the -r resume of svtplay-dl is a old legacy thing for RTMP streams so this will not help in this case.

Well I was talking about YouTube-dl and not svtplay-dl.

6b6561 commented 3 years ago

I have now run multiple wget --debug download loops, and the wget's always fails in the same way, connection goes into connected state but for some reason it's not able to setup the SSL connection. I guess something similar happens in svtplay-dl.

Converted file name 'cmaf-video-avc-4-291.mp4' (UTF-8) -> 'cmaf-video-avc-4-291.mp4' (UTF-8)
--2021-01-11 12:40:01--  https://svt-vod-6n.akamaized.net/d0/se/20201216/e006ced8-038b-431c-bce4-dcc60f35c35f/cmaf-video-avc-4/cmaf-video-avc-4-291.mp4
Certificates loaded: 126
Resolving svt-vod-6n.akamaized.net (svt-vod-6n.akamaized.net)... 79.136.65.137, 79.136.65.138, 79.136.65.139, ...
Caching svt-vod-6n.akamaized.net => 79.136.65.137 79.136.65.138 79.136.65.139 79.136.65.140 79.136.65.141 79.136.65.142 79.136.65.143 79.136.65.144 79.136.65.136
Connecting to svt-vod-6n.akamaized.net (svt-vod-6n.akamaized.net)|79.136.65.137|:443... connected.
Created socket 3.
Releasing 0x000055a6861c97c0 (new refcount 1).
GnuTLS: The TLS connection was non-properly terminated.
Closed fd 3
Unable to establish SSL connection.
6b6561 commented 3 years ago

Idea about a possible resume mechanism:

Downloading all the segments one by one and keeping them on disc for final assembly is a pain disc wise as you said.

I would approach this with:

On a resume I would continue from the segment in the status file. This would be fairly fail safe, besides a scenario where the commit fails, as then you wouldn't know if/what has been committed. Maybe put the status as "committing segment xxx, not resume-able" in the status file while downloading would have status "Downloading segment xxx".

Edit 1: Look's like this is the approach used in youtube-dl.

SYSophie commented 3 years ago

Idea about a possible resume mechanism:

Downloading all the segments one by one and keeping them on disc for final assembly is a pain disc wise as you said.

I would approach this with:

  • Always download the segments to a temp file, once it's fully downloaded, then I would commit the segment to the large output file.
  • In addition I would have a download-status file, this would be a single liner keeping book on which segment is being downloaded.

On a resume I would continue from the segment in the status file. This would be fairly fail safe, besides a scenario where the commit fails, as then you wouldn't know if/what has been committed. Maybe put the status as "committing segment xxx, not resume-able" in the status file while downloading would have status "Downloading segment xxx".

Edit 1: Look's like this is the approach used in youtube-dl.

🥳

6b6561 commented 3 years ago

Well I was talking about YouTube-dl and not svtplay-dl.

youtube-dl is working quite well for me, but it requires some TCL to keep running... some ctrl-c and some restarts but the good thing is that is able to resume.

So far I have collected two different errors from it, the second looks like what I'm seeing with wget.