Download freezes

[orthica]

I've tried downloading stream: https://areena.yle.fi/1-64233068 several times over the last week or so, but the download freezes around 2h21min or 2h22min. This happened to me with another stream a few months ago, but I can't remember which one. Other streams I have tried, have downloaded without any issues. Any solutions?

[DarrenPIngram]

Just tested it for you, and "alas" it works. So maybe there was a localised disturbance at Areena or your ISP/local network?

yle-dl "https://areena.yle.fi/1-64233068"
yle-dl 20240927: Download media files from Yle Areena and Elävä Arkisto
Copyright (C) 2009-2024 Antti Ajanki <antti.ajanki@iki.fi>, license: GPLv3

mime type is not rfc8216 compliant
[hls @ 0x1291046f0] Can't support the subtitle(uri: hdntl=exp=1730727000~acl=%2fvod%2ffinland%2fhls%2f94-1_ol523w8o%2f,v_1280_720_1500.mp4,v_320_176_120.mp4,v_640_360_400.mp4,v_640_360_900.mp4,v_1280_720_2500.mp4,v_1920_1080_4000.mp4,lang%2fswe%2fs_swe_t.srt,lang%2ffin%2fs_fin_t.srt,.urlset%2f*~data=hdntl~hmac=35054814e489d31817d0ce3b1ea293bf0e3a4d25f81259ceddadffd3ab9d451d/index-f7.m3u8)
[hls @ 0x1291046f0] Can't support the subtitle(uri: hdntl=exp=1730727000~acl=%2fvod%2ffinland%2fhls%2f94-1_ol523w8o%2f,v_1280_720_1500.mp4,v_320_176_120.mp4,v_640_360_400.mp4,v_640_360_900.mp4,v_1280_720_2500.mp4,v_1920_1080_4000.mp4,lang%2fswe%2fs_swe_t.srt,lang%2ffin%2fs_fin_t.srt,.urlset%2f*~data=hdntl~hmac=35054814e489d31817d0ce3b1ea293bf0e3a4d25f81259ceddadffd3ab9d451d/index-f8.m3u8)
Unsupported codec with id 98313 for input stream 2
Unsupported codec with id 98313 for input stream 5
Unsupported codec with id 98313 for input stream 8
Unsupported codec with id 98313 for input stream 11
Unsupported codec with id 98313 for input stream 14
Unsupported codec with id 98313 for input stream 17
Output file: Drive My Car_ 2023-11-15T06_00.mkv
mime type is not rfc8216 compliant
[webvtt @ 0x117ba52d0] Dropping 853 duplicated subtitle events
[webvtt @ 0x117a5fe30] Dropping 720 duplicated subtitle events
frame=257688 fps=2284 q=-1.0 Lsize= 4845555KiB time=02:48:38.93 bitrate=3922.8kbits/s speed=89.7x
Stream saved to Drive My Car_ 2023-11-15T06_00.mkv

(Mac / yle-dl via brew) 4.96Gb filesize. Quick audition seemed ok.

[aajanki]

Are there any errors or does it just hang indefinitely? Are you able to watch movie on web browser beyond the freeze-point?

There have been similar bugs reports before but I have never found a clear explanation.

[orthica]

It doesn't give any errors, just hangs. I've tried downloading it at least a dozen times over the last two weeks or so. Twice I've left it running overnight, but it just gets stuck in some frame and doesn't get past that one. I just tried a few downloads just now and those went through normally, just not this one. I've ran all the updates, tried restarting my computer and my router, but this download always gets stuck very soon after 2h20min. I can watch the film from Areena without any issues. Strange issue.

[DarrenPIngram]

It doesn't give any errors, just hangs. I've tried downloading it at least a dozen times over the last two weeks or so. Twice I've left it running overnight, but it just gets stuck in some frame and doesn't get past that one. I just tried a few downloads

Just thinking in case it helps the developer, maybe mention the machine (ram, proc), OS, version etc. Are you using a VPN by any chance? Plenty of disk space in case it is writing out to process? Sometimes the smallest thing can be overlooked.

Are you running the code native rather than in a sandbox or Docker (if that's available) as in the past I've seen developers chase things down and find that, for whatever reason, it was not liking Docker.

[orthica]

Yes,

Macbook Air M2 2022, 16gb of RAM, MacOS Sequioa 15.0.1, 90+gb of free disk space and not using VPN. I installed yle-dl with homebrew and use it in terminal.

[DarrenPIngram]

Macbook Air M2 2022, 16gb of RAM, MacOS Sequioa 15.0.1, 90+gb of free disk space and not using VPN. I installed yle-dl with homebrew and use it in terminal. You are "well endowed" machine-wise (mine is a MacMini M2 2023 with Apple's generous 8Gb.

So I'm stumped. Could you get any insight at all running the download with --verbose debug output ?

Edit: You are not using cellular data by any chance? Just in case there was some latency weirdness that broke things.

[orthica]

No, not using cellular data.

[orthica]

I did run the --verbose and it gave me this:

Last login: Sun Nov 3 17:46:17 on ttys000

xxx@xxx-MacBook-Air ~ % yle-dl https://areena.yle.fi/1-64233068 --verbose yle-dl 20240927: Download media files from Yle Areena and Elävä Arkisto Copyright (C) 2009-2024 Antti Ajanki antti.ajanki@iki.fi, license: GPLv3

DEBUG: https://areena.yle.fi/1-64233068 is an Areena URL DEBUG: HTTP GET https://areena.yle.fi/1-64233068 DEBUG: HTTP status code: 200 DEBUG: HTTP response headers: DEBUG: Server: CloudFront DEBUG: Content-Type: text/html; charset=utf-8 DEBUG: Transfer-Encoding: chunked DEBUG: Connection: keep-alive DEBUG: Date: Sun, 03 Nov 2024 15:46:23 GMT DEBUG: Content-Security-Policy: default-src 'self' .yle.fi .ylestatic.fi ping.chartbeat.net; script-src 'self' 'unsafe-eval' 'unsafe-inline' .yle.fi .ylestatic.fi .userreport.com .kaltura.com www.gstatic.com; connect-src 'self' .yle.fi wss://.yle.fi .ylestatic.fi .demdex.net endpoint.finnpanel.fi .kaltura.com .akamaized.net .litix.io; style-src 'self' 'unsafe-inline' .yle.fi .ylestatic.fi .googleapis.com; font-src 'self' data: .yle.fi .ylestatic.fi .googleapis.com .gstatic.com; img-src 'self' data: yle.fi .yle.fi .ylestatic.fi yleisradio.d3.sc.omtrdc.net ping.chartbeat.net .userreport.com .kaltura.com; media-src 'self' data: blob: .akamaized.net .kaltura.com .akamaihd.net; frame-src 'self' .userreport.com .yle.fi; worker-src blob:; object-src 'none'; report-uri https://csp.aws.yle.fi/index; report-to https://csp.aws.yle.fi/index DEBUG: Referrer-Policy: strict-origin-when-cross-origin DEBUG: Strict-Transport-Security: max-age=7776000 DEBUG: X-Content-Type-Options: nosniff DEBUG: Cache-Control: max-age=60 DEBUG: X-Powered-By: Next.js DEBUG: Etag: "scxpqgopc1osz" DEBUG: Content-Encoding: gzip DEBUG: Vary: Cookie,Accept-Encoding DEBUG: Via: 1.1 a9717fb92179a05f5da85fabc586e750.cloudfront.net (CloudFront) DEBUG: Alt-Svc: h3=":443"; ma=86400 DEBUG: Age: 18 DEBUG: Set-Cookie: yle_selva=17306488018289389992; Domain=yle.fi; Path=/; Expires=Fri, 02 Mar 2029 15:46:41 GMT DEBUG: X-Cache: Hit from cloudfront DEBUG: X-Amz-Cf-Pop: ZRH55-P1 DEBUG: X-Amz-Cf-Id: 7M1k5u1Elr8JTBeiNcRv3aFfOxL6yDj5mEOCfh84IiE8FMofZhrInA== DEBUG: Not a playlist DEBUG: HTTP GET https://player.api.yle.fi/v1/preview/1-64233068.json?language=fin&ssl=true&countryCode=FI&host=areenaylefi&app_id=player_static_prod&app_key=8930d72170e48303cf5f3867780d549b&isPortabilityRegion=true DEBUG: HTTP status code: 200 DEBUG: HTTP response headers: DEBUG: Content-Type: application/json; charset=utf-8 DEBUG: Transfer-Encoding: chunked DEBUG: Connection: keep-alive DEBUG: Date: Sun, 03 Nov 2024 15:46:23 GMT DEBUG: Server: nginx DEBUG: Cache-Control: max-age=30 DEBUG: Access-Control-Allow-Methods: GET, HEAD, OPTIONS DEBUG: Access-Control-Allow-Origin: https://areena.yle.fi DEBUG: Access-Control-Allow-Credentials: true DEBUG: Strict-Transport-Security: max-age=31536000; includeSubDomains; preload DEBUG: Content-Encoding: gzip DEBUG: Vary: Accept-Encoding DEBUG: X-Cache: Hit from cloudfront DEBUG: Via: 1.1 55dabc4e08599983026bc6c8234017e8.cloudfront.net (CloudFront) DEBUG: X-Amz-Cf-Pop: HEL51-P2 DEBUG: X-Amz-Cf-Id: M6sPKFWOfNE366mzDvpFffp3evI3lx17kTGMJYaXUgoETMYdvrlFcA== DEBUG: Age: 18 DEBUG: preview data: { "meta": { "id": "1-64233068" }, "data": { "ongoing_ondemand": { "description": { "swe": "En skildring om att bli av med f\u00f6rkv\u00e4vt hat och sorg som f\u00e5tt recensioner med fem stj\u00e4rnor. Huvudpersonerna i ber\u00e4ttelsen \u00e4r en sk\u00e5despelare som \u00e4r \u00e4nkling, hans 20-\u00e5riga chauff\u00f6r och en r\u00f6d Saab 900. Handlingen utspelas i Hiroshima. En annorlunda roadmovie som inspirerats av en novell av Haruki Murakami och som vann Oscar f\u00f6r b\u00e4sta internationella l\u00e5ngfilm 2022. Regi: Ryusuke Hamaguchi. I huvudrollerna: Hidetoshi Nishijima, T\u014dko Miura, Masaki Okada. (Doraibu mai k\u0101, Japan 2021)", "fin": "Viiden t\u00e4hden arvioita ker\u00e4nnyt kuvaus tukahdutetun vihan ja surun voittamisesta. Tarinan p\u00e4\u00e4henkil\u00f6it\u00e4 ovat leskeksi j\u00e4\u00e4nyt n\u00e4yttelij\u00e4, t\u00e4m\u00e4n 20-vuotias autonkuljettaja ja punainen Saab 900. Tapahtumapaikka on Hiroshima. Haruki Murakamin novellista innoituksensa saanut erilainen road movie voitti parhaan ulkomaisen elokuvan Oscarin 2022. Ohjaus Ryusuke Hamaguchi, p\u00e4\u00e4osissa Hidetoshi Nishijima, T\u014dko Miura, Masaki Okada. (Doraibu mai k\u0101, Japani 2021)" }, "subtitles": [], "auto_subtitles": true, "media_id": "85-fbc520aa6b394425957d63b135ec321f", "publication_id": "4-66694870", "program_id": "1-64233068", "dvr_window_in_seconds": 0, "cuepoints": [], "start_time": "2023-11-15T06:00:00.000+02:00", "duration": { "duration_in_seconds": 10307, "h": 0, "m": 0, "s": 10307 }, "audio_tracks": [ { "format": "stereo", "type": "AudioTrack", "language": "eng", "kind": "AudioTrack" } ], "is_content_protected": true, "title": { "swe": "Drive My Car", "fin": "Drive My Car" }, "region": "Finland", "content_type": "VideoObject", "is_areena_visible": true, "image": { "id": "13-1-64233068-1697607129475", "version": 1730129776 }, "service_id": "yle-areena", "manifest_url": "https://ylekvodmod01.akamaized.net/vod/finland/hls/94-1_ol523w8o/,v_1280_720_1500.mp4,v_320_176_120.mp4,v_640_360_400.mp4,v_640_360_900.mp4,v_1280_720_2500.mp4,v_1920_1080_4000.mp4,lang/swe/s_swe_t.srt,lang/fin/s_fin_t.srt,.urlset/master.m3u8?hdnts=exp=1730652383~acl=/vod/finland/hls/94-1_ol523w8o/,v_1280_720_1500.mp4,v_320_176_120.mp4,v_640_360_400.mp4,v_640_360_900.mp4,v_1280_720_2500.mp4,v_1920_1080_4000.mp4,lang/swe/s_swe_t.srt,lang/fin/s_fin_t.srt,.urlset/~hmac=8504d13432af74e30891163351607f5d61316fa38acf6c98a8e8d10ab2da719a", "content_rating": { "age_restriction": 12, "reasons": [ "sex" ] } } } } DEBUG: Detected an HTML5 media DEBUG: Probing for stream flavors mime type is not rfc8216 compliant [hls @ 0x127e06de0] Can't support the subtitle(uri: hdntl=exp=1730735202~acl=%2fvod%2ffinland%2fhls%2f94-1_ol523w8o%2f,v_1280_720_1500.mp4,v_320_176_120.mp4,v_640_360_400.mp4,v_640_360_900.mp4,v_1280_720_2500.mp4,v_1920_1080_4000.mp4,lang%2fswe%2fs_swe_t.srt,lang%2ffin%2fs_fin_t.srt,.urlset%2f~data=hdntl~hmac=03106bc9dfa5272d6885c13c885fcca9cf154148f95ea21c4dd2f16229c9256a/index-f7.m3u8) [hls @ 0x127e06de0] Can't support the subtitle(uri: hdntl=exp=1730735202~acl=%2fvod%2ffinland%2fhls%2f94-1_ol523w8o%2f,v_1280_720_1500.mp4,v_320_176_120.mp4,v_640_360_400.mp4,v_640_360_900.mp4,v_1280_720_2500.mp4,v_1920_1080_4000.mp4,lang%2fswe%2fs_swe_t.srt,lang%2ffin%2fs_fin_t.srt,.urlset%2f~data=hdntl~hmac=03106bc9dfa5272d6885c13c885fcca9cf154148f95ea21c4dd2f16229c9256a/index-f8.m3u8) Unsupported codec with id 98313 for input stream 2 Unsupported codec with id 98313 for input stream 5 Unsupported codec with id 98313 for input stream 8 Unsupported codec with id 98313 for input stream 11 Unsupported codec with id 98313 for input stream 14 Unsupported codec with id 98313 for input stream 17 DEBUG: Available flavors: DEBUG: bitrate: 113.204, height: 176, width: 320, backends: ffmpeg DEBUG: bitrate: 453.428, height: 360, width: 640, backends: ffmpeg DEBUG: bitrate: 927.869, height: 360, width: 640, backends: ffmpeg DEBUG: bitrate: 1436.588, height: 720, width: 1280, backends: ffmpeg DEBUG: bitrate: 2554.573, height: 720, width: 1280, backends: ffmpeg DEBUG: bitrate: 3895.416, height: 1080, width: 1920, backends: ffmpeg DEBUG: max_height: None, max_bitrate: None DEBUG: Selected flavor: StreamFlavor(media_type='video', height=1080, width=1920, bitrate=3895.416, streams=[<yledl.backends.DASHHLSBackend object at 0x107a97650>]) DEBUG: Now trying downloader ffmpeg Output file: Drive My Car: 2023-11-15T06:00.mkv DEBUG: Executing: DEBUG: ffmpeg -y -headers 'X-Forwarded-For: ' -loglevel warning -thread_queue_size 2048 -seekable 0 -strict experimental -stats -analyzeduration 10000000 -probesize 80000000 -i 'https://ylekvodmod01.akamaized.net/vod/finland/hls/94-1_ol523w8o/,v_1280_720_1500.mp4,v_320_176_120.mp4,v_640_360_400.mp4,v_640_360_900.mp4,v_1280_720_2500.mp4,v_1920_1080_4000.mp4,lang/swe/s_swe_t.srt,lang/fin/s_fin_t.srt,.urlset/master.m3u8?hdnts=exp=1730652383~acl=/vod/finland/hls/94-1_ol523w8o/,v_1280_720_1500.mp4,v_320_176_120.mp4,v_640_360_400.mp4,v_640_360_900.mp4,v_1280_720_2500.mp4,v_1920_1080_4000.mp4,lang/swe/s_swe_t.srt,lang/fin/s_fin_t.srt,.urlset/*~hmac=8504d13432af74e30891163351607f5d61316fa38acf6c98a8e8d10ab2da719a' -metadata:s:v:0 'description=Viiden tähden arvioita kerännyt kuvaus tukahdutetun vihan ja surun voittamisesta. Tarinan päähenkilöitä ovat leskeksi jäänyt näyttelijä, tämän 20-vuotias autonkuljettaja ja punainen Saab 900. Tapahtumapaikka on Hiroshima. Haruki Murakamin novellista innoituksensa saanut erilainen road movie voitti parhaan ulkomaisen elokuvan Oscarin 2022. Ohjaus Ryusuke Hamaguchi, pääosissa Hidetoshi Nishijima, Tōko Miura, Masaki Okada. (Doraibu mai kā, Japani 2021)' -metadata creation_time=2023-11-15T06:00:00+02:00 -map '0:p:5:v?' -map '0:p:5:a?' -scodec srt -map '0:p:5:s?' -bsf:a aac_adtstoasc -vcodec copy -acodec copy -dn 'file:Drive My Car: 2023-11-15T06:00.mkv' mime type is not rfc8216 compliant [webvtt @ 0x128c1fa30] Dropping 853 duplicated subtitle events [webvtt @ 0x118705230] Dropping 720 duplicated subtitle events frame=212749 fps=232 q=-1.0 size= 4069120KiB time=02:21:49.96 bitrate=3917.1kbits/s speed=9.27x

[DarrenPIngram]

I couldn't see anything, so over to the expert dev. The only thing I reacted to was the difference in speed at the end. Could there be some flapping on your network (but it would be a strange thing if it was going up and down, and yet you've not experienced issues elsewhere).

Oh you may want to edit the previous post for privacy, assuming you are a elisa-laajakaista.fi user... you pasted it in your IP number.

[orthica]

No, I don't have Elisa, but I removed it anyway, took me a moment to find that ip from that wall of text. Thanks for heads up! The speed begins to decrease once the download hits the wall, and if I leave it on for long enough, it will get down to "zero".

[DarrenPIngram]

No, I don't have Elisa, but I removed it anyway, took me a moment to Maybe the IP which refers to Elisa is a different IP if you're using ISP x that might have been bought by them, or marketed under a different name. I didn't want to quote post it for obvious reasons :)

The speed begins to decrease once the download hits the wall, and if I leave it on for long enough, it will get down to "zero". I don't know the programming logic, or whether there would be scope to add further logging, but if the download is hitting a wall maybe it sort of times out or breaks if it hits zero for a similarly low figure for a certain period of time.

ffmpeg does have options that may give additional options, whether you can try amending it from the quoted debug output. I cannot claim if it is a wild goose chase or even incorrect advice.

-reconnect 1 Enables automatic reconnection. -reconnect_streamed 1 Enables reconnection for streamed data. -reconnect_delay_max 5 Sets the maximum delay in seconds before a reconnect attempt is made. You can adjust this as needed.

I have seen mention in the past for low speed networks, that setting things using flags like -timeout 5000000 -maxrate 500k (as example) may make a difference, but I didn't get the "why?"

I might try tomorrow if you don't get chance/have any luck. I'm supposed to be finishing off a task but this seems more interesting :)

[aajanki]

I don't see anything out of ordinary in the log either. This is a bit of a mystery.

Does it freeze also if you try a lower resolution by adding --resolution 720 to the yle-dl command line?

The IP address was actually not your IP. It's a randomly generated Finnish IP used to get past Areena geo-blocking.

[orthica]

Just out of curiosity, I decided to try to download with my computer connected to my phone as a hotspot. The speed was a lot lower (my home broadband is 100mbps), but the download was completed. I then tried to connect back to my home wifi and download, but the freeze happened again. So that would mean the issue has something to do with my home broadband, but it doesn't really make any sense why it always happens with this url (and once in the past with some other url), but all others always work perfectly. Even with longer films than this one I have never had any issues. 720p seems to download as well.

[DarrenPIngram]

The IP address was actually not your IP. It's a randomly generated Finnish IP used to get past Areena geo-blocking. I sit corrected.

Would there/should there be maybe a check for the user's IP to avoid sending traffic via that proxy which might not otherwise need it? (as in whoever is providing it, e.g. yourself, might have less used resources)?

Edit: based on the OP's latest update, is there a flag (I didn't see) to ignore the proxy, so they could try that from their "troubled" home broadband, just to rule out some weird traffic shaping or other stuff their ISP may be doing, if it is hitting a certain range for the proxies?

An interesting issue, but maybe it gives some extra data points for the problem you've noticed before but haven't tracked down, e.g. try a different connection, try without proxy (if there's a flag) and observe if the speed appears to be low(er) when it gives up.

[aajanki]

Interesting observation that it depends on the network connnection. I still don't understand why it should freeze on that one file on one of the connections.

Just to clarify: There is no actual proxy involved. Yle-dl just pretends to be a proxy to by-pass any geo-restrictions.

[orthica]

Thank you already with your help so far, and sorry about the hassle.

I tried a few things out of curiosity:

I tested if it could be film length issue, so I tried https://areena.yle.fi/1-50497668 which is over 2 hours, but it downloaded just fine. I found out the file size was "only" 2,44gb, where as the earlier film always froze around 4,15gb, so I tried with https://areena.yle.fi/1-3554229, which froze at 4,41gb and https://areena.yle.fi/1-4468364 which froze multiple times at 4,15gb, so not only does it appear to be an issue with my home broadband wifi (since the cellular data hotspot download worked for the first stream.), but perhaps also something to do with the file size? That would also "explain" why the download worked with 720 resolution. But maybe enough with this headache already for one night.