search

CasparCG / server

CasparCG Server is a Windows and Linux software used to play out professional graphics, audio and video to multiple outputs. It has been in 24/7 broadcast production since 2006. Ready-to-use downloads are available under the Releases tab https://casparcg.com.
GNU General Public License v3.0
905 stars 268 forks source link

Bug: HLS stream URLs get truncated in ffmpeg #1548

Open leftearsons opened 3 months ago

leftearsons commented 3 months ago

Observed Behavior

It is observed that in some cases, the internal ffmpeg library, truncates the HLS URL which causes valuable authentication data to be discarded. This leads to invalid video stream urls and eventually to stopping HLS playback.
By looking at the logs, the issue seems to take place inside the 'https' thread, since the 'hls' thread always logs the correct (full length) URL path.

The pattern of the failing URLs makes me think that the root cause of this case, could be the query parameter length of the HLS URL, since it is only observed when the URL contains the authentication data in query parameters, instead of inside the URL path.
We have been loading HLS video streams from the same video providers for a long time and no change has been made on the video source, meaning that the same HLS URLs format used to work well with CCG v2.07.

I am adding for your reference the log files for those two cases, a faulty one (in which the URL contains path parameters) and a successful one (in which the URL contains only path parameters).

Faulty case:

[info]    Received message from 127.0.0.1: PLAY 1-50 "https://lc-live-http-ipv4.akamaized.net/30714/4830385/desktop/table_tennis_international_tt_elite_series_retail_4830385_1364k.m3u8?cid=30714&mid=50853377&ecid=4830385&pid=1&dtid=1&sid=848252974901&gc=pu4&gsd=oA&grm=1&hdnts=ip=85.73.218.100~exp=1718859799~acl=%2F30714%2F4830385%2Fdesktop%2F%2A~hmac=d3766115ad77c74e068ac5e898cdf8e7a270c3ffc30f62ba160c5b8efa5856c9"\r\n
[info]    Sent message to 127.0.0.1:202 PLAY OK\r\n
[info]    [ffmpeg] [hls @ 0000022E3604E300] Skip ('#EXT-X-VERSION:3')
[info]    
[info]    [ffmpeg] [hls @ 0000022E3604E300] Opening 'https://lc-live-http-ipv4.akamaized.net/30714/4830385/desktop/table_tennis_international_tt_elite_series_retail_4830385_1364k_1718772001_0000690.ts?cid=30714&mid=50853377&ecid=4830385&pid=1&dtid=1&sid=848252974901&gc=pu4&gsd=oA&grm=1&hdnts=ip=85.73.218.100~exp=1718859799~acl=%2F30714%2F4830385%2Fdesktop%2F%2A~hmac=d3766115ad77c74e068ac5e898cdf8e7a270c3ffc30f62ba160c5b8efa5856c9' for reading
[info]    
[info]    [ffmpeg] [hls @ 0000022E3604E300] Opening 'https://lc-live-http-ipv4.akamaized.net/30714/4830385/desktop/table_tennis_international_tt_elite_series_retail_4830385_1364k_1718772001_0000691.ts?cid=30714&mid=50853377&ecid=4830385&pid=1&dtid=1&sid=848252974901&gc=pu4&gsd=oA&grm=1&hdnts=ip=85.73.218.100~exp=1718859799~acl=%2F30714%2F4830385%2Fdesktop%2F%2A~hmac=d3766115ad77c74e068ac5e898cdf8e7a270c3ffc30f62ba160c5b8efa5856c9' for reading
[info]    
[info]    [ffmpeg] [https @ 0000022E328F2040] Opening 'https://lc-live-http-ipv4.akamaized.net/30714/4830385/desktop/table_tennis_international_tt_elite_series_retail_4830385_1364k_1718772001_0000692.ts?cid=30714&mid=50853377&ecid=4830385&pid=1&dtid=1&sid=848252974901&gc=pu4&gsd=oA&grm=1&hdnts=ip=85.73.218.100~exp=1718859799~acl=%2F30714%2F4830385%2Fdesktop%2F%2A~hmac=d3766115ad' for reading
[info]    
[warning] ffmpeg[https://lc-live-http-ipv4.akamaized.net/30714/4830385/desktop/table_tennis_international_tt_elite_series_retail_4830385_1364k.m3u8?cid=30714&mid=50853377&ecid=4830385&pid=1&dtid=1&sid=848252974901&gc=pu4&gsd=oA&grm=1&hdnts=ip=85.73.218.100~exp=1718859799~acl=%2F30714%2F4830385%2Fdesktop%2F%2A~hmac=d3766115ad77c74e068ac5e898cdf8e7a270c3ffc30f62ba160c5b8efa5856c9|0.0000/0.0000] Latency: 41
[warning] [ffmpeg] [https @ 0000022E5DEA41C0] HTTP error 403 Forbidden
[warning] 
[warning] [ffmpeg] [hls @ 0000022E3604E300] keepalive request failed for 'https://lc-live-http-ipv4.akamaized.net/30714/4830385/desktop/table_tennis_international_tt_elite_series_retail_4830385_1364k_1718772001_0000692.ts?cid=30714&mid=50853377&ecid=4830385&pid=1&dtid=1&sid=848252974901&gc=pu4&gsd=oA&grm=1&hdnts=ip=85.73.218.100~exp=1718859799~acl=%2F30714%2F4830385%2Fdesktop%2F%2A~hmac=d3766115ad' with error: 'Server returned 403 Forbidden (access denied)' when opening url, retrying with new connection
[warning] 
[info]    [ffmpeg] [hls @ 0000022E3604E300] Opening 'https://lc-live-http-ipv4.akamaized.net/30714/4830385/desktop/table_tennis_international_tt_elite_series_retail_4830385_1364k_1718772001_0000692.ts?cid=30714&mid=50853377&ecid=4830385&pid=1&dtid=1&sid=848252974901&gc=pu4&gsd=oA&grm=1&hdnts=ip=85.73.218.100~exp=1718859799~acl=%2F30714%2F4830385%2Fdesktop%2F%2A~hmac=d3766115ad' for reading
[info]    
[warning] [ffmpeg] [https @ 0000022E57BF8D40] HTTP error 403 Forbidden
[warning] 
[warning] [ffmpeg] [hls @ 0000022E3604E300] Failed to open segment 692 of playlist 0

Successful case:

[info]    Received message from 127.0.0.1: PLAY 1-50 "https://744133.akamaized.net/hls/live/716232/ch12/89be75f87f0cbc976470fc2feb7ddd34db9a5390/video-m5_video/hdntl=exp=1719398046~acl=%2f*~id=668b56aecfe14366a26602c686807954~data=hdntl~hmac=be026824432c7fdbeb164ebe0cf92d159a38e2e18191633937b89732a8367be1/video.m3u8"\r\n
[info]    Sent message to 127.0.0.1:202 PLAY OK\r\n
[info]    [ffmpeg] [hls @ 000001DB7B8BE0C0] Skip ('#EXT-X-VERSION:3')
[info]    
[info]    [ffmpeg] [hls @ 000001DB7B8BE0C0] Skip ('#EXT-X-INDEPENDENT-SEGMENTS')
[info]    
[info]    [ffmpeg] [hls @ 000001DB7B8BE0C0] Skip ('#EXT-X-PROGRAM-DATE-TIME:2024-06-25T10:34:00.000Z')
[info]    
[info]    [ffmpeg] [hls @ 000001DB7B8BE0C0] Skip ('#EXT-X-PROGRAM-DATE-TIME:2024-06-25T10:34:02.000Z')
[info]    
[info]    [ffmpeg] [hls @ 000001DB7B8BE0C0] Skip ('#EXT-X-PROGRAM-DATE-TIME:2024-06-25T10:34:04.000Z')
[info]    
[info]    [ffmpeg] [hls @ 000001DB7B8BE0C0] Opening 'https://744133.akamaized.net/hls/live/716232/ch12/89be75f87f0cbc976470fc2feb7ddd34db9a5390/video-m5_video/hdntl=exp=1719398046~acl=%2f*~id=668b56aecfe14366a26602c686807954~data=hdntl~hmac=be026824432c7fdbeb164ebe0cf92d159a38e2e18191633937b89732a8367be1/4/file2357.ts' for reading
[info]    
[info]    [ffmpeg] [hls @ 000001DB7B8BE0C0] Opening 'https://744133.akamaized.net/hls/live/716232/ch12/89be75f87f0cbc976470fc2feb7ddd34db9a5390/video-m5_video/hdntl=exp=1719398046~acl=%2f*~id=668b56aecfe14366a26602c686807954~data=hdntl~hmac=be026824432c7fdbeb164ebe0cf92d159a38e2e18191633937b89732a8367be1/4/file2358.ts' for reading
[info]    
[info]    [ffmpeg] [https @ 000001DB7BA9A2C0] Opening 'https://744133.akamaized.net/hls/live/716232/ch12/89be75f87f0cbc976470fc2feb7ddd34db9a5390/video-m5_video/hdntl=exp=1719398046~acl=%2f*~id=668b56aecfe14366a26602c686807954~data=hdntl~hmac=be026824432c7fdbeb164ebe0cf92d159a38e2e18191633937b89732a8367be1/4/file2359.ts' for reading
[info]    
[warning] ffmpeg[https://744133.akamaized.net/hls/live/716232/ch12/89be75f87f0cbc976470fc2feb7ddd34db9a5390/video-m5_video/hdntl=exp=1719398046~acl=%2f*~id=668b56aecfe14366a26602c686807954~data=hdntl~hmac=be026824432c7fdbeb164ebe0cf92d159a38e2e18191633937b89732a8367be1/video.m3u8|0.0000/0.0000] Latency: 39

Expected behaviour

When loading an HLS stream, we would expect CasparCG to start the playback of it's chunks without altering their URL, eventually causing validation errors on the video streaming server.

Steps to reproduce

  1. Load an HLS video stream with PLAY AMCP command, which uses long list of query parameters with authentication tokens
  2. Observe the CasparCG logs and check for truncated HLS video stream URL when comparing the hls ffmpeg thread and the https ffmpeg thread logs, which would eventually cause HTTP validation errors (403 in our case)

Environment

Issue was not present in version 2.07

tdotmgc commented 6 days ago

I have the same problem when downloading videos from PornHub. The site generates three types of links to videos. Two types of links are short and one type is long. FFmpeg cuts off part of the address, so the video does not download.

For example: Source URL of the video https://cv-h.phncdn.com/hls/videos/202106/29/000000000/1080P_4000K_000000000.mp4/index-v1-a1.m3u8?6MB50lVyuE3XsoDstxOgtlFIS5WiZafP8gykHFLhkeDN6QJHJBdWP584qIVpgkovqyG90vdUcANTJdoyQlEEIfY01AREoa2bE6fFAIgGstWPRzp1XagL4Q1HkrjiEh4ABMzCcuW3ncZCD-Hczh2S2Gc_Jq2YdMpAmbrbbQwUJO0FbaTzlGiD5mri4xWZD0MTNeizdYeJK-w Last 16 simbols are cutted by FFmpeg https://cv-h.phncdn.com/hls/videos/202106/29/000000000/1080P_4000K_000000000.mp4/index-v1-a1.m3u8?6MB50lVyuE3XsoDstxOgtlFIS5WiZafP8gykHFLhkeDN6QJHJBdWP584qIVpgkovqyG90vdUcANTJdoyQlEEIfY01AREoa2bE6fFAIgGstWPRzp1XagL4Q1HkrjiEh4ABMzCcuW3ncZCD-Hczh2S2Gc_Jq2YdMpAmbrbbQwUJO0FbaTzlGiD5mri4xW

Julusian commented 2 days ago

I'm not able to reproduce this on linux, even when using much longer urls.

leftearsons commented 4 hours ago

@Julusian are there any chances that this only takes place in Windows? Just to add here that this seems to be happening only with some specific video providers and not always (8-9 out of 10 times though). Any thoughts what could possibly cause such issues? Is it possible that this could be related to any 3rd party library installed or even to the specific stream URLs format? This issue was first started about a year ago and was not there in the past, even for the same video provider that currently faces the issue. I would really appreciate if you could have a second look on this, since this affects our production systems, causing a lot of issues. Thank you!