search

pabloromeo / clusterplex

ClusterPlex is an extended version of Plex, which supports distributed Workers across a cluster to handle transcoding requests.
MIT License
445 stars 36 forks source link

k8s localRelayEnabled results in transcoder error (502 Bad Gateway) #328

Open AngellusMortis opened 1 month ago

AngellusMortis commented 1 month ago

Describe the bug

Using the default setting for the local relay in the Helm chart for k8s results in the transcoder failing. In the plex-worker logs, it looks like it is getting a 502 Bad Gateway when it tries to post the chunks back to the relay.

To Reproduce Steps to reproduce the behavior:

  1. Deploy Cluster Plex with Helm and Remote Only Workers
  2. Try to transcode

Expected behavior

Transcoder should work as expected.

Screenshots

plex-worker log

[http @ 0x7f9b9426be40] HTTP error 502 Bad Gateway
[tcp @ 0x7f9b9e14ee80] Starting connection attempt to 10.101.77.235 port 32499
[tcp @ 0x7f9b9e14ee80] Successfully connected to 10.101.77.235 port 32499

nginx error log for relay

2024/08/17 00:29:43 [error] 867#867: *3332 upstream sent duplicate header line: "Content-Length: 83", previous value: "Content-Length: 83" while reading response header from upstream, client: 10.244.3.24, server: , request: "PUT /video/:/transcode/session/rwcyq466ntin5daii1hxd2kh/9007548b-bb8b-45e4-ae4f-72f857e89460/progress?progress=0.5&size=-22&remaining=32324&vdec_packets=1318&vdec_hw_ok=1299&speed=0.4&vdec_hw_status=1 HTTP/1.1", upstream: "http://127.0.0.1:32400/video/:/transcode/session/rwcyq466ntin5daii1hxd2kh/9007548b-bb8b-45e4-ae4f-72f857e89460/progress?progress=0.5&size=-22&remaining=32324&vdec_packets=1318&vdec_hw_ok=1299&speed=0.4&vdec_hw_status=1", host: "plex-pms:32499"
2024/08/17 00:29:43 [error] 867#867: *3335 upstream sent duplicate header line: "Content-Length: 83", previous value: "Content-Length: 83" while reading response header from upstream, client: 10.244.3.24, server: , request: "PUT /video/:/transcode/session/rwcyq466ntin5daii1hxd2kh/9007548b-bb8b-45e4-ae4f-72f857e89460/progress?progress=0.5&size=-22&remaining=32324&vdec_packets=1318&vdec_hw_ok=1299&speed=0.4&vdec_hw_status=1 HTTP/1.1", upstream: "http://127.0.0.1:32400/video/:/transcode/session/rwcyq466ntin5daii1hxd2kh/9007548b-bb8b-45e4-ae4f-72f857e89460/progress?progress=0.5&size=-22&remaining=32324&vdec_packets=1318&vdec_hw_ok=1299&speed=0.4&vdec_hw_status=1", host: "plex-pms:32499"
2024/08/17 00:29:44 [error] 867#867: *3337 upstream sent duplicate header line: "Content-Length: 83", previous value: "Content-Length: 83" while reading response header from upstream, client: 10.244.3.24, server: , request: "PUT /video/:/transcode/session/rwcyq466ntin5daii1hxd2kh/9007548b-bb8b-45e4-ae4f-72f857e89460/progress?progress=0.5&size=-22&remaining=30546&vdec_packets=1323&vdec_hw_ok=1304&speed=0.4&vdec_hw_status=1 HTTP/1.1", upstream: "http://127.0.0.1:32400/video/:/transcode/session/rwcyq466ntin5daii1hxd2kh/9007548b-bb8b-45e4-ae4f-72f857e89460/progress?progress=0.5&size=-22&remaining=30546&vdec_packets=1323&vdec_hw_ok=1304&speed=0.4&vdec_hw_status=1", host: "plex-pms:32499"
2024/08/17 00:29:44 [error] 867#867: *3339 upstream sent duplicate header line: "Content-Length: 83", previous value: "Content-Length: 83" while reading response header from upstream, client: 10.244.3.24, server: , request: "PUT /video/:/transcode/session/rwcyq466ntin5daii1hxd2kh/9007548b-bb8b-45e4-ae4f-72f857e89460/progress?progress=0.5&size=-22&remaining=34969&vdec_packets=1327&vdec_hw_ok=1308&speed=0.3&vdec_hw_status=1 HTTP/1.1", upstream: "http://127.0.0.1:32400/video/:/transcode/session/rwcyq466ntin5daii1hxd2kh/9007548b-bb8b-45e4-ae4f-72f857e89460/progress?progress=0.5&size=-22&remaining=34969&vdec_packets=1327&vdec_hw_ok=1308&speed=0.3&vdec_hw_status=1", host: "plex-pms:32499"
2024/08/17 00:29:45 [error] 867#867: *3341 upstream sent duplicate header line: "Content-Length: 83", previous value: "Content-Length: 83" while reading response header from upstream, client: 10.244.3.24, server: , request: "PUT /video/:/transcode/session/rwcyq466ntin5daii1hxd2kh/9007548b-bb8b-45e4-ae4f-72f857e89460/progress?progress=0.5&size=-22&remaining=35121&vdec_packets=1332&vdec_hw_ok=1313&speed=0.4&vdec_hw_status=1 HTTP/1.1", upstream: "http://127.0.0.1:32400/video/:/transcode/session/rwcyq466ntin5daii1hxd2kh/9007548b-bb8b-45e4-ae4f-72f857e89460/progress?progress=0.5&size=-22&remaining=35121&vdec_packets=1332&vdec_hw_ok=1313&speed=0.4&vdec_hw_status=1", host: "plex-pms:32499"
2024/08/17 00:29:45 [error] 867#867: *3343 upstream sent duplicate header line: "Content-Length: 83", previous value: "Content-Length: 83" while reading response header from upstream, client: 10.244.3.24, server: , request: "PUT /video/:/transcode/session/rwcyq466ntin5daii1hxd2kh/9007548b-bb8b-45e4-ae4f-72f857e89460/progress?progress=0.5&size=-22&remaining=31380&vdec_packets=1337&vdec_hw_ok=1318&speed=0.4&vdec_hw_status=1 HTTP/1.1", upstream: "http://127.0.0.1:32400/video/:/transcode/session/rwcyq466ntin5daii1hxd2kh/9007548b-bb8b-45e4-ae4f-72f857e89460/progress?progress=0.5&size=-22&remaining=31380&vdec_packets=1337&vdec_hw_ok=1318&speed=0.4&vdec_hw_status=1", host: "plex-pms:32499"
2024/08/17 00:29:45 [error] 867#867: *3356 upstream sent duplicate header line: "Content-Length: 83", previous value: "Content-Length: 83" while reading response header from upstream, client: 10.244.3.24, server: , request: "PUT /video/:/transcode/session/rwcyq466ntin5daii1hxd2kh/9007548b-bb8b-45e4-ae4f-72f857e89460/progress?progress=0.5&size=-22&remaining=31380&vdec_packets=1337&vdec_hw_ok=1318&speed=0.4&vdec_hw_status=1 HTTP/1.1", upstream: "http://127.0.0.1:32400/video/:/transcode/session/rwcyq466ntin5daii1hxd2kh/9007548b-bb8b-45e4-ae4f-72f857e89460/progress?progress=0.5&size=-22&remaining=31380&vdec_packets=1337&vdec_hw_ok=1318&speed=0.4&vdec_hw_status=1", host: "plex-pms:32499"
2024/08/17 00:29:46 [error] 867#867: *3358 upstream sent duplicate header line: "Content-Length: 83", previous value: "Content-Length: 83" while reading response header from upstream, client: 10.244.3.24, server: , request: "PUT /video/:/transcode/session/rwcyq466ntin5daii1hxd2kh/9007548b-bb8b-45e4-ae4f-72f857e89460/progress?progress=0.5&size=-22&remaining=31276&vdec_packets=1342&vdec_hw_ok=1323&speed=0.4&vdec_hw_status=1 HTTP/1.1", upstream: "http://127.0.0.1:32400/video/:/transcode/session/rwcyq466ntin5daii1hxd2kh/9007548b-bb8b-45e4-ae4f-72f857e89460/progress?progress=0.5&size=-22&remaining=31276&vdec_packets=1342&vdec_hw_ok=1323&speed=0.4&vdec_hw_status=1", host: "plex-pms:32499"
2024/08/17 00:29:46 [error] 867#867: *3361 upstream sent duplicate header line: "Content-Length: 83", previous value: "Content-Length: 83" while reading response header from upstream, client: 10.244.3.24, server: , request: "PUT /video/:/transcode/session/rwcyq466ntin5daii1hxd2kh/9007548b-bb8b-45e4-ae4f-72f857e89460/progress?progress=0.5&size=-22&remaining=31276&vdec_packets=1342&vdec_hw_ok=1323&speed=0.4&vdec_hw_status=1 HTTP/1.1", upstream: "http://127.0.0.1:32400/video/:/transcode/session/rwcyq466ntin5daii1hxd2kh/9007548b-bb8b-45e4-ae4f-72f857e89460/progress?progress=0.5&size=-22&remaining=31276&vdec_packets=1342&vdec_hw_ok=1323&speed=0.4&vdec_hw_status=1", host: "plex-pms:32499"

nginx access log for relay

10.244.3.24 - - [17/Aug/2024:00:29:48 -0400] "POST /video/:/transcode/session/rwcyq466ntin5daii1hxd2kh/9007548b-bb8b-45e4-ae4f-72f857e89460/progress/log?level=3&message=%5Bdash%20%40%200x7f9bc9c35040%5D%20Representation%201%20media%20segment%2055%20written%20to%3A%20chunk-stream1-00054.m4s HTTP/1.1" 200 0 "-" "Lavf/LIBAVFORMAT_VERSION"
10.244.3.24 - - [17/Aug/2024:00:29:48 -0400] "POST /video/:/transcode/session/rwcyq466ntin5daii1hxd2kh/9007548b-bb8b-45e4-ae4f-72f857e89460/progress/log?level=2&message=%5Bdash%20%40%200x7f9bc9c35040%5D%20Opening%20%27http%3A%2F%2Fplex-pms%3A32499%2Fvideo%2F%3A%2Ftranscode%2Fsession%2Frwcyq466ntin5daii1hxd2kh%2F9007548b-bb8b-45e4-ae4f-72f857e89460%2Fmanifest%3FX-Plex-Http-Pipeline%3Dinfinite%27%20for%20writing HTTP/1.1" 200 0 "-" "Lavf/LIBAVFORMAT_VERSION"
10.244.3.24 - - [17/Aug/2024:00:29:48 -0400] "POST /video/:/transcode/session/rwcyq466ntin5daii1hxd2kh/9007548b-bb8b-45e4-ae4f-72f857e89460/progress/log?level=3&message=%5Btcp%20%40%200x7f9bcb0cd580%5D%20Starting%20connection%20attempt%20to%2010.101.77.235%20port%2032499 HTTP/1.1" 200 0 "-" "Lavf/LIBAVFORMAT_VERSION"
10.244.3.24 - - [17/Aug/2024:00:29:48 -0400] "POST /video/:/transcode/session/rwcyq466ntin5daii1hxd2kh/9007548b-bb8b-45e4-ae4f-72f857e89460/progress/log?level=3&message=%5Btcp%20%40%200x7f9bcb0cd580%5D%20Successfully%20connected%20to%2010.101.77.235%20port%2032499 HTTP/1.1" 200 0 "-" "Lavf/LIBAVFORMAT_VERSION"
10.244.3.24 - - [17/Aug/2024:00:29:48 -0400] "POST /video/:/transcode/session/rwcyq466ntin5daii1hxd2kh/9007548b-bb8b-45e4-ae4f-72f857e89460/progress/log?level=3&message=%5BAVIOContext%20%40%200x7f9b9426b040%5D%20Statistics%3A%202111%20bytes%20written%2C%200%20seeks%2C%201%20writeouts HTTP/1.1" 200 0 "-" "Lavf/LIBAVFORMAT_VERSION"
10.244.3.24 - - [17/Aug/2024:00:29:48 -0400] "POST /video/:/transcode/session/rwcyq466ntin5daii1hxd2kh/9007548b-bb8b-45e4-ae4f-72f857e89460/manifest?X-Plex-Http-Pipeline=infinite HTTP/1.1" 499 0 "-" "Lavf/LIBAVFORMAT_VERSION"
10.244.3.24 - - [17/Aug/2024:00:29:48 -0400] "POST /video/:/transcode/session/rwcyq466ntin5daii1hxd2kh/9007548b-bb8b-45e4-ae4f-72f857e89460/progress/log?level=2&message=%5Bdash%20%40%200x7f9bc9c35040%5D%20Opening%20%27chunk-stream0-00055.m4s.tmp%27%20for%20writing HTTP/1.1" 200 0 "-" "Lavf/LIBAVFORMAT_VERSION"
10.244.3.24 - - [17/Aug/2024:00:29:48 -0400] "POST /video/:/transcode/session/rwcyq466ntin5daii1hxd2kh/9007548b-bb8b-45e4-ae4f-72f857e89460/progress/log?level=2&message=%5Bdash%20%40%200x7f9bc9c35040%5D%20Opening%20%27chunk-stream1-00055.m4s.tmp%27%20for%20writing HTTP/1.1" 200 0 "-" "Lavf/LIBAVFORMAT_VERSION"

Desktop (please complete the following information):

(not a client issue)

Additional context

Cluster is on bare metal running Talos OS 1.7.6 and k8s 1.30.3. PMS is 1.40.5. Cluster Plex is 1.4.13 (latest everything, just installed today).

Disabling the relay, whitelisting all IPs and adding the plex-pms service IP to the pmsIP in the Helm chart fixes everything.

pabloromeo commented 2 weeks ago

Let me try to reproduce it