RTP Packet Lost when video proxy

wolverin-a commented 2 months ago

Which version are you using?

v1.8.0

Which operating system are you using?

[x] Linux amd64 standard
[ ] Linux amd64 Docker
[ ] Linux arm64 standard
[ ] Linux arm64 Docker
[ ] Linux arm7 standard
[ ] Linux arm7 Docker
[ ] Linux arm6 standard
[ ] Linux arm6 Docker
[ ] Windows amd64 standard
[ ] Windows amd64 Docker (WSL backend)
[ ] macOS amd64 standard
[ ] macOS amd64 Docker
[ ] Other (please describe)

Debian 10 amd64 E3-1230v5, 32 GB RAM WAN 1000 mBit/s

Describe the issue

Hello. I'm using your app as a video proxy. All paths are added via the API.

Describe how to replicate the issue

But additionally, I specified the following parameters in the configuration file

writeQueueSize: 524288
readBufferCount: 32768
udpMaxPayloadSize: 1472
protocols: [tcp]

and added in /etc/sysctl.conf

net.core.rmem_max = 16777216
net.core.wmem_max = 16777216

net.ipv4.tcp_mem  = 8192 131072 16777216
net.ipv4.tcp_rmem = 8192 131072 16777216
net.ipv4.tcp_wmem = 8192 131072 16777216

Did you attach the server logs?

yes

But it still causes the loss of RTP packets and sometimes I see it in the logs

апр 11 14:46:12 dev rtsp-simple-server[17143]: 2024/04/11 14:46:12 INF [RTSP] [conn 46.229.101.16:37748] closed: EOF
апр 11 14:46:12 dev rtsp-simple-server[17143]: 2024/04/11 14:46:12 WAR [path 02c00081491be26a] [RTSP source] 16 RTP packets lost
апр 11 14:46:12 dev rtsp-simple-server[17143]: 2024/04/11 14:46:12 WAR [RTSP] [session a15ea9c4] write queue is full

апр 11 14:48:15 dev rtsp-simple-server[17143]: 2024/04/11 14:48:15 DEB [path 02c00081491be26a] [RTSP source] [s->c] RTSP/1.0 200 OK
апр 11 14:48:15 dev rtsp-simple-server[17143]: 2024/04/11 14:48:15 INF [RTSP] [session 583dc942] is reading from path '02c00081491be26a', with TCP, 1 track (H264)
апр 11 14:48:15 dev rtsp-simple-server[17143]: RTP-Info: url=rtsp://ip:port/02c00081491be26a/trackID=0;seq=4917;rtptime=2113445524
апр 11 14:48:17 dev rtsp-simple-server[17143]: 2024/04/11 14:48:17 WAR [path 02c00081491be26a] [RTSP source] 20 RTP packets lost
апр 11 14:48:18 dev rtsp-simple-server[17143]: 2024/04/11 14:48:18 WAR [path 02c00081491be26a] [RTSP source] 20 RTP packets lost
апр 11 14:48:20 dev rtsp-simple-server[17143]: 2024/04/11 14:48:20 WAR [path 02c00081491be26a] [RTSP source] 20 RTP packets lost
апр 11 14:48:22 dev rtsp-simple-server[17143]: 2024/04/11 14:48:22 WAR [path 02c00081491be26a] [RTSP source] 20 RTP packets lost
апр 11 14:48:24 dev rtsp-simple-server[17143]: 2024/04/11 14:48:24 WAR [path 02c00081491be26a] [RTSP source] 20 RTP packets lost
апр 11 14:48:26 dev rtsp-simple-server[17143]: 2024/04/11 14:48:26 WAR [path 02c00081491be26a] [RTSP source] 20 RTP packets lost
апр 11 14:48:27 dev rtsp-simple-server[17143]: 2024/04/11 14:48:27 WAR [path 02c00081491be26a] [RTSP source] 20 RTP packets lost
апр 11 14:48:29 dev rtsp-simple-server[17143]: 2024/04/11 14:48:29 WAR [path 02c00081491be26a] [RTSP source] 20 RTP packets lost

# cat /proc/net/tcp
  sl  local_address rem_address   st tx_queue rx_queue tr tm->when retrnsmt   uid  timeout inode
   0: 0100007F:0019 00000000:0000 0A 00000000:00000000 00:00000000 00000000     0        0 21981 1 0000000047af9b79 100 0 0 10 20
   1: 00000000:280A 00000000:0000 0A 00000000:00000000 00:00000000 00000000     0        0 8983109 1 000000004d754855 100 0 0 10 0
   2: 0100007F:0CEA 00000000:0000 0A 00000000:00000000 00:00000000 00000000   106        0 21520 1 00000000a6464b5c 100 0 0 10 0
   3: 00000000:280B 00000000:0000 0A 00000000:00000000 00:00000000 00000000     0        0 8983110 1 000000004ef99328 100 0 0 10 0
   4: 0100007F:13AE 00000000:0000 0A 00000000:00000000 00:00000000 00000000   111        0 18831504 1 00000000e3413727 100 0 0 10 0
   5: 00000000:286E 00000000:0000 0A 00000000:00000000 00:00000000 00000000     0        0 8983107 1 000000002e59b09a 100 0 0 10 0
   6: 00000000:280F 00000000:0000 0A 00000000:00000000 00:00000000 00000000     0        0 8983108 1 000000008ac28112 100 0 0 10 0
   7: 00000000:006F 00000000:0000 0A 00000000:00000000 00:00000000 00000000     0        0 13556 1 00000000008c0bcb 100 0 0 10 0
   8: 00000000:1F90 00000000:0000 0A 00000000:00000000 00:00000000 00000000     0        0 8983111 1 0000000037e48d60 100 0 0 10 0
   9: 00000000:0050 00000000:0000 0A 00000000:00000000 00:00000000 00000000     0        0 8983106 1 000000001885dada 100 0 0 10 0
  10: 0100007F:2651 00000000:0000 0A 00000000:00000000 00:00000000 00000000     0        0 25155112 1 0000000094091943 100 0 0 10 0
  11: 00000000:7DF4 00000000:0000 0A 00000000:00000000 00:00000000 00000000   111        0 18831512 1 000000009057f590 100 0 0 10 0
  12: 00000000:0016 00000000:0000 0A 00000000:00000000 00:00000000 00000000     0        0 831490 1 000000009a3b7650 100 0 0 10 0
  13: 00000000:1F98 00000000:0000 0A 00000000:00000000 00:00000000 00000000   111        0 18831503 1 000000000fb5c601 100 0 0 10 0
  14: 4C3AF957:006F 3567DF4D:C346 01 00000000:00000000 00:00000000 00000000   109        0 4793397 1 00000000d6ffd21e 22 0 0 10 -1
  15: 0100870A:D688 77F9870A:022A 01 00000000:00000000 02:000001ED 00000000     0        0 38221656 2 00000000dc4abffa 24 4 30 10 -1
  16: 0100870A:ABB2 EEFA870A:022A 01 00000000:00000000 02:00000169 00000000     0        0 38221617 2 00000000a4d7883e 23 4 30 10 -1
  17: 0100870A:8754 34FE870A:022A 01 00000000:00000000 02:00000369 00000000     0        0 38227400 2 00000000e153d385 22 4 30 10 -1
  18: 4C3AF957:006F 459EC0D4:E9A5 01 00000000:00000000 00:00000000 00000000   109        0 14523620 1 00000000c1cd4e3e 22 0 0 10 -1
  19: 0100870A:969A B0FB870A:022A 01 00000000:00000000 02:0000059C 00000000     0        0 38229569 2 00000000a16ca61c 24 4 24 10 -1
  20: 4C3AF957:280B D0871F5E:8808 06 00000000:00000000 03:0000170C 00000000     0        0 0 3 000000007862a404
  21: 4C3AF957:280A D0871F5E:A3C0 06 00000000:00000000 03:000014AC 00000000     0        0 0 3 0000000005e59d21
  22: 0100870A:CF86 4AF8870A:022A 06 00000000:00000000 03:00000F9B 00000000     0        0 0 3 000000008959f69e
  23: 4C3AF957:006F E7589BC0:DF44 01 00000000:00000000 00:00000000 00000000   109        0 321315 1 0000000083a418e9 44 4 30 10 -1
  24: 4C3AF957:006F 499EC0D4:B22D 01 00000000:00000000 00:00000000 00000000   109        0 15103547 1 0000000085d06531 22 0 0 10 -1
  25: 4C3AF957:006F 479EC0D4:9BBF 01 00000000:00000000 00:00000000 00000000   109        0 14405377 1 00000000e4aa314c 22 0 0 10 -1
  26: 0100870A:B200 F7FD870A:022A 01 00000000:00000000 02:000004E9 00000000     0        0 38224169 2 00000000af8e518d 23 4 30 10 -1
  27: 4C3AF957:280B D0871F5E:8804 06 00000000:00000000 03:000014D8 00000000     0        0 0 3 0000000024c5e72b
  28: 4C3AF957:006F 3567DF4D:C348 01 00000000:00000000 00:00000000 00000000   109        0 4793396 1 000000007a94419b 22 0 0 10 -1
  29: 4C3AF957:006F 086D212D:B2BC 01 00000000:00000000 00:00000000 00000000   109        0 33826637 1 000000002516cb4c 58 4 30 10 -1
  30: 0100870A:DE48 88F9870A:022A 01 00000000:00000000 02:00000250 00000000     0        0 38213460 2 00000000586191b0 24 4 30 10 -1
  31: 0100870A:D580 86F6870A:022A 01 00000000:00000000 02:0000059C 00000000     0        0 38214870 2 000000009602ec6c 23 4 30 10 -1
  32: 4C3AF957:006F 3567DF4D:E017 01 00000000:00000000 00:00000000 00000000   109        0 380611 1 00000000ad523181 22 0 0 10 -1
  33: 4C3AF957:006F 7A44D0B0:C184 01 00000000:00000000 00:00000000 00000000   109        0 15257655 1 00000000d06a55ac 42 0 0 10 -1
  34: 0100870A:8168 CFF9870A:022A 06 00000000:00000000 03:00000000 00000000     0        0 0 3 0000000014a44e8e
  35: 4C3AF957:006F 4A9EC0D4:986D 01 00000000:00000000 00:00000000 00000000   109        0 14548241 1 00000000f11c0739 22 0 0 10 -1
  36: 4C3AF957:280B D0871F5E:8806 06 00000000:00000000 03:000014E6 00000000     0        0 0 3 00000000e39ba928
  37: 0100870A:B058 2FFB870A:022A 01 00000000:00000000 02:00000569 00000000     0        0 38229253 2 00000000dfc0db9e 24 4 30 10 -1
  38: 4C3AF957:006F E7589BC0:DF47 01 00000000:00000000 00:00000000 00000000   109        0 321319 1 000000000ad14568 42 4 30 10 -1
  39: 4C3AF957:280B 128BE75E:C5DE 06 00000000:00000000 03:00000793 00000000     0        0 0 3 00000000c775585f
  40: 4C3AF957:006F 3567DF4D:D513 01 00000000:00000000 00:00000000 00000000   109        0 364516 1 00000000cacb76ad 22 0 0 10 -1
  41: 4C3AF957:006F 489EC0D4:B826 01 00000000:00000000 00:00000000 00000000   109        0 16668715 1 00000000f120a542 23 0 0 10 -1
  42: 4C3AF957:280B 128BE75E:C5DC 06 00000000:00000000 03:00000789 00000000     0        0 0 3 000000001a1cb32a
  43: 4C3AF957:0016 06D7F957:C76C 01 000011E0:00000000 01:00000026 00000000     0        0 10989914 4 000000008bae560d 50 4 31 10 13
  44: 4C3AF957:006F 086D212D:B2B6 01 00000000:00000000 00:00000000 00000000   109        0 33821658 1 000000006143438a 74 4 30 10 -1
  45: 0100870A:E6CC 86F9870A:022A 01 00000000:00000000 02:000001D6 00000000     0        0 38186984 2 000000005f170a32 26 4 30 10 -1
  46: 0100870A:B70A 81FA870A:022A 06 00000000:00000000 03:00000E04 00000000     0        0 0 3 00000000c5be5d7a
  47: 4C3AF957:006F 3245AF55:8C31 01 00000000:00000000 00:00000000 00000000   109        0 14360573 1 0000000064fa0e40 24 0 0 10 -1
  48: 4C3AF957:006F 3567DF4D:98C4 01 00000000:00000000 00:00000000 00000000   109        0 13061049 1 000000000688ca7c 22 0 0 10 -1
  49: 4C3AF957:006F 469EC0D4:C94E 01 00000000:00000000 00:00000000 00000000   109        0 18071464 1 000000007cac9947 22 0 0 10 -1
  50: 4C3AF957:006F 469EC0D4:C950 01 00000000:00000000 00:00000000 00000000   109        0 18067296 1 0000000046968467 21 0 0 10 -1
  51: 4C3AF957:006F 489EC0D4:9BE5 01 00000000:00000000 00:00000000 00000000   109        0 35447380 1 000000006742d16b 22 0 0 10 -1
  52: 4C3AF957:006F ED18EBC6:CA43 01 00000000:00000000 00:00000000 00000000   109        0 35086218 1 0000000037b066ad 196 0 0 10 -1
  53: 4C3AF957:006F 7243D0B0:DC80 01 00000000:00000000 00:00000000 00000000   109        0 15198505 1 0000000096170087 52 0 0 10 -1
  54: 4C3AF957:006F 3567DF4D:E015 01 00000000:00000000 00:00000000 00000000   109        0 380612 1 00000000ca55d5bb 22 0 0 10 -1
  55: 4C3AF957:006F 3245AF55:8C30 01 00000000:00000000 00:00000000 00000000   109        0 14360576 1 00000000da2422a3 24 0 0 10 -1
  56: 0100870A:97BA 81FA870A:022A 01 00000000:00000000 02:000002D6 00000000     0        0 38225624 2 0000000055876681 24 4 24 10 -1
  57: 0100870A:8D26 4EFC870A:022A 01 00000000:00000000 02:000005B6 00000000     0        0 38221552 2 00000000839e66b2 24 4 30 10 -1
  58: 4C3AF957:006F 4C9EC0D4:C640 01 00000000:00000000 00:00000000 00000000   109        0 18028279 1 00000000fcf7f23c 22 0 0 10 -1
  59: 4C3AF957:0016 06D7F957:CF10 01 00000000:00000000 02:00014F52 00000000     0        0 29690887 2 00000000ee5080be 24 4 28 10 30
  60: 4C3AF957:006F 459EC0D4:E9A3 01 00000000:00000000 00:00000000 00000000   109        0 14523621 1 0000000071b100f7 22 0 0 10 -1
  61: 0100870A:ADDE E0FC870A:022A 06 00000000:00000000 03:000014BC 00000000     0        0 0 3 00000000eecb1446
  62: 0100870A:DC12 94FC870A:022A 01 00000000:00000000 02:00000503 00000000     0        0 38163830 2 000000003b58f934 24 4 30 10 -1
  63: 4C3AF957:006F ACA84F2D:EB80 01 00000000:00000000 00:00000000 00000000   109        0 14738165 1 0000000097946988 50 4 30 10 -1
  64: 0100870A:DA12 9CFC870A:022A 01 00000000:00000000 02:00000549 00000000     0        0 38190246 2 00000000f36379b6 23 4 30 10 -1
  65: 4C3AF957:006F 489EC0D4:B828 01 00000000:00000000 00:00000000 00000000   109        0 16668714 1 0000000071500100 24 0 0 10 -1
  66: 4C3AF957:280A 128BE75E:A7E2 06 00000000:00000000 03:00000770 00000000     0        0 0 3 00000000f98f94a3
  67: 0100870A:8782 CDFA870A:022A 01 00000000:00000000 02:00000336 00000000     0        0 38214596 3 000000000a752c38 33 4 30 10 -1
  68: 4C3AF957:006F 3567DF4D:98C6 01 00000000:00000000 00:00000000 00000000   109        0 13061048 1 00000000fcfbbbb6 22 0 0 10 -1
  69: 4C3AF957:006F 2E384AC6:B721 01 00000000:00000000 00:00000000 00000000   109        0 14741764 1 000000002c9e18a5 51 4 30 10 -1
  70: 4C3AF957:006F 4C9EC0D4:C642 01 00000000:00000000 00:00000000 00000000   109        0 18028278 1 00000000dc73abfc 21 0 0 10 -1
  71: 4C3AF957:006F 499EC0D4:B22F 01 00000000:00000000 00:00000000 00000000   109        0 15103546 1 00000000286b873b 21 0 0 10 -1
  72: 0100007F:13AE 0100007F:90DE 06 00000000:00000000 03:0000011E 00000000     0        0 0 3 00000000c32805de
  73: 4C3AF957:006F 3245AF55:8C2F 01 00000000:00000000 00:00000000 00000000   109        0 14360575 1 0000000039730c93 24 0 0 10 -1
  74: 0100870A:EC7E 0AFA870A:022A 01 00000000:00000000 02:00000543 00000000     0        0 38228870 2 0000000050a97ad1 24 4 24 10 -1
  75: 4C3AF957:280B 128BE75E:C5E0 06 00000000:00000000 03:000009B2 00000000     0        0 0 3 0000000039e0ae67
  76: 4C3AF957:006F 3567DF4D:D511 01 00000000:00000000 00:00000000 00000000   109        0 364517 1 00000000e9894b00 22 0 0 10 -1
  77: 4C3AF957:006F ACA84F2D:EB84 01 00000000:00000000 00:00000000 00000000   109        0 14743620 1 00000000233e1d93 49 4 30 10 -1
  78: 4C3AF957:006F 4A9EC0D4:986B 01 00000000:00000000 00:00000000 00000000   109        0 14548243 1 000000004d11b661 22 0 0 10 -1
  79: 4C3AF957:006F 086D212D:B2B5 01 00000000:00000000 00:00000000 00000000   109        0 33830030 1 00000000be415995 58 4 30 10 -1
  80: 4C3AF957:006F 946886B7:BD3B 01 00000000:00000000 00:00000000 00000000   109        0 2282471 1 0000000028b46fcc 76 4 31 10 -1
  81: 4C3AF957:006F 7A44D0B0:C186 01 00000000:00000000 00:00000000 00000000   109        0 15257654 1 0000000051874008 46 0 0 10 -1
  82: 4C3AF957:006F 479EC0D4:9BBD 01 00000000:00000000 00:00000000 00000000   109        0 14405379 1 000000007dca56f0 22 0 0 10 -1
  83: 4C3AF957:006F 489EC0D4:9BE7 01 00000000:00000000 00:00000000 00000000   109        0 35447379 1 000000009402d2ef 22 0 0 10 -1

# grep . /proc/sys/net/ipv4/tcp*
/proc/sys/net/ipv4/tcp_abort_on_overflow:0
/proc/sys/net/ipv4/tcp_adv_win_scale:1
/proc/sys/net/ipv4/tcp_allowed_congestion_control:reno cubic
/proc/sys/net/ipv4/tcp_app_win:31
/proc/sys/net/ipv4/tcp_autocorking:1
/proc/sys/net/ipv4/tcp_available_congestion_control:reno cubic
/proc/sys/net/ipv4/tcp_base_mss:1024
/proc/sys/net/ipv4/tcp_challenge_ack_limit:1000
/proc/sys/net/ipv4/tcp_comp_sack_delay_ns:1000000
/proc/sys/net/ipv4/tcp_comp_sack_nr:44
/proc/sys/net/ipv4/tcp_congestion_control:cubic
/proc/sys/net/ipv4/tcp_dsack:1
/proc/sys/net/ipv4/tcp_early_demux:1
/proc/sys/net/ipv4/tcp_early_retrans:3
/proc/sys/net/ipv4/tcp_ecn:2
/proc/sys/net/ipv4/tcp_ecn_fallback:1
/proc/sys/net/ipv4/tcp_fack:0
/proc/sys/net/ipv4/tcp_fastopen:1
/proc/sys/net/ipv4/tcp_fastopen_blackhole_timeout_sec:3600
/proc/sys/net/ipv4/tcp_fastopen_key:06af0aa5-a23a74c1-ecfd9d92-93098b2d
/proc/sys/net/ipv4/tcp_fin_timeout:60
/proc/sys/net/ipv4/tcp_frto:2
/proc/sys/net/ipv4/tcp_fwmark_accept:0
/proc/sys/net/ipv4/tcp_invalid_ratelimit:500
/proc/sys/net/ipv4/tcp_keepalive_intvl:75
/proc/sys/net/ipv4/tcp_keepalive_probes:9
/proc/sys/net/ipv4/tcp_keepalive_time:7200
/proc/sys/net/ipv4/tcp_l3mdev_accept:0
/proc/sys/net/ipv4/tcp_limit_output_bytes:262144
/proc/sys/net/ipv4/tcp_low_latency:0
/proc/sys/net/ipv4/tcp_max_orphans:131072
/proc/sys/net/ipv4/tcp_max_reordering:300
/proc/sys/net/ipv4/tcp_max_syn_backlog:1024
/proc/sys/net/ipv4/tcp_max_tw_buckets:131072
/proc/sys/net/ipv4/tcp_mem:8192 131072  16777216
/proc/sys/net/ipv4/tcp_min_rtt_wlen:300
/proc/sys/net/ipv4/tcp_min_snd_mss:48
/proc/sys/net/ipv4/tcp_min_tso_segs:2
/proc/sys/net/ipv4/tcp_moderate_rcvbuf:1
/proc/sys/net/ipv4/tcp_mtu_probing:0
/proc/sys/net/ipv4/tcp_no_metrics_save:0
/proc/sys/net/ipv4/tcp_notsent_lowat:4294967295
/proc/sys/net/ipv4/tcp_orphan_retries:0
/proc/sys/net/ipv4/tcp_pacing_ca_ratio:120
/proc/sys/net/ipv4/tcp_pacing_ss_ratio:200
/proc/sys/net/ipv4/tcp_probe_interval:600
/proc/sys/net/ipv4/tcp_probe_threshold:8
/proc/sys/net/ipv4/tcp_recovery:1
/proc/sys/net/ipv4/tcp_reordering:3
/proc/sys/net/ipv4/tcp_retrans_collapse:1
/proc/sys/net/ipv4/tcp_retries1:3
/proc/sys/net/ipv4/tcp_retries2:15
/proc/sys/net/ipv4/tcp_rfc1337:0
/proc/sys/net/ipv4/tcp_rmem:8192        131072  16777216
/proc/sys/net/ipv4/tcp_sack:1
/proc/sys/net/ipv4/tcp_slow_start_after_idle:1
/proc/sys/net/ipv4/tcp_stdurg:0
/proc/sys/net/ipv4/tcp_synack_retries:5
/proc/sys/net/ipv4/tcp_syncookies:1
/proc/sys/net/ipv4/tcp_syn_retries:6
/proc/sys/net/ipv4/tcp_thin_linear_timeouts:0
/proc/sys/net/ipv4/tcp_timestamps:1
/proc/sys/net/ipv4/tcp_tso_win_divisor:3
/proc/sys/net/ipv4/tcp_tw_reuse:2
/proc/sys/net/ipv4/tcp_window_scaling:1
/proc/sys/net/ipv4/tcp_wmem:8192        131072  16777216
/proc/sys/net/ipv4/tcp_workaround_signed_windows:0

iptraf on an interface that deals only with video proxies

Did you attach a network dump?

yes

on proxy (iphost) with mediamtx 10502 to my vlc 60515 network card MTU 1500

sudo tcpdump -nn host iphost and not port 22

14:27:24.959811 IP .10502 > .60515: Flags [P.], seq 147560:148942, ack 57, win 126, length 1382
14:27:24.959852 IP .10502 > .60515: Flags [P.], seq 148942:150306, ack 57, win 126, length 1364
14:27:24.959889 IP .10502 > .60515: Flags [P.], seq 150306:151873, ack 57, win 126, length 1567
14:27:24.959916 IP .10502 > .60515: Flags [P.], seq 151873:152044, ack 57, win 126, length 171
14:27:24.959944 IP .10502 > .60515: Flags [P.], seq 152044:153494, ack 57, win 126, length 1450
14:27:24.959992 IP .10502 > .60515: Flags [P.], seq 153494:155555, ack 57, win 126, length 2061
14:27:24.960060 IP .10502 > .60515: Flags [.], seq 155555:158475, ack 57, win 126, length 2920
14:27:24.996270 IP .60515 > .10502: Flags [.], ack 150306, win 65335, length 0
14:27:24.996307 IP .10502 > .60515: Flags [.], seq 158475:161395, ack 57, win 126, length 2920
14:27:24.996322 IP .10502 > .60515: Flags [.], seq 161395:164315, ack 57, win 126, length 2920
14:27:24.997506 IP .60515 > .10502: Flags [.], ack 153494, win 65335, length 0
14:27:24.997542 IP .10502 > .60515: Flags [.], seq 164315:167235, ack 57, win 126, length 2920
14:27:24.997563 IP .10502 > .60515: Flags [.], seq 167235:170155, ack 57, win 126, length 2920

wolverin-a commented 2 months ago

I looked at the statistics through ffprobe: data comes directly from the camera evenly, through the mediamtx with pauses and accelerations

wolverin-a commented 2 months ago

since the situation is hopeless, I decided to post rtsp links, maybe this will help

From the camer over Nat-router rtsp://94.31.247.14:5541/cam

From the mediamtx rtsp://593358:Zlbl9NMh96@87.249.58.76:10502/02c00081491be26a

wolverin-a commented 1 month ago

updated to version 1.8.1, for which "h264: support 4K videos encoded with tune=zerolatency" and "various performance improvements" are specified, but there is no difference

jpetazzo commented 1 week ago

Hi! I'm experiencing similar issues with version 1.8.3 on amd64 running in a Docker container.

I'm pushing 4 video streams from ffmpeg to mediamtx, using RTSP in TCP mode, and consuming the streams over HLS and WebRTC.

Interestingly, I'm pushing from localhost in this test, so I'm really wondering where the packets get lost 😅

I'll keep testing and hopefully have more to report later.

mediamtx-1  | 2024/06/19 07:42:43 INF [RTSP] [conn [::1]:55070] opened
mediamtx-1  | 2024/06/19 07:42:43 INF [RTSP] [conn [::1]:55078] opened
mediamtx-1  | 2024/06/19 07:42:43 INF [RTSP] [conn [::1]:55068] opened
mediamtx-1  | 2024/06/19 07:42:43 INF [RTSP] [conn [::1]:55080] opened
mediamtx-1  | 2024/06/19 07:42:43 INF [RTSP] [session 72bb2d3a] created by [::1]:55068
mediamtx-1  | 2024/06/19 07:42:43 INF [RTSP] [session 1e6c8c87] created by [::1]:55080
mediamtx-1  | 2024/06/19 07:42:43 INF [RTSP] [session 675053b5] created by [::1]:55070
mediamtx-1  | 2024/06/19 07:42:43 INF [RTSP] [session 72bb2d3a] is publishing to path 'live/stream1', 2 tracks (Opus, H264)
mediamtx-1  | 2024/06/19 07:42:43 INF [HLS] [muxer live/stream1] created automatically
mediamtx-1  | 2024/06/19 07:42:43 INF [HLS] [muxer live/stream1] is converting into HLS, 2 tracks (Opus, H264)
mediamtx-1  | 2024/06/19 07:42:43 INF [RTSP] [session 1e6c8c87] is publishing to path 'live/stream3', 2 tracks (Opus, H264)
mediamtx-1  | 2024/06/19 07:42:43 INF [HLS] [muxer live/stream3] created automatically
mediamtx-1  | 2024/06/19 07:42:43 INF [HLS] [muxer live/stream3] is converting into HLS, 2 tracks (Opus, H264)
mediamtx-1  | 2024/06/19 07:42:43 INF [RTSP] [session 675053b5] is publishing to path 'live/stream2', 2 tracks (Opus, H264)
mediamtx-1  | 2024/06/19 07:42:43 INF [HLS] [muxer live/stream2] created automatically
mediamtx-1  | 2024/06/19 07:42:43 INF [HLS] [muxer live/stream2] is converting into HLS, 2 tracks (Opus, H264)
mediamtx-1  | 2024/06/19 07:42:43 INF [RTSP] [session d6695592] created by [::1]:55078
mediamtx-1  | 2024/06/19 07:42:43 INF [RTSP] [session d6695592] is publishing to path 'live/stream4', 2 tracks (Opus, H264)
mediamtx-1  | 2024/06/19 07:42:43 INF [HLS] [muxer live/stream4] created automatically
mediamtx-1  | 2024/06/19 07:42:43 INF [HLS] [muxer live/stream4] is converting into HLS, 2 tracks (Opus, H264)
mediamtx-1  | 2024/06/19 07:42:45 INF [WebRTC] [session e2f829b2] created by 127.0.0.1:59868
mediamtx-1  | 2024/06/19 07:42:45 INF [WebRTC] [session e2f829b2] peer connection established, local candidate: host/udp/192.168.39.1/8189, remote candidate: prflx/udp/10.0.0.23/52514
mediamtx-1  | 2024/06/19 07:42:45 INF [WebRTC] [session e2f829b2] is reading from path 'live/stream1', 2 tracks (H264, Opus)
mediamtx-1  | 2024/06/19 07:44:48 WAR [RTSP] [session 72bb2d3a] 11 RTP packets lost
mediamtx-1  | 2024/06/19 07:44:48 WAR [RTSP] [session 675053b5] 8 RTP packets lost
mediamtx-1  | 2024/06/19 07:45:07 WAR [RTSP] [session 72bb2d3a] 5 RTP packets lost
mediamtx-1  | 2024/06/19 07:45:07 WAR [RTSP] [session 675053b5] 3 RTP packets lost
mediamtx-1  | 2024/06/19 07:45:08 WAR [RTSP] [session 1e6c8c87] 3 RTP packets lost
mediamtx-1  | 2024/06/19 07:48:17 WAR [RTSP] [session 72bb2d3a] 30 RTP packets lost
mediamtx-1  | 2024/06/19 07:48:17 WAR [RTSP] [session 675053b5] 22 RTP packets lost
mediamtx-1  | 2024/06/19 07:48:18 WAR [RTSP] [session 1e6c8c87] 1 RTP packet lost
mediamtx-1  | 2024/06/19 07:48:29 WAR [RTSP] [session 72bb2d3a] 3 RTP packets lost
mediamtx-1  | 2024/06/19 07:48:30 WAR [RTSP] [session 1e6c8c87] 2 RTP packets lost
mediamtx-1  | 2024/06/19 07:48:31 WAR [RTSP] [session 72bb2d3a] 7 RTP packets lost
mediamtx-1  | 2024/06/19 07:48:32 WAR [RTSP] [session 675053b5] 8 RTP packets lost
mediamtx-1  | 2024/06/19 07:48:32 WAR [RTSP] [session 1e6c8c87] 6 RTP packets lost
mediamtx-1  | 2024/06/19 07:48:33 WAR [RTSP] [session 72bb2d3a] 11 RTP packets lost
mediamtx-1  | 2024/06/19 07:48:34 WAR [RTSP] [session 675053b5] 9 RTP packets lost
mediamtx-1  | 2024/06/19 07:48:34 WAR [RTSP] [session 1e6c8c87] 7 RTP packets lost
mediamtx-1  | 2024/06/19 07:48:37 WAR [RTSP] [session 72bb2d3a] 1 RTP packet lost
mediamtx-1  | 2024/06/19 07:48:38 WAR [RTSP] [session 675053b5] 1 RTP packet lost
mediamtx-1  | 2024/06/19 07:48:53 WAR [RTSP] [session 72bb2d3a] 1 RTP packet lost
mediamtx-1  | 2024/06/19 07:48:54 WAR [RTSP] [session 675053b5] 1 RTP packet lost
mediamtx-1  | 2024/06/19 07:48:54 WAR [RTSP] [session 1e6c8c87] 1 RTP packet lost
mediamtx-1  | 2024/06/19 07:50:57 WAR [RTSP] [session 72bb2d3a] 65 RTP packets lost
mediamtx-1  | 2024/06/19 07:50:57 WAR [RTSP] [session 675053b5] 17 RTP packets lost
mediamtx-1  | 2024/06/19 07:51:33 WAR [RTSP] [session 72bb2d3a] 10 RTP packets lost
mediamtx-1  | 2024/06/19 07:52:02 WAR [RTSP] [session 72bb2d3a] 24 RTP packets lost
mediamtx-1  | 2024/06/19 07:54:13 WAR [RTSP] [session 1e6c8c87] 16 RTP packets lost
mediamtx-1  | 2024/06/19 07:54:13 WAR [RTSP] [session 72bb2d3a] 10 RTP packets lost
mediamtx-1  | 2024/06/19 07:54:13 WAR [RTSP] [session 675053b5] 10 RTP packets lost
mediamtx-1  | 2024/06/19 07:54:14 WAR [RTSP] [session d6695592] 12 RTP packets lost
mediamtx-1  | 2024/06/19 07:57:14 WAR [RTSP] [session 72bb2d3a] 18 RTP packets lost
mediamtx-1  | 2024/06/19 07:57:14 WAR [RTSP] [session 675053b5] 20 RTP packets lost
mediamtx-1  | 2024/06/19 07:57:15 WAR [RTSP] [session 1e6c8c87] 3 RTP packets lost
mediamtx-1  | 2024/06/19 07:57:15 WAR [RTSP] [session d6695592] 13 RTP packets lost
mediamtx-1  | 2024/06/19 07:59:08 WAR [RTSP] [session 72bb2d3a] 13 RTP packets lost
mediamtx-1  | 2024/06/19 07:59:11 WAR [RTSP] [session 72bb2d3a] 15 RTP packets lost
mediamtx-1  | 2024/06/19 07:59:28 WAR [RTSP] [session 72bb2d3a] 71 RTP packets lost

moenodedev commented 1 week ago

We see the same issue with RTSP proxy on version 1.8.3. We would provide logs but they only say "RTP packets lost"

jpetazzo commented 5 days ago

I've done a bit of digging. I was wondering where that error message came from. It seems to be generated in gortsplib, more specifically here - which surprised me, because this seems to be when using UDP, and I thought I was using TCP.

It turns out that I was using the wrong ffmpeg flags (I'm using tee, and I therefore had to specify the rtsp_transport=tcp flag within the tee output block).

Now that I'm using TCP, the issue disappeared and I don't have frame corruption and hiccups anymore.

So if you can switch to TCP, I obviously recommend it, it'll fix that issue :)

If you have to use UDP, unfortunately I haven't found a fix, sorry!

jpetazzo commented 5 days ago

... I wanted to try and help others, so I switched back to UDP to reproduce the issue.

Then, I identified the socket to which the RTP packets were sent (with tcpdump -peni any udp - warning, this generates a lot of output). Apparently it's on UDP port 8000:

17:42:04.704718 lo    In  ifindex 1 00:00:00:00:00:00 ethertype IPv6 (0x86dd), length 1540: ::1.22014 > ::1.8000: UDP, length 1472
17:42:04.704727 lo    In  ifindex 1 00:00:00:00:00:00 ethertype IPv6 (0x86dd), length 1540: ::1.26900 > ::1.8000: UDP, length 1472
17:42:04.704727 lo    In  ifindex 1 00:00:00:00:00:00 ethertype IPv6 (0x86dd), length 1540: ::1.22014 > ::1.8000: UDP, length 1472
17:42:04.704734 lo    In  ifindex 1 00:00:00:00:00:00 ethertype IPv6 (0x86dd), length 1540: ::1.22014 > ::1.8000: UDP, length 1472
17:42:04.704736 lo    In  ifindex 1 00:00:00:00:00:00 ethertype IPv6 (0x86dd), length 1540: ::1.26900 > ::1.8000: UDP, length 1472

I tried to use ss to find that socket; unfortunately it didn't work (perhaps because I'm running MediaMTX in a container). Instead, I checked /proc/net/udp (no match) and /proc/net/udp6 (match!). This file has port numbers in hexadecimal; port 8000 is 1F40 and indeed I had a line like that one:

$ grep 1F40 /proc/net/udp6
10893: 00000000000000000000000000000000:1F40 00000000000000000000000000000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 28483833 2 0000000039f8d5d2 4398

The last value is the number of drops (probably because the receive buffer wasn't big enough).

I checked the value of net.core.rmem_max (the default value for all receive buffers of all kinds) and it was 256 KiB. I increased it to 100MB with sysctl net.core.rmem_max=100000000, restarted MediaMTX, and then the problem was gone - no more RTP packet lost.

Certainly, a 100MB receive buffer might be a bit excessive, but I wanted to check if this would help or not - and clearly, it did!

Don't forget to restart MediaMTX after increasing the sysctl value, and of course, make sure that you set that sysctl value permanently so that it persists across reboots.

That being said, it's likely that the issue reported by @wolverin-a is different, alas (since their receive buffer was already 16 MiB apparently).

I hope that will still help others!

bluenviron / mediamtx