search

ossrs / srs

SRS is a simple, high-efficiency, real-time video server supporting RTMP, WebRTC, HLS, HTTP-FLV, SRT, MPEG-DASH, and GB28181.
https://ossrs.io
MIT License
24.86k stars 5.29k forks source link

WebRTC: SRS 5 docker crash when RTMP2RTC over TCP. #3784

Closed namnm closed 1 month ago

namnm commented 10 months ago

Note: Please read FAQ before file an issue, see #2716

Description

While I was testing with rtmp to webrtc, the webrtc stream worked well. Then I refreshed the browser, and srs crashed

  1. SRS Version: 5

  2. SRS Log:

[2023-08-26 02:12:23.990][INFO][1][85uf3e42] HTTP #0 172.19.0.1:43648 GET http://localhost:8080/players/rtc_player.html?autostart=true, content-length=-1
[2023-08-26 02:12:23.991][INFO][1][85uf3e42] http match file=./objs/nginx/html/players/rtc_player.html, pattern=/, upath=/players/rtc_player.html
[2023-08-26 02:12:23.993][INFO][1][85uf3e42] TCP: before dispose resource(HttpConn)(0x60700005e7b0), conns=3, zombies=0, ign=0, inz=0, ind=0
[2023-08-26 02:12:23.993][WARN][1][85uf3e42][104] client disconnect peer. ret=1007
[2023-08-26 02:12:23.993][INFO][1][95fw2y11] TCP: clear zombies=1 resources, conns=3, removing=0, unsubs=0
[2023-08-26 02:12:23.993][INFO][1][85uf3e42] TCP: disposing #0 resource(HttpConn)(0x60700005e7b0), conns=3, disposing=1, zombies=0
[2023-08-26 02:12:24.004][INFO][1][5114w0fh] DTLS: After done, got 39 bytes
[2023-08-26 02:12:24.004][INFO][1][5114w0fh] DTLS: State Passive RECV, done=1, arq=0, r0=39, len=39, cnt=21, size=26, hs=0
[2023-08-26 02:12:24.005][WARN][1][5114w0fh][0] DTLS: SSL3 alert method=read type=warning, desc=CN(close notify), where=16388, ret=256, r1=0
[2023-08-26 02:12:24.005][INFO][1][5114w0fh] RTC: session destroy by DTLS alert(warning CN), username=p81x7808:tctP
[2023-08-26 02:12:24.005][INFO][1][5114w0fh] RTC: before dispose resource(RtcConn)(0x61d000194680), conns=1, zombies=0, ign=0, inz=0, ind=0
[2023-08-26 02:12:24.005][INFO][1][5114w0fh] RTC: session detach from [5114w0fh](RtcConn), disposing=1
[2023-08-26 02:12:24.005][INFO][1][5114w0fh] RTC: tcp conn diposing, because of rtc connection
[2023-08-26 02:12:24.005][INFO][1][396m2q82] TCP: before dispose resource(Tcp)(0x60c0000a4b00), conns=2, zombies=0, ign=0, inz=0, ind=0
[2023-08-26 02:12:24.005][ERROR][1][5114w0fh][0] serve error code=1070(StThreadInterrupt)(ST thread is interrupted) : rtc tcp conn : interrupted
thread [1][5114w0fh]: do_cycle() [./src/app/srs_app_rtc_network.cpp:811][errno=0]
thread [1][5114w0fh]: interrupt() [./src/app/srs_app_st.cpp:257][errno=0]
[2023-08-26 02:12:24.005][INFO][1][e0i043rg] RTC: clear zombies=1 resources, conns=1, removing=0, unsubs=2
[2023-08-26 02:12:24.005][INFO][1][5114w0fh] RTC: disposing #0 resource(RtcConn)(0x61d000194680), conns=1, disposing=1, zombies=0
[2023-08-26 02:12:24.005][INFO][1][95fw2y11] TCP: clear zombies=1 resources, conns=2, removing=0, unsubs=0
[2023-08-26 02:12:24.005][INFO][1][396m2q82] TCP: disposing #0 resource(Tcp)(0x60c0000a4b00), conns=2, disposing=1, zombies=0
=================================================================
==1==ERROR: AddressSanitizer: heap-use-after-free on address 0x6040013a9358 at pc 0x5646a13fec4f bp 0x7f57effffd60 sp 0x7f57effffd50
READ of size 8 at 0x6040013a9358 thread T1 (srs-hybrid-2)
    #0 0x5646a13fec4e in SrsRtcTcpNetwork::write(void*, unsigned long, long*) src/app/srs_app_rtc_network.cpp:672
    #1 0x5646a1389e3b in SrsRtcConnection::do_send_packet(SrsRtpPacket*) src/app/srs_app_rtc_conn.cpp:2472
    #2 0x5646a143e0c2 in SrsRtcAudioSendTrack::on_rtp(SrsRtpPacket*) src/app/srs_app_rtc_source.cpp:2804
    #3 0x5646a1374461 in SrsRtcPlayStream::send_packet(SrsRtpPacket*&) src/app/srs_app_rtc_conn.cpp:734
    #4 0x5646a1373254 in SrsRtcPlayStream::cycle() src/app/srs_app_rtc_conn.cpp:669
    #5 0x5646a10f5e93 in SrsFastCoroutine::cycle() src/app/srs_app_st.cpp:285
    #6 0x5646a10f5fe3 in SrsFastCoroutine::pfn(void*) src/app/srs_app_st.cpp:300
    #7 0x5646a14a70c9 in _st_thread_main /srs/trunk/objs/Platform-SRS5-Linux-5.15.0-GCC9.4.0-x86_64/st-srs/sched.c:380
    #8 0x5646a14a79ef in st_thread_create /srs/trunk/objs/Platform-SRS5-Linux-5.15.0-GCC9.4.0-x86_64/st-srs/sched.c:666

0x6040013a9358 is located 8 bytes inside of 48-byte region [0x6040013a9350,0x6040013a9380)
freed by thread T1 (srs-hybrid-2) here:
    #0 0x5646a0ddbf6f in operator delete(void*) (/usr/local/srs/objs/srs+0x4aef6f)

previously allocated by thread T1 (srs-hybrid-2) here:
    #0 0x5646a0ddafd7 in operator new(unsigned long) (/usr/local/srs/objs/srs+0x4adfd7)

Thread T1 (srs-hybrid-2) created by T0 here:
    #0 0x5646a0d06295 in pthread_create (/usr/local/srs/objs/srs+0x3d9295)
    #1 0x5646a1333697 in SrsThreadPool::execute(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, SrsCplxError* (*)(void*), void*) src/app/srs_app_threads.cpp:671
    #2 0x5646a14a59fe in run_in_thread_pool() src/main/srs_main_server.cpp:517
    #3 0x5646a14a5466 in run_directly_or_daemon() src/main/srs_main_server.cpp:456
    #4 0x5646a14a25c2 in do_main(int, char**, char**) src/main/srs_main_server.cpp:245
    #5 0x5646a14a28dd in main src/main/srs_main_server.cpp:256
    #6 0x7f57f7aa5082 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x24082)

SUMMARY: AddressSanitizer: heap-use-after-free src/app/srs_app_rtc_network.cpp:672 in SrsRtcTcpNetwork::write(void*, unsigned long, long*)
Shadow bytes around the buggy address:
  0x0c088026d210: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fd
  0x0c088026d220: fa fa fd fd fd fd fd fa fa fa 00 00 00 00 00 fa
  0x0c088026d230: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c088026d240: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c088026d250: fa fa fa fa fa fa fa fa fa fa fd fd fd fa fa fa
=>0x0c088026d260: fa fa fd fd fd fa fa fa fa fa fd[fd]fd fd fd fd
  0x0c088026d270: fa fa fd fd fd fa fa fa fa fa fa fa fa fa fa fa
  0x0c088026d280: fa fa fa fa fa fa fa fa fa fa fd fd fd fd fd fa
  0x0c088026d290: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fd
  0x0c088026d2a0: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fa
  0x0c088026d2b0: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
[2023-08-26 02:12:24.054][ERROR][1][5114w0fh][0] =================================================================
[2023-08-26 02:12:24.054][ERROR][1][5114w0fh][0] ==1==ERROR: AddressSanitizer: heap-use-after-free on address 0x6040013a9358 at pc 0x5646a13fec4f bp 0x7f57effffd60 sp 0x7f57effffd50
[2023-08-26 02:12:24.054][ERROR][1][5114w0fh][0] READ of size 8 at 0x6040013a9358 thread T1 (srs-hybrid-2)
[2023-08-26 02:12:24.054][ERROR][1][5114w0fh][0]     #0 0x5646a13fec4e in SrsRtcTcpNetwork::write(void*, unsigned long, long*) src/app/srs_app_rtc_network.cpp:672, r0=1093
[2023-08-26 02:12:24.054][ERROR][1][5114w0fh][0]     #1 0x5646a1389e3b in SrsRtcConnection::do_send_packet(SrsRtpPacket*) src/app/srs_app_rtc_conn.cpp:2472, r0=1093
[2023-08-26 02:12:24.054][ERROR][1][5114w0fh][0]     #2 0x5646a143e0c2 in SrsRtcAudioSendTrack::on_rtp(SrsRtpPacket*) src/app/srs_app_rtc_source.cpp:2804, r0=1093
[2023-08-26 02:12:24.054][ERROR][1][5114w0fh][0]     #3 0x5646a1374461 in SrsRtcPlayStream::send_packet(SrsRtpPacket*&) src/app/srs_app_rtc_conn.cpp:734, r0=1093
[2023-08-26 02:12:24.054][ERROR][1][5114w0fh][0]     #4 0x5646a1373254 in SrsRtcPlayStream::cycle() src/app/srs_app_rtc_conn.cpp:669, r0=1093
[2023-08-26 02:12:24.054][ERROR][1][5114w0fh][0]     #5 0x5646a10f5e93 in SrsFastCoroutine::cycle() src/app/srs_app_st.cpp:285, r0=1093
[2023-08-26 02:12:24.054][ERROR][1][5114w0fh][0]     #6 0x5646a10f5fe3 in SrsFastCoroutine::pfn(void*) src/app/srs_app_st.cpp:300, r0=1093
[2023-08-26 02:12:24.054][ERROR][1][5114w0fh][0]     #7 0x5646a14a70c9 in _st_thread_main /srs/trunk/objs/Platform-SRS5-Linux-5.15.0-GCC9.4.0-x86_64/st-srs/sched.c:380, r0=1093
[2023-08-26 02:12:24.054][ERROR][1][5114w0fh][0]     #8 0x5646a14a79ef in st_thread_create /srs/trunk/objs/Platform-SRS5-Linux-5.15.0-GCC9.4.0-x86_64/st-srs/sched.c:666, r0=1093
[2023-08-26 02:12:24.054][ERROR][1][5114w0fh][0] 0x6040013a9358 is located 8 bytes inside of 48-byte region [0x6040013a9350,0x6040013a9380)
[2023-08-26 02:12:24.054][ERROR][1][5114w0fh][0] freed by thread T1 (srs-hybrid-2) here:
sh: 1: addr2line: not found
[2023-08-26 02:12:24.057][ERROR][1][5114w0fh][0]     #0 0x5646a0ddbf6f in operator delete(void*) (/usr/local/srs/objs/srs+0x4aef6f), r0=1094
[2023-08-26 02:12:24.057][ERROR][1][5114w0fh][0] previously allocated by thread T1 (srs-hybrid-2) here:
[2023-08-26 02:12:24.057][ERROR][1][5114w0fh][0]     #0 0x5646a0ddafd7 in operator new(unsigned long) (/usr/local/srs/objs/srs+0x4adfd7), r0=1094
[2023-08-26 02:12:24.057][ERROR][1][5114w0fh][0] Thread T1 (srs-hybrid-2) created by T0 here:
sh: 1: addr2line: not found
sh: 1: addr2line: not found
[2023-08-26 02:12:24.058][ERROR][1][5114w0fh][0]     #0 0x5646a0d06295 in pthread_create (/usr/local/srs/objs/srs+0x3d9295), r0=1094
[2023-08-26 02:12:24.058][ERROR][1][5114w0fh][0]     #1 0x5646a1333697 in SrsThreadPool::execute(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, SrsCplxError* (*)(void*), void*) src/app/srs_app_threads.cpp:671, r0=1093
[2023-08-26 02:12:24.058][ERROR][1][5114w0fh][0]     #2 0x5646a14a59fe in run_in_thread_pool() src/main/srs_main_server.cpp:517, r0=1093
[2023-08-26 02:12:24.058][ERROR][1][5114w0fh][0]     #3 0x5646a14a5466 in run_directly_or_daemon() src/main/srs_main_server.cpp:456, r0=1093
[2023-08-26 02:12:24.058][ERROR][1][5114w0fh][0]     #4 0x5646a14a25c2 in do_main(int, char**, char**) src/main/srs_main_server.cpp:245, r0=1093
[2023-08-26 02:12:24.058][ERROR][1][5114w0fh][0]     #5 0x5646a14a28dd in main src/main/srs_main_server.cpp:256, r0=1093
sh: 1: addr2line: not found
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0]     #6 0x7f57f7aa5082 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x24082), r0=1094
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0] SUMMARY: AddressSanitizer: heap-use-after-free src/app/srs_app_rtc_network.cpp:672 in SrsRtcTcpNetwork::write(void*, unsigned long, long*)
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0] Shadow bytes around the buggy address:
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0]   0x0c088026d210: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fd
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0]   0x0c088026d220: fa fa fd fd fd fd fd fa fa fa 00 00 00 00 00 fa
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0]   0x0c088026d230: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0]   0x0c088026d240: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0]   0x0c088026d250: fa fa fa fa fa fa fa fa fa fa fd fd fd fa fa fa
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0] =>0x0c088026d260: fa fa fd fd fd fa fa fa fa fa fd[fd]fd fd fd fd
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0]   0x0c088026d270: fa fa fd fd fd fa fa fa fa fa fa fa fa fa fa fa
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0]   0x0c088026d280: fa fa fa fa fa fa fa fa fa fa fd fd fd fd fd fa
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0]   0x0c088026d290: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fd
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0]   0x0c088026d2a0: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fa
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0]   0x0c088026d2b0: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fa
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0] Shadow byte legend (one shadow byte represents 8 application bytes):
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0]   Addressable:           00
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0]   Partially addressable: 01 02 03 04 05 06 07 
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0]   Heap left redzone:       fa
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0]   Freed heap region:       fd
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0]   Stack left redzone:      f1
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0]   Stack mid redzone:       f2
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0]   Stack right redzone:     f3
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0]   Stack after return:      f5
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0]   Stack use after scope:   f8
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0]   Global redzone:          f9
==1==ABORTING
[2023-08-26 02:12:24.062][ERROR][1][5114w0fh][0]   Global init order:       f6
[2023-08-26 02:12:24.062][ERROR][1][5114w0fh][0]   Poisoned by user:        f7
[2023-08-26 02:12:24.062][ERROR][1][5114w0fh][0]   Container overflow:      fc
[2023-08-26 02:12:24.062][ERROR][1][5114w0fh][0]   Array cookie:            ac
[2023-08-26 02:12:24.062][ERROR][1][5114w0fh][0]   Intra object redzone:    bb
[2023-08-26 02:12:24.062][ERROR][1][5114w0fh][0]   ASan internal:           fe
[2023-08-26 02:12:24.062][ERROR][1][5114w0fh][0]   Left alloca redzone:     ca
[2023-08-26 02:12:24.062][ERROR][1][5114w0fh][0]   Right alloca redzone:    cb
[2023-08-26 02:12:24.062][ERROR][1][5114w0fh][0]   Shadow gap:              cc
  1. SRS Config:
listen 1935;
daemon off;
srs_log_tank console;
# srs_log_level warn;

srt_server {
  enabled on;
  listen 10080;
}

rtc_server {
  enabled on;
  protocol tcp;
  candidate 192.168.5.234;
  tcp {
    enabled on;
    listen 8000;
  }
}
http_api {
  enabled on;
  listen 1985;
  crossdomain on;
}
http_server {
  enabled on;
  listen 8080;
  dir ./objs/nginx/html;
}

vhost __defaultVhost__ {
  rtc {
    enabled on;
    rtmp_to_rtc on;
  }
  http_remux {
    enabled on;
    mount [vhost]/[app]/[stream].flv;
    hstrs on;
  }
}

Replay

While I was testing with rtmp to webrtc, the webrtc stream worked well. Then I refreshed the browser, and srs crashed

Expect

Do not crash

winlinvip commented 10 months ago

What browser are you using, what is your video source, does it always occur? If it does, could you please attach the video source file.

TRANS_BY_GPT4

namnm commented 10 months ago

I used latest GoogleChrome on macOS. It only occurs after refreshing the browser, and not always occur just rarely. About the video source, it happened on several videos I tried, so I guess the issue could be at the browser refreshing which caused an error in the connection. I can see some related info in the above log.

winlinvip commented 10 months ago

When the browser refreshes, it is possible to disconnect the TCP connection and then re-establish a new TCP connection. At this time, it is indeed possible to trigger boundary conditions.

Please spend more time researching under what conditions it is more likely to occur, such as whether it is specific video content, a specific browser, or a specific operation method.

TRANS_BY_GPT4

winlinvip commented 3 months ago

For #3505 SRS crashed at closing a active fd:

void srs_close_stfd(srs_netfd_t& stfd)
{
    if (stfd) {
        // we must ensure the close is ok.
        int r0 = st_netfd_close((st_netfd_t)stfd);
        if (r0) {
            // By _st_epoll_fd_close or _st_kq_fd_close
            if (errno == EBUSY) srs_assert(!r0);
winlinvip commented 1 month ago

Fixed, see https://github.com/ossrs/srs/pull/4083