ossrs / srs

SRS is a simple, high-efficiency, real-time video server supporting RTMP, WebRTC, HLS, HTTP-FLV, SRT, MPEG-DASH, and GB28181.
https://ossrs.io
MIT License
25.16k stars 5.31k forks source link

SRT disconnect and memory leak issue in 4.0.228 #3614

Closed EvgIL74 closed 4 months ago

EvgIL74 commented 1 year ago

Note: Please read FAQ before file an issue, see #2716

Description

SRT server in version 4.0.228 disconnects the client immediately(not considered - connection timeout parameter) when the Larix broadcaster starts to lose packets in any Larix algorithm. I tried 5.0.b1, and the issue doesn't repeat at all. I can't use 5.0.b1 because the 5.0.b1 has performance and memory issue that it was reported 3 weeks ago. https://github.com/ossrs/srs/issues/3573

  1. SRS Version: 4.0.228

  2. SRS Log:

    [2023-07-04 19:12:14.908][Trace][262728][915x3j35][MAIN] SRS/4.0.269(Leo), MIT
    [2023-07-04 19:12:14.908][Trace][262728][915x3j35] authors: https://github.com/ossrs/srs/blob/4.0release/trunk/AUTHORS.txt
    [2023-07-04 19:12:14.908][Trace][262728][915x3j35] contributors: Winlin<winlin@vip.126.com> Winlin<winterserver@126.com> Winlin<chengli.ycl@alibaba-inc.com> Wenjie<zhaowenjie@tal.com> Wenjie<740936897@qq.com> Wenjie<wenjie.zhao@chinacache.com> xiangcheng.liu<liuxc0116@foxmail.com> naijia.liu<youngcow@youngcow.net> alcoholyi<alcoholyi@qq.com> byteman<wangchen2011@gmail.com> chad.wang<chad.wang.cn@gmail.com> suhetao<suhetao@gmail.com> Johnny<fengjihu@163.com> karthikeyan<keyanmca@gmail.com> StevenLiu<lq@chinaffmpeg.org> zhengfl<zhengfl_1989@126.com> tufang14<breadbean1449@gmail.com> allspace<allspace@gmail.com> niesongsong<nie950@gmail.com> rudeb0t<nimrod@themanxgroup.tw> CallMeNP<np.liamg@gmail.com> synote<synote@qq.com> lovecat<littlefawn@163.com> panda1986<542638787@qq.com> YueHonghui<hongf.yue@hotmail.com> ThomasDreibholz<dreibh@simula.no> JuntaoLiu<juntliu@gmail.com> RocFang<fangpeng1986@gmail.com> MakarovYaroslav<yaroslav.makarov.97@mail.ru> MirkoVelic<mvelic@inoxx.net> HuiZhang(huzhang2)<huzhang2@cisco.com> OtterWa<simpleotter23@gmail.com> walkermi<172192667@qq.com> haofz<fuzhuang.hao@vhall.com> ME_Kun_Han<hanvskun@hotmail.com> ljx0305<ljx0305@gmail.com> cenxinwei<censhanhe@163.com> StarBrilliant<m13253@hotmail.com> xubin<xubin@chnvideo.com> intliang<yintiliang@gmail.com> flowerwrong<sysuyangkang@gmail.com> YLX<568414379@qq.com> J<guotaojiang@qq.com> Harlan<hailiang@gvrcraft.com> hankun<hankun@bravovcloud.com> JonathanBarratt<jonathan.barratt@gmail.com> KeeganH<keeganwharris@gmail.com> StevenLiu<lingjiujianke@gmail.com> liuxc0116<liuxc0116@gmail.com> ChengdongZhang<lmajzcd@sina.com> lovacat<lovecat@china.sina.com> qiang.li<qiang.li@verycdn.com.cn> HungMingWu<u9089000@gmail.com> Himer<xishizhaohua@qq.com> XiaLixin<xialixin@kanzhun.com> XiaLixin<68469352@qq.com> XiaLixin<xlx0625@163.com> XiaLixin<xialx@yuntongxun.com> alphonsetai<tyh_123@163.com> Michael.Ma<wnpllr@gmail.com> lam2003<linmin3@yy.com> ShiWei<shiwei05@kuaishou.com> ShiWei<shi.weibd@hotmail.com> XiaofengWang<wasphin@gmail.com> XiaoZhihong<hondaxiao@tencent.com> XiaoZhihong<xiaozhihong8@gmail.com> XiaoZhihong<xiaozhihong@huya.com> yanghuiwen<cainiaodj@qq.com> WuPengqiang<309554135@qq.com> WuPengqiang<pengqiang.wpq@alibaba-inc.com> l<22312935+lam2003@github> xfalcon<x-falcon@github> ChenGuanghua<jinxue.cgh@alibaba-inc.com> ChenGuanghua<chengh_math@126.com> LiPeng<mozhan.lp@alibaba-inc.com> LiPeng<lipeng19811218@gmail.com> yajun18<yajun18@staff.sina.com.cn> liulichuan<liulichuan@kuaishou.com> yapingcat<caoyapingneu@163.com> chenchengbin<chenchengbin@yy.com> ChenHaibo<495810242@qq.com> ChenHaibo<nmgchenhaibo@foxmail.com> jasongwq<jasongwq@gmail.com> yinjiaoyuan<yinjiaoyuan@163.com> PieerePi<pihuibin@hotmail.com> JesseXi<jesse.jinjin@wo.cn> PieerePi<40780488+PieerePi@github> ghostsf<ghost_sf@163.com> xbpeng121<53243357+xbpeng121@github> johzzy<hellojinqiang@gmail.com> stone<bluestn@163.com> cfw11<34058899+cfw11@github> Hung-YiChen<gaod.chen@gmail.com> long<liyalong12345@126.com> matthew1838<77285055+matthew1838@github> rise<rise.worlds@outlook.com> 
    [2023-07-04 19:12:14.908][Trace][262728][915x3j35] cwd=/srs/trunk, work_dir=./, build: 2023-06-26 09:50:51, configure: --x86-x64 --srt=on, uname: Linux nginx-il-jerus-01 5.15.0-1037-oracle #43~20.04.1-Ubuntu SMP Tue Jun 6 20:51:15 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux, osx: 0, pkg: , region: , source: 
    [2023-07-04 19:12:14.908][Trace][262728][915x3j35] configure detail: --prefix=/usr/local/srs --config=conf/srs.conf --hls=on --hds=off --dvr=on --ssl=on --https=on --ssl-1-0=off --ssl-local=off --sys-ssl=off --transcode=on --ingest=on --stat=on --http-callback=on --http-server=on --stream-caster=on --http-api=on --utest=off --cherrypy=off --srt=on --rtc=on --simulator=off --cxx11=on --cxx14=off --ffmpeg-fit=on --nasm=on --srtp-nasm=on --clean=on --gperf=off --gmc=off --gmd=off --gmp=off --gcp=off --gprof=off --static=off --shared-st=off --shared-srt=off --shared-ffmpeg=off --log-verbose=off --log-info=off --log-trace=on --gcov=off --debug=off --debug-stats=off --cross-build=off --cc=gcc --cxx=g++ --ar=ar --ld=ld --randlib=randlib
    [2023-07-04 19:12:14.908][Trace][262728][915x3j35] srs checking config...
    [2023-07-04 19:12:14.908][Trace][262728][915x3j35] ips, iface[0] ens3 ipv4 0x11043 10.0.0.187, iface[1] ens3 ipv6 0x11043 fe80::17ff:fe00:1b1%ens3
    [2023-07-04 19:12:14.908][Trace][262728][915x3j35] devices, intranet ens3 10.0.0.187, intranet ens3 fe80::17ff:fe00:1b1%ens3
    [2023-07-04 19:12:14.908][Warn][262728][915x3j35][22] stats network use index=0, ip=10.0.0.187, ifname=ens3
    [2023-07-04 19:12:14.908][Warn][262728][915x3j35][22] stats disk not configed, disk iops disabled.
    [2023-07-04 19:12:14.908][Trace][262728][915x3j35] you can check log by: tail -n 30 -f ./objs/logs/srs.log
    [2023-07-04 19:12:14.908][Trace][262728][915x3j35] please check SRS by: ./etc/init.d/srs status
    [2023-07-04 19:12:14.908][Trace][262728][915x3j35] features, rch:on, dash:on, hls:on, hds:off, srt:on, hc:on, ha:on, hs:on, hp:on, dvr:on, trans:on, inge:on, stat:on, sc:on
    [2023-07-04 19:12:14.908][Trace][262728][915x3j35] SRS on  amd64 x86_64, conf:conf/srt2rtc.conf, limit:1000, writev:1024, encoding:little-endian, HZ:100
    [2023-07-04 19:12:14.908][Trace][262728][915x3j35] mw sleep:350ms. mr enabled:on, default:0, sleep:350ms
    [2023-07-04 19:12:14.908][Trace][262728][915x3j35] gc:on, pq:30000ms, cscc:[0,16), csa:on, tn:on(may hurts performance), ss:auto(guess by merged write)
    [2023-07-04 19:12:14.908][Trace][262728][915x3j35] system default latency(ms): mw(0-350) + mr(0-350) + play-queue(0-30000)
    [2023-07-04 19:12:14.910][Trace][262728][915x3j35] fingerprint=17:59:45:49:C7:C9:8B:D3:A2:1F:38:92:2E:6D:16:B2:7B:AD:A8:A5:79:5C:2A:D9:67:AA:54:B0:9A:2D:02:0C
    [2023-07-04 19:12:14.910][Trace][262728][915x3j35] CircuitBreaker: enabled=1, high=2x90, critical=1x95, dying=5x99
    [2023-07-04 19:12:14.910][Trace][262728][915x3j35] http flv live stream, vhost=__defaultVhost__, mount=[vhost]/[app]/[stream].flv
    [2023-07-04 19:12:14.910][Trace][262728][915x3j35] http: root mount to /mnt/hls
    [2023-07-04 19:12:14.910][Trace][262728][915x3j35] server main cid=915x3j35, pid=262728, ppid=262727, asprocess=0
    [2023-07-04 19:12:14.910][Trace][262728][915x3j35] write pid=262728 to ./objs/srs.pid success!
    [2023-07-04 19:12:14.910][Trace][262728][915x3j35] RTMP listen at tcp://0.0.0.0:1936, fd=8
    [2023-07-04 19:12:14.910][Trace][262728][915x3j35] HTTP-API listen at tcp://0.0.0.0:1985, fd=9
    [2023-07-04 19:12:14.910][Trace][262728][915x3j35] HTTPS-API listen at tcp://0.0.0.0:1986, fd=10
    [2023-07-04 19:12:14.910][Trace][262728][915x3j35] HTTP-Server listen at tcp://0.0.0.0:8080, fd=11
    [2023-07-04 19:12:14.910][Trace][262728][915x3j35] signal installed, reload=1, reopen=10, fast_quit=15, grace_quit=3
    [2023-07-04 19:12:14.910][Trace][262728][915x3j35] http: api mount /console to /mnt/hls/console
    [2023-07-04 19:12:14.910][Trace][262728][915x3j35] srt2rtmp start coroutine...
    [2023-07-04 19:12:14.911][Trace][262728][915x3j35] rtc listen at udp://0.0.0.0:8000, fd=14
    [2023-07-04 19:12:14.911][Trace][262728][60081xy6] Hybrid cpu=0.00%,15MB
    [2023-07-04 19:12:14.911][Warn][262728][14mr399r][22] use private address as ip: 10.0.0.187, ifname=ens3
    [2023-07-04 19:12:14.911][Trace][262728][14mr399r] Startup query id=vid-190n4a3, session=vid-1600y71, eip=10.0.0.187, wait=300s
    [2023-07-04 19:12:14.911][Trace][262728][443l8r4r] TCP: connection manager run, conns=0
    [2023-07-04 19:12:14.911][Trace][262728][6tt95789] srt server is enabled...
    [2023-07-04 19:12:14.911][Trace][262728][6tt95789] srt server listen port:9401
    [2023-07-04 19:12:14.911][Trace][262728][6tt95789] init srt parameter, maxbw:1000000000, mss:1500, tlpkdrop:1, connect timeout:4000, send buff:12058624, recv buff:12058624, payload size:1316, latency:0, recv latency:0, peer latency:0
    [2023-07-04 19:12:14.911][Trace][262728][6tt95789] srt server listen port=9401, server_fd=548790949
    [2023-07-04 19:12:14.911][Trace][262728][6tt95789] srt server is starting... port(9401)
    [2023-07-04 19:12:14.911][Trace][262728][6tt95789] srt server is working port(9401)
    [2023-07-04 19:12:14.911][Trace][262728][11367b80] UDP #14 LISTEN at 0.0.0.0:8000, SO_SNDBUF(default=212992, expect=10485760, actual=425984, r0=0), SO_RCVBUF(default=212992, expect=10485760, actual=425984, r0=0)
    [2023-07-04 19:12:14.911][Trace][262728][rs058021] RTC: connection manager run, conns=0
    [2023-07-04 19:12:18.215][Trace][262728][6tt95789] srt connect construct streamid:#!::h=live/GMILWP_Court101,m=publish, mode:2, subpath:live/GMILWP_Court101, vhost:__default_host__
    [2023-07-04 19:12:18.215][Trace][262728][6tt95789] new srt connection streamid:#!::h=live/GMILWP_Court101,m=publish, fd:548790948, vhost:__default_host__
    [2023-07-04 19:12:18.215][Trace][262728][6tt95789] srto SRTO_LATENCY=2000
    [2023-07-04 19:12:18.215][Trace][262728][6tt95789] srto SRTO_PEERLATENCY=120
    [2023-07-04 19:12:18.215][Trace][262728][6tt95789] srto SRTO_RCVLATENCY=2000
    [2023-07-04 19:12:18.215][Trace][262728][6tt95789] srto SRTO_SNDBUF=12058624
    [2023-07-04 19:12:18.215][Trace][262728][6tt95789] srto SRTO_RCVBUF=12058624
    [2023-07-04 19:12:18.215][Trace][262728][6tt95789] srto SRTO_MAXBW=1000000000
    [2023-07-04 19:12:18.215][Trace][262728][6tt95789] srt mix_correct is enable.
    [2023-07-04 19:12:18.215][Trace][262728][6tt95789] srt h264 sei filter is enable.
    [2023-07-04 19:12:18.215][Trace][262728][6tt95789] srt_handle add new pusher streamid:#!::h=live/GMILWP_Court101,m=publish, subpath:live/GMILWP_Court101, sid:live/GMILWP_Court101
    [2023-07-04 19:12:18.215][Trace][262728][6tt95789] new conn added fd:548790948, event:0x00000009
    [2023-07-04 19:12:18.276][Trace][262728][6tt95789] srt connect construct streamid:#!::h=live/GMILWP_Court102,m=publish, mode:2, subpath:live/GMILWP_Court102, vhost:__default_host__
    [2023-07-04 19:12:18.276][Trace][262728][6tt95789] new srt connection streamid:#!::h=live/GMILWP_Court102,m=publish, fd:548790947, vhost:__default_host__
    [2023-07-04 19:12:18.276][Trace][262728][6tt95789] srto SRTO_LATENCY=2000
    [2023-07-04 19:12:18.276][Trace][262728][6tt95789] srto SRTO_PEERLATENCY=120
    [2023-07-04 19:12:18.276][Trace][262728][6tt95789] srto SRTO_RCVLATENCY=2000
    [2023-07-04 19:12:18.276][Trace][262728][6tt95789] srto SRTO_SNDBUF=12058624
    [2023-07-04 19:12:18.276][Trace][262728][6tt95789] srto SRTO_RCVBUF=12058624
    [2023-07-04 19:12:18.276][Trace][262728][6tt95789] srto SRTO_MAXBW=1000000000
    [2023-07-04 19:12:18.276][Trace][262728][6tt95789] srt mix_correct is enable.
    [2023-07-04 19:12:18.276][Trace][262728][6tt95789] srt h264 sei filter is enable.
    [2023-07-04 19:12:18.276][Trace][262728][6tt95789] srt_handle add new pusher streamid:#!::h=live/GMILWP_Court102,m=publish, subpath:live/GMILWP_Court102, sid:live/GMILWP_Court102
    [2023-07-04 19:12:18.276][Trace][262728][6tt95789] new conn added fd:548790947, event:0x00000009
    [2023-07-04 19:12:19.907][Trace][262728][60081xy6] Hybrid cpu=1.00%,16MB
    [2023-07-04 19:12:20.254][Trace][262728][6tt95789] new rtmp client for srt upstream, key_path:live/GMILWP_Court101
    [2023-07-04 19:12:20.254][Trace][262728][6tt95789] rtmp client construct url:rtmp://127.0.0.1:1936/live/GMILWP_Court101?upstream=srt
    [2023-07-04 19:12:20.254][Trace][262728][48937a38] RTMP client ip=127.0.0.1:58474, fd=16
    [2023-07-04 19:12:20.255][Trace][262728][6tt95789] complex handshake success.
    [2023-07-04 19:12:20.255][Trace][262728][6tt95789] protocol in.buffer=0, in.ack=0, out.ack=0, in.chunk=128, out.chunk=128
    [2023-07-04 19:12:20.255][Trace][262728][48937a38] complex handshake success
    [2023-07-04 19:12:20.296][Trace][262728][48937a38] connect app, tcUrl=rtmp://127.0.0.1:1936/live, pageUrl=, swfUrl=, schema=rtmp, vhost=127.0.0.1, port=1936, app=live, args=(obj)
    [2023-07-04 19:12:20.296][Trace][262728][48937a38] edge-srs ip=10.0.0.187, version=4.0.269, pid=262728, id=0
    [2023-07-04 19:12:20.296][Trace][262728][48937a38] protocol in.buffer=0, in.ack=0, out.ack=0, in.chunk=128, out.chunk=128
    [2023-07-04 19:12:20.340][Trace][262728][6tt95789] connected, version=4.0.269.0, ip=127.0.0.1, pid=262728, id=0, dsu=1
    [2023-07-04 19:12:20.341][Trace][262728][6tt95789] new rtmp client for srt upstream, key_path:live/GMILWP_Court102
    [2023-07-04 19:12:20.341][Trace][262728][6tt95789] rtmp client construct url:rtmp://127.0.0.1:1936/live/GMILWP_Court102?upstream=srt
    [2023-07-04 19:12:20.341][Trace][262728][48937a38] client identified, type=flash-publish, vhost=127.0.0.1, app=live, stream=GMILWP_Court101, param=?upstream=srt&vhost=127.0.0.1, duration=0ms
    [2023-07-04 19:12:20.341][Trace][262728][48937a38] connected stream, tcUrl=rtmp://127.0.0.1:1936/live, pageUrl=, swfUrl=, schema=rtmp, vhost=__defaultVhost__, port=1936, app=live, stream=GMILWP_Court101, param=?upstream=srt&vhost=127.0.0.1, args=(obj)
    [2023-07-04 19:12:20.341][Trace][262728][48937a38] new source, stream_url=/live/GMILWP_Court101
    [2023-07-04 19:12:20.341][Trace][262728][48937a38] source url=/live/GMILWP_Court101, ip=127.0.0.1, cache=0, is_edge=0, source_id=/
    [2023-07-04 19:12:20.341][Trace][262728][48937a38] new source, stream_url=/live/GMILWP_Court101
    [2023-07-04 19:12:20.342][Trace][262728][48937a38] RTC bridge from RTMP, rtmp2rtc=1, keep_bframe=0, merge_nalus=0
    [2023-07-04 19:12:20.342][Trace][262728][48937a38] hls: win=60000ms, frag=2000ms, prefix=, path=/mnt/hls, m3u8=[stream].m3u8, ts=[stream]-[timestamp].ts, aof=2.00, floor=0, clean=1, waitk=1, dispose=0ms, dts_directly=1
    [2023-07-04 19:12:20.342][Trace][262728][48937a38] ignore disabled exec for vhost=__defaultVhost__
    [2023-07-04 19:12:20.342][Trace][262728][48937a38] http: mount flv stream for sid=/live/GMILWP_Court101, mount=/live/GMILWP_Court101.flv
    [2023-07-04 19:12:20.342][Trace][262728][48937a38] start publish mr=0/350, p1stpt=20000, pnt=5000, tcp_nodelay=0
    [2023-07-04 19:12:20.342][Trace][262728][1n188271] RTMP client ip=127.0.0.1:58478, fd=18
    [2023-07-04 19:12:20.343][Trace][262728][6tt95789] complex handshake success.
    [2023-07-04 19:12:20.343][Trace][262728][6tt95789] protocol in.buffer=0, in.ack=0, out.ack=0, in.chunk=128, out.chunk=128
    [2023-07-04 19:12:20.343][Trace][262728][1n188271] complex handshake success
    [2023-07-04 19:12:20.384][Trace][262728][1n188271] connect app, tcUrl=rtmp://127.0.0.1:1936/live, pageUrl=, swfUrl=, schema=rtmp, vhost=127.0.0.1, port=1936, app=live, args=(obj)
    [2023-07-04 19:12:20.384][Trace][262728][1n188271] edge-srs ip=10.0.0.187, version=4.0.269, pid=262728, id=0
    [2023-07-04 19:12:20.384][Trace][262728][1n188271] protocol in.buffer=0, in.ack=0, out.ack=0, in.chunk=128, out.chunk=128
    [2023-07-04 19:12:20.428][Trace][262728][6tt95789] connected, version=4.0.269.0, ip=127.0.0.1, pid=262728, id=0, dsu=1
    [2023-07-04 19:12:20.429][Trace][262728][1n188271] client identified, type=flash-publish, vhost=127.0.0.1, app=live, stream=GMILWP_Court102, param=?upstream=srt&vhost=127.0.0.1, duration=0ms
    [2023-07-04 19:12:20.429][Trace][262728][1n188271] connected stream, tcUrl=rtmp://127.0.0.1:1936/live, pageUrl=, swfUrl=, schema=rtmp, vhost=__defaultVhost__, port=1936, app=live, stream=GMILWP_Court102, param=?upstream=srt&vhost=127.0.0.1, args=(obj)
    [2023-07-04 19:12:20.429][Trace][262728][1n188271] new source, stream_url=/live/GMILWP_Court102
    [2023-07-04 19:12:20.429][Trace][262728][1n188271] source url=/live/GMILWP_Court102, ip=127.0.0.1, cache=0, is_edge=0, source_id=/
    [2023-07-04 19:12:20.429][Trace][262728][1n188271] new source, stream_url=/live/GMILWP_Court102
    [2023-07-04 19:12:20.429][Trace][262728][1n188271] RTC bridge from RTMP, rtmp2rtc=1, keep_bframe=0, merge_nalus=0
    [2023-07-04 19:12:20.430][Trace][262728][1n188271] hls: win=60000ms, frag=2000ms, prefix=, path=/mnt/hls, m3u8=[stream].m3u8, ts=[stream]-[timestamp].ts, aof=2.00, floor=0, clean=1, waitk=1, dispose=0ms, dts_directly=1
    [2023-07-04 19:12:20.430][Trace][262728][1n188271] ignore disabled exec for vhost=__defaultVhost__
    [2023-07-04 19:12:20.430][Trace][262728][1n188271] http: mount flv stream for sid=/live/GMILWP_Court102, mount=/live/GMILWP_Court102.flv
    [2023-07-04 19:12:20.430][Trace][262728][1n188271] start publish mr=0/350, p1stpt=20000, pnt=5000, tcp_nodelay=0
    [2023-07-04 19:12:20.672][Trace][262728][1n188271] 40B video sh,  codec(7, profile=High, level=Other, 1920x1080, 0kbps, 0.0fps, 0.0s)
    [2023-07-04 19:12:20.793][Trace][262728][1n188271] 4B audio sh, codec(10, profile=LC, 2channels, 0kbps, 44100HZ), flv(16bits, 2channels, 44100HZ)
    [2023-07-04 19:12:20.885][Trace][262728][48937a38] 40B video sh,  codec(7, profile=High, level=Other, 1920x1080, 0kbps, 0.0fps, 0.0s)
    [2023-07-04 19:12:20.928][Trace][262728][48937a38] 4B audio sh, codec(10, profile=LC, 2channels, 0kbps, 44100HZ), flv(16bits, 2channels, 44100HZ)
    [2023-07-04 19:12:22.139][Warn][262728][6tt95789][11] resize iovs 2048 => 4096, max_msgs=128
    [2023-07-04 19:12:22.292][Warn][262728][6tt95789][11] resize iovs 2048 => 4096, max_msgs=128
    [2023-07-04 19:12:22.293][Warn][262728][6tt95789][11] resize iovs 4096 => 8192, max_msgs=128
    [2023-07-04 19:12:22.293][Warn][262728][6tt95789][11] c0c3 cache header too small, recoment to 4112
    [2023-07-04 19:12:24.908][Trace][262728][60081xy6] Hybrid cpu=4.00%,27MB
    [2023-07-04 19:12:29.909][Trace][262728][60081xy6] Hybrid cpu=4.00%,27MB, cid=8,1, timer=62,0,0, clock=0,46,2,0,0,1,0,0,0, objs=(pkt:218,raw:67,fua:149,msg:485,oth:1,buf:61)
    [2023-07-04 19:12:30.396][Trace][262728][1n188271] -> HLS time=9967524ms, sno=4, ts=GMILWP_Court102-1688487148867.ts, dur=1533ms, dva=0p
    [2023-07-04 19:12:33.209][Warn][262728][6tt95789][11] Update SRT stream to suitable publish format:live/fantastic
    [2023-07-04 19:12:33.209][Warn][262728][6tt95789][11] Update SRT stream to suitable publish format:live/fantastic
    [2023-07-04 19:12:33.209][Trace][262728][6tt95789] srt connect construct streamid:live/fantastic, mode:2, subpath:live/fantastic, vhost:__default_host__
    [2023-07-04 19:12:33.209][Trace][262728][6tt95789] new srt connection streamid:live/fantastic, fd:548790946, vhost:__default_host__
    [2023-07-04 19:12:33.209][Trace][262728][6tt95789] srto SRTO_LATENCY=2000
    [2023-07-04 19:12:33.209][Trace][262728][6tt95789] srto SRTO_PEERLATENCY=120
    [2023-07-04 19:12:33.209][Trace][262728][6tt95789] srto SRTO_RCVLATENCY=2000
    [2023-07-04 19:12:33.209][Trace][262728][6tt95789] srto SRTO_SNDBUF=12058624
    [2023-07-04 19:12:33.209][Trace][262728][6tt95789] srto SRTO_RCVBUF=12058624
    [2023-07-04 19:12:33.209][Trace][262728][6tt95789] srto SRTO_MAXBW=1000000000
    [2023-07-04 19:12:33.209][Trace][262728][6tt95789] srt mix_correct is enable.
    [2023-07-04 19:12:33.209][Trace][262728][6tt95789] srt h264 sei filter is enable.
    [2023-07-04 19:12:33.209][Trace][262728][6tt95789] srt_handle add new pusher streamid:live/fantastic, subpath:live/fantastic, sid:live/fantastic
    [2023-07-04 19:12:33.209][Trace][262728][6tt95789] new conn added fd:548790946, event:0x00000009
    [2023-07-04 19:12:34.909][Trace][262728][60081xy6] Hybrid cpu=3.00%,27MB, cid=8,1, timer=62,0,0, clock=0,46,2,0,0,1,0,0,0, objs=(pkt:218,raw:67,fua:149,msg:485,oth:1,buf:61)
    [2023-07-04 19:12:35.968][Trace][262728][6tt95789] new rtmp client for srt upstream, key_path:live/fantastic
    [2023-07-04 19:12:35.968][Trace][262728][6tt95789] rtmp client construct url:rtmp://127.0.0.1:1936/live/fantastic?upstream=srt
    [2023-07-04 19:12:35.968][Trace][262728][6u27ocj2] RTMP client ip=127.0.0.1:52628, fd=22
    [2023-07-04 19:12:35.969][Trace][262728][6tt95789] complex handshake success.
    [2023-07-04 19:12:35.969][Trace][262728][6tt95789] protocol in.buffer=0, in.ack=0, out.ack=0, in.chunk=128, out.chunk=128
    [2023-07-04 19:12:35.969][Trace][262728][6u27ocj2] complex handshake success
    [2023-07-04 19:12:36.012][Trace][262728][6u27ocj2] connect app, tcUrl=rtmp://127.0.0.1:1936/live, pageUrl=, swfUrl=, schema=rtmp, vhost=127.0.0.1, port=1936, app=live, args=(obj)
    [2023-07-04 19:12:36.012][Trace][262728][6u27ocj2] edge-srs ip=10.0.0.187, version=4.0.269, pid=262728, id=0
    [2023-07-04 19:12:36.012][Trace][262728][6u27ocj2] protocol in.buffer=0, in.ack=0, out.ack=0, in.chunk=128, out.chunk=128
    [2023-07-04 19:12:36.055][Trace][262728][6tt95789] connected, version=4.0.269.0, ip=127.0.0.1, pid=262728, id=0, dsu=1
    [2023-07-04 19:12:36.058][Trace][262728][6u27ocj2] client identified, type=flash-publish, vhost=127.0.0.1, app=live, stream=fantastic, param=?upstream=srt&vhost=127.0.0.1, duration=0ms
    [2023-07-04 19:12:36.058][Trace][262728][6u27ocj2] connected stream, tcUrl=rtmp://127.0.0.1:1936/live, pageUrl=, swfUrl=, schema=rtmp, vhost=__defaultVhost__, port=1936, app=live, stream=fantastic, param=?upstream=srt&vhost=127.0.0.1, args=(obj)
    [2023-07-04 19:12:36.058][Trace][262728][6u27ocj2] new source, stream_url=/live/fantastic
    [2023-07-04 19:12:36.058][Trace][262728][6u27ocj2] source url=/live/fantastic, ip=127.0.0.1, cache=0, is_edge=0, source_id=/
    [2023-07-04 19:12:36.058][Trace][262728][6u27ocj2] new source, stream_url=/live/fantastic
    [2023-07-04 19:12:36.059][Trace][262728][6u27ocj2] RTC bridge from RTMP, rtmp2rtc=1, keep_bframe=0, merge_nalus=0
    [2023-07-04 19:12:36.059][Trace][262728][6u27ocj2] hls: win=60000ms, frag=2000ms, prefix=, path=/mnt/hls, m3u8=[stream].m3u8, ts=[stream]-[timestamp].ts, aof=2.00, floor=0, clean=1, waitk=1, dispose=0ms, dts_directly=1
    [2023-07-04 19:12:36.059][Trace][262728][6u27ocj2] ignore disabled exec for vhost=__defaultVhost__
    [2023-07-04 19:12:36.059][Trace][262728][6u27ocj2] http: mount flv stream for sid=/live/fantastic, mount=/live/fantastic.flv
    [2023-07-04 19:12:36.059][Trace][262728][6u27ocj2] start publish mr=0/350, p1stpt=20000, pnt=5000, tcp_nodelay=0
    [2023-07-04 19:12:36.577][Trace][262728][6u27ocj2] 32B video sh,  codec(7, profile=High, level=4, 1920x1080, 0kbps, 0.0fps, 0.0s)
    [2023-07-04 19:12:37.704][Error][262728][6tt95789][11] send media data error:code=3041 : demux annexb : annexb start code
    thread [262728][6tt95789]: on_ts_video() [./src/srt/srt_to_rtmp.cpp:517][errno=11]
    thread [262728][6tt95789]: annexb_demux() [src/protocol/srs_raw_avc.cpp:39][errno=11](Resource temporarily unavailable)
    [2023-07-04 19:12:37.705][Trace][262728][6tt95789] close rtmp session which key_path is live/fantastic
    [2023-07-04 19:12:37.705][Trace][262728][6tt95789] rtmp client close url:rtmp://127.0.0.1:1936/live/fantastic?upstream=srt
    [2023-07-04 19:12:37.705][Error][262728][6tt95789][11] handle_push_data srt to rtmp error:3041, fd:548790946(Resource temporarily unavailable)
    [2023-07-04 19:12:37.706][Trace][262728][6u27ocj2] cleanup when unpublish
    [2023-07-04 19:12:37.706][Trace][262728][6u27ocj2] cleanup when unpublish, created=1, deliver=1
    [2023-07-04 19:12:37.706][Trace][262728][6u27ocj2] TCP: before dispose resource(RtmpConn)(0x55f50bd199e0), conns=3, zombies=0, ign=0, inz=0, ind=0
    [2023-07-04 19:12:37.706][Warn][262728][6u27ocj2][4] client disconnect peer. ret=1007
    [2023-07-04 19:12:37.706][Trace][262728][443l8r4r] TCP: clear zombies=1 resources, conns=3, removing=0, unsubs=0
    [2023-07-04 19:12:37.706][Trace][262728][6u27ocj2] TCP: disposing #0 resource(RtmpConn)(0x55f50bd199e0), conns=3, disposing=1, zombies=0
    [2023-07-04 19:12:39.909][Trace][262728][60081xy6] Hybrid cpu=4.99%,27MB, cid=8,1, timer=62,0,0, clock=0,46,2,0,0,1,0,0,0, objs=(pkt:218,raw:67,fua:149,msg:485,oth:1,buf:61)
    [2023-07-04 19:12:40.821][Warn][262728][6tt95789][11] Update SRT stream to suitable publish format:live/fantastic
    [2023-07-04 19:12:40.821][Warn][262728][6tt95789][11] Update SRT stream to suitable publish format:live/fantastic
    [2023-07-04 19:12:40.821][Trace][262728][6tt95789] srt connect construct streamid:live/fantastic, mode:2, subpath:live/fantastic, vhost:__default_host__
    [2023-07-04 19:12:40.821][Trace][262728][6tt95789] new srt connection streamid:live/fantastic, fd:548790945, vhost:__default_host__
    [2023-07-04 19:12:40.821][Trace][262728][6tt95789] srto SRTO_LATENCY=2000
    [2023-07-04 19:12:40.821][Trace][262728][6tt95789] srto SRTO_PEERLATENCY=120
    [2023-07-04 19:12:40.821][Trace][262728][6tt95789] srto SRTO_RCVLATENCY=2000
    [2023-07-04 19:12:40.821][Trace][262728][6tt95789] srto SRTO_SNDBUF=12058624
    [2023-07-04 19:12:40.821][Trace][262728][6tt95789] srto SRTO_RCVBUF=12058624
    [2023-07-04 19:12:40.821][Trace][262728][6tt95789] srto SRTO_MAXBW=1000000000
    [2023-07-04 19:12:40.821][Trace][262728][6tt95789] srt mix_correct is enable.
    [2023-07-04 19:12:40.821][Trace][262728][6tt95789] srt h264 sei filter is enable.
    [2023-07-04 19:12:40.821][Trace][262728][6tt95789] srt_handle add new pusher streamid:live/fantastic, subpath:live/fantastic, sid:live/fantastic
    [2023-07-04 19:12:40.821][Trace][262728][6tt95789] new conn added fd:548790945, event:0x00000009
    [2023-07-04 19:12:42.869][Trace][262728][6tt95789] new rtmp client for srt upstream, key_path:live/fantastic
    [2023-07-04 19:12:42.869][Trace][262728][6tt95789] rtmp client construct url:rtmp://127.0.0.1:1936/live/fantastic?upstream=srt
    [2023-07-04 19:12:42.870][Trace][262728][6x8a8244] RTMP client ip=127.0.0.1:56864, fd=22
    [2023-07-04 19:12:42.871][Trace][262728][6tt95789] complex handshake success.
    [2023-07-04 19:12:42.871][Trace][262728][6tt95789] protocol in.buffer=0, in.ack=0, out.ack=0, in.chunk=128, out.chunk=128
    [2023-07-04 19:12:42.871][Trace][262728][6x8a8244] complex handshake success
    [2023-07-04 19:12:42.911][Trace][262728][6x8a8244] connect app, tcUrl=rtmp://127.0.0.1:1936/live, pageUrl=, swfUrl=, schema=rtmp, vhost=127.0.0.1, port=1936, app=live, args=(obj)
    [2023-07-04 19:12:42.911][Trace][262728][6x8a8244] edge-srs ip=10.0.0.187, version=4.0.269, pid=262728, id=0
    [2023-07-04 19:12:42.911][Trace][262728][6x8a8244] protocol in.buffer=0, in.ack=0, out.ack=0, in.chunk=128, out.chunk=128
    [2023-07-04 19:12:42.955][Trace][262728][6tt95789] connected, version=4.0.269.0, ip=127.0.0.1, pid=262728, id=0, dsu=1
    [2023-07-04 19:12:42.967][Trace][262728][6x8a8244] client identified, type=flash-publish, vhost=127.0.0.1, app=live, stream=fantastic, param=?upstream=srt&vhost=127.0.0.1, duration=0ms
    [2023-07-04 19:12:42.967][Trace][262728][6x8a8244] connected stream, tcUrl=rtmp://127.0.0.1:1936/live, pageUrl=, swfUrl=, schema=rtmp, vhost=__defaultVhost__, port=1936, app=live, stream=fantastic, param=?upstream=srt&vhost=127.0.0.1, args=(obj)
    [2023-07-04 19:12:42.967][Trace][262728][6x8a8244] source url=/live/fantastic, ip=127.0.0.1, cache=0, is_edge=0, source_id=/6u27ocj2
    [2023-07-04 19:12:42.967][Trace][262728][6x8a8244] RTC bridge from RTMP, rtmp2rtc=1, keep_bframe=0, merge_nalus=0
    [2023-07-04 19:12:42.967][Trace][262728][6x8a8244] hls: win=60000ms, frag=2000ms, prefix=, path=/mnt/hls, m3u8=[stream].m3u8, ts=[stream]-[timestamp].ts, aof=2.00, floor=0, clean=1, waitk=1, dispose=0ms, dts_directly=1
    [2023-07-04 19:12:42.967][Trace][262728][6x8a8244] ignore disabled exec for vhost=__defaultVhost__
    [2023-07-04 19:12:42.967][Trace][262728][6x8a8244] start publish mr=0/350, p1stpt=20000, pnt=5000, tcp_nodelay=0
    [2023-07-04 19:12:43.292][Warn][262728][6tt95789][11] resize iovs 2048 => 4096, max_msgs=128
    [2023-07-04 19:12:43.292][Trace][262728][6x8a8244] 32B video sh,  codec(7, profile=High, level=4, 1920x1080, 0kbps, 0.0fps, 0.0s)
    [2023-07-04 19:12:43.293][Trace][262728][6x8a8244] -> HLS time=7220571ms, sno=2, ts=fantastic-1688487162967.ts, dur=0ms, dva=0p
    [2023-07-04 19:12:43.293][Trace][262728][6x8a8244] 4B audio sh, codec(10, profile=LC, 2channels, 0kbps, 48000HZ), flv(16bits, 2channels, 44100HZ)
    [2023-07-04 19:12:44.150][Error][262728][6tt95789][11] send media data error:code=3046 : demux adts : not adts
    thread [262728][6tt95789]: on_ts_audio() [./src/srt/srt_to_rtmp.cpp:624][errno=11]
    thread [262728][6tt95789]: adts_demux() [src/protocol/srs_raw_avc.cpp:305][errno=11](Resource temporarily unavailable)
    [2023-07-04 19:12:44.180][Error][262728][6tt95789][11] send media data error:code=3041 : demux annexb : annexb start code
    thread [262728][6tt95789]: on_ts_video() [./src/srt/srt_to_rtmp.cpp:517][errno=11]
    thread [262728][6tt95789]: annexb_demux() [src/protocol/srs_raw_avc.cpp:39][errno=11](Resource temporarily unavailable)
    [2023-07-04 19:12:44.180][Trace][262728][6tt95789] close rtmp session which key_path is live/fantastic
    [2023-07-04 19:12:44.180][Trace][262728][6tt95789] rtmp client close url:rtmp://127.0.0.1:1936/live/fantastic?upstream=srt
    [2023-07-04 19:12:44.181][Error][262728][6tt95789][11] handle_push_data srt to rtmp error:3046, fd:548790945(Resource temporarily unavailable)
    [2023-07-04 19:12:44.182][Trace][262728][6x8a8244] cleanup when unpublish
    [2023-07-04 19:12:44.182][Trace][262728][6x8a8244] cleanup when unpublish, created=1, deliver=1
    [2023-07-04 19:12:44.182][Warn][262728][6x8a8244][4] 1 frames left in the queue on closing
    [2023-07-04 19:12:44.182][Trace][262728][6x8a8244] TCP: before dispose resource(RtmpConn)(0x55f50bbeef10), conns=3, zombies=0, ign=0, inz=0, ind=0
    [2023-07-04 19:12:44.182][Warn][262728][6x8a8244][4] client disconnect peer. ret=1007
    [2023-07-04 19:12:44.182][Trace][262728][443l8r4r] TCP: clear zombies=1 resources, conns=3, removing=0, unsubs=0
    [2023-07-04 19:12:44.182][Trace][262728][6x8a8244] TCP: disposing #0 resource(RtmpConn)(0x55f50bbeef10), conns=3, disposing=1, zombies=0
    [2023-07-04 19:12:44.909][Trace][262728][60081xy6] Hybrid cpu=10.00%,28MB, cid=12,1, timer=62,0,0, clock=0,45,2,1,0,0,0,0,0, free=1, objs=(pkt:333,raw:124,fua:208,msg:789,oth:1,buf:103)
    [2023-07-04 19:12:45.342][Trace][262728][48937a38] <- CPB time=19986349, okbps=1,0,0, ikbps=822,0,0, mr=0/350, p1stpt=20000, pnt=5000
    [2023-07-04 19:12:46.230][Trace][262728][6tt95789] srt connect construct streamid:#!::h=live/GMILWP_Court101,m=request, mode:1, subpath:live/GMILWP_Court101, vhost:__default_host__
    [2023-07-04 19:12:46.230][Trace][262728][6tt95789] new srt connection streamid:#!::h=live/GMILWP_Court101,m=request, fd:548790944, vhost:__default_host__
    [2023-07-04 19:12:46.230][Trace][262728][6tt95789] srto SRTO_LATENCY=0
    [2023-07-04 19:12:46.230][Trace][262728][6tt95789] srto SRTO_PEERLATENCY=120
    [2023-07-04 19:12:46.230][Trace][262728][6tt95789] srto SRTO_RCVLATENCY=0
    [2023-07-04 19:12:46.230][Trace][262728][6tt95789] srto SRTO_SNDBUF=12058624
    [2023-07-04 19:12:46.230][Trace][262728][6tt95789] srto SRTO_RCVBUF=12058624
    [2023-07-04 19:12:46.230][Trace][262728][6tt95789] srto SRTO_MAXBW=1000000000
    [2023-07-04 19:12:46.230][Trace][262728][6tt95789] srt mix_correct is enable.
    [2023-07-04 19:12:46.230][Trace][262728][6tt95789] srt h264 sei filter is enable.
    [2023-07-04 19:12:46.230][Trace][262728][6tt95789] add new puller fd:548790944, streamid:live/GMILWP_Court101
    [2023-07-04 19:12:46.230][Trace][262728][6tt95789] new conn added fd:548790944, event:0x00000009
    [2023-07-04 19:12:46.811][Trace][262728][6tt95789] srt connect construct streamid:#!::h=live/GMILWP_Court102,m=request, mode:1, subpath:live/GMILWP_Court102, vhost:__default_host__
    [2023-07-04 19:12:46.811][Trace][262728][6tt95789] new srt connection streamid:#!::h=live/GMILWP_Court102,m=request, fd:548790943, vhost:__default_host__
    [2023-07-04 19:12:46.811][Trace][262728][6tt95789] srto SRTO_LATENCY=0
    [2023-07-04 19:12:46.811][Trace][262728][6tt95789] srto SRTO_PEERLATENCY=120
    [2023-07-04 19:12:46.811][Trace][262728][6tt95789] srto SRTO_RCVLATENCY=0
    [2023-07-04 19:12:46.811][Trace][262728][6tt95789] srto SRTO_SNDBUF=12058624
    [2023-07-04 19:12:46.811][Trace][262728][6tt95789] srto SRTO_RCVBUF=12058624
    [2023-07-04 19:12:46.811][Trace][262728][6tt95789] srto SRTO_MAXBW=1000000000
    [2023-07-04 19:12:46.811][Trace][262728][6tt95789] srt mix_correct is enable.
    [2023-07-04 19:12:46.811][Trace][262728][6tt95789] srt h264 sei filter is enable.
    [2023-07-04 19:12:46.811][Trace][262728][6tt95789] add new puller fd:548790943, streamid:live/GMILWP_Court102
    [2023-07-04 19:12:46.811][Trace][262728][6tt95789] new conn added fd:548790943, event:0x00000009
    [2023-07-04 19:12:47.331][Warn][262728][6tt95789][11] Update SRT stream to suitable publish format:live/fantastic
    [2023-07-04 19:12:47.331][Warn][262728][6tt95789][11] Update SRT stream to suitable publish format:live/fantastic
    [2023-07-04 19:12:47.331][Trace][262728][6tt95789] srt connect construct streamid:live/fantastic, mode:2, subpath:live/fantastic, vhost:__default_host__
    [2023-07-04 19:12:47.331][Trace][262728][6tt95789] new srt connection streamid:live/fantastic, fd:548790942, vhost:__default_host__
    [2023-07-04 19:12:47.331][Trace][262728][6tt95789] srto SRTO_LATENCY=2000
    [2023-07-04 19:12:47.331][Trace][262728][6tt95789] srto SRTO_PEERLATENCY=120
    [2023-07-04 19:12:47.331][Trace][262728][6tt95789] srto SRTO_RCVLATENCY=2000
    [2023-07-04 19:12:47.331][Trace][262728][6tt95789] srto SRTO_SNDBUF=12058624
    [2023-07-04 19:12:47.331][Trace][262728][6tt95789] srto SRTO_RCVBUF=12058624
    [2023-07-04 19:12:47.331][Trace][262728][6tt95789] srto SRTO_MAXBW=1000000000
    [2023-07-04 19:12:47.331][Trace][262728][6tt95789] srt mix_correct is enable.
    [2023-07-04 19:12:47.331][Trace][262728][6tt95789] srt h264 sei filter is enable.
    [2023-07-04 19:12:47.331][Trace][262728][6tt95789] srt_handle add new pusher streamid:live/fantastic, subpath:live/fantastic, sid:live/fantastic
    [2023-07-04 19:12:47.331][Trace][262728][6tt95789] new conn added fd:548790942, event:0x00000009
    [2023-07-04 19:12:49.910][Trace][262728][60081xy6] Hybrid cpu=7.99%,28MB, cid=12,1, timer=62,0,0, clock=0,45,2,1,0,0,0,0,0, free=1, objs=(pkt:333,raw:124,fua:208,msg:789,oth:1,buf:103)
    [2023-07-04 19:12:49.963][Trace][262728][6tt95789] new rtmp client for srt upstream, key_path:live/fantastic
    [2023-07-04 19:12:49.963][Trace][262728][6tt95789] rtmp client construct url:rtmp://127.0.0.1:1936/live/fantastic?upstream=srt
    [2023-07-04 19:12:49.963][Trace][262728][6tt95789] close rtmp session which key_path is live/fantastic
    [2023-07-04 19:12:49.963][Error][262728][6tt95789][11] handle_push_data srt to rtmp error:3041, fd:548790942(Resource temporarily unavailable)
    [2023-07-04 19:12:54.911][Trace][262728][60081xy6] Hybrid cpu=7.00%,28MB, cid=5,0, timer=62,0,0, clock=0,45,3,0,0,0,0,0,0, objs=(pkt:352,raw:118,fua:232,msg:791,oth:1,buf:101)
    [2023-07-04 19:12:55.431][Trace][262728][1n188271] <- CPB time=29987978, okbps=0,1,0, ikbps=1117,1080,0, mr=0/350, p1stpt=20000, pnt=5000
    [2023-07-04 19:12:57.848][Trace][262728][48937a38] -> HLS time=37489435ms, sno=11, ts=GMILWP_Court101-1688487176993.ts, dur=833ms, dva=0p
    [2023-07-04 19:12:59.912][Trace][262728][60081xy6] Hybrid cpu=5.00%,28MB, cid=5,0, timer=62,0,0, clock=0,45,3,0,0,0,0,0,0, objs=(pkt:352,raw:118,fua:232,msg:791,oth:1,buf:101)
    [2023-07-04 19:13:04.912][Trace][262728][60081xy6] Hybrid cpu=7.99%,28MB, cid=5,0, timer=62,0,0, clock=0,45,3,0,0,0,0,0,0, objs=(pkt:352,raw:118,fua:232,msg:791,oth:1,buf:101)
    [2023-07-04 19:13:05.432][Trace][262728][1n188271] <- CPB time=40002037, okbps=0,1,0, ikbps=1195,1080,0, mr=0/350, p1stpt=20000, pnt=5000
    [2023-07-04 19:13:06.076][Trace][262728][1796k464] RTMP client ip=34.90.254.141:56147, fd=21
    [2023-07-04 19:13:06.142][Trace][262728][1796k464] complex handshake success
    [2023-07-04 19:13:06.206][Trace][262728][1796k464] connect app, tcUrl=rtmp://psnginx-meast-israel-01.playsight.com:1936/live, pageUrl=, swfUrl=, schema=rtmp, vhost=psnginx-meast-israel-01.playsight.com, port=1936, app=live, args=null
    [2023-07-04 19:13:06.206][Trace][262728][1796k464] protocol in.buffer=0, in.ack=0, out.ack=0, in.chunk=128, out.chunk=128
    [2023-07-04 19:13:06.491][Trace][262728][1796k464] client identified, type=fmle-publish, vhost=psnginx-meast-israel-01.playsight.com, app=live, stream=parent_a9b8f2c71902c50e_0f7347a0-a421-4029-ab67-c5f386199417, param=, duration=0ms
    [2023-07-04 19:13:06.492][Trace][262728][1796k464] connected stream, tcUrl=rtmp://psnginx-meast-israel-01.playsight.com:1936/live, pageUrl=, swfUrl=, schema=rtmp, vhost=__defaultVhost__, port=1936, app=live, stream=parent_a9b8f2c71902c50e_0f7347a0-a421-4029-ab67-c5f386199417, param=, args=null
    [2023-07-04 19:13:06.492][Trace][262728][1796k464] new source, stream_url=/live/parent_a9b8f2c71902c50e_0f7347a0-a421-4029-ab67-c5f386199417
    [2023-07-04 19:13:06.492][Trace][262728][1796k464] source url=/live/parent_a9b8f2c71902c50e_0f7347a0-a421-4029-ab67-c5f386199417, ip=34.90.254.141, cache=0, is_edge=0, source_id=/
    [2023-07-04 19:13:06.772][Trace][262728][v9856x87] RTMP client ip=35.204.239.70:56131, fd=22
    [2023-07-04 19:13:06.775][Trace][262728][1796k464] new source, stream_url=/live/parent_a9b8f2c71902c50e_0f7347a0-a421-4029-ab67-c5f386199417
    [2023-07-04 19:13:06.776][Trace][262728][1796k464] RTC bridge from RTMP, rtmp2rtc=1, keep_bframe=0, merge_nalus=0
    [2023-07-04 19:13:06.776][Trace][262728][1796k464] hls: win=60000ms, frag=2000ms, prefix=, path=/mnt/hls, m3u8=[stream].m3u8, ts=[stream]-[timestamp].ts, aof=2.00, floor=0, clean=1, waitk=1, dispose=0ms, dts_directly=1
    [2023-07-04 19:13:06.776][Trace][262728][1796k464] ignore disabled exec for vhost=__defaultVhost__
    [2023-07-04 19:13:06.776][Trace][262728][1796k464] http: mount flv stream for sid=/live/parent_a9b8f2c71902c50e_0f7347a0-a421-4029-ab67-c5f386199417, mount=/live/parent_a9b8f2c71902c50e_0f7347a0-a421-4029-ab67-c5f386199417.flv
    [2023-07-04 19:13:06.776][Trace][262728][1796k464] start publish mr=0/350, p1stpt=20000, pnt=5000, tcp_nodelay=0
    [2023-07-04 19:13:06.843][Trace][262728][v9856x87] complex handshake success
    [2023-07-04 19:13:06.907][Trace][262728][v9856x87] connect app, tcUrl=rtmp://psnginx-meast-israel-01.playsight.com:1936/live, pageUrl=, swfUrl=, schema=rtmp, vhost=psnginx-meast-israel-01.playsight.com, port=1936, app=live, args=null
    [2023-07-04 19:13:06.907][Trace][262728][v9856x87] protocol in.buffer=0, in.ack=0, out.ack=0, in.chunk=128, out.chunk=128
    [2023-07-04 19:13:06.983][Trace][262728][1796k464] got metadata, width=1920, height=1080, vcodec=7, acodec=10
    [2023-07-04 19:13:06.983][Trace][262728][1796k464] 45B video sh,  codec(7, profile=High, level=Other, 1920x1080, 0kbps, 0.0fps, 0.0s)
    [2023-07-04 19:13:06.983][Trace][262728][1796k464] 7B audio sh, codec(10, profile=LC, 1channels, 0kbps, 44100HZ), flv(16bits, 2channels, 44100HZ)
    [2023-07-04 19:13:07.184][Trace][262728][v9856x87] client identified, type=fmle-publish, vhost=psnginx-meast-israel-01.playsight.com, app=live, stream=parent_4aeeff5ded82a0bf_f05b7577-f1e7-4583-a713-363fb8ecae1b, param=, duration=0ms
    [2023-07-04 19:13:07.184][Trace][262728][v9856x87] connected stream, tcUrl=rtmp://psnginx-meast-israel-01.playsight.com:1936/live, pageUrl=, swfUrl=, schema=rtmp, vhost=__defaultVhost__, port=1936, app=live, stream=parent_4aeeff5ded82a0bf_f05b7577-f1e7-4583-a713-363fb8ecae1b, param=, args=null
    [2023-07-04 19:13:07.184][Trace][262728][v9856x87] new source, stream_url=/live/parent_4aeeff5ded82a0bf_f05b7577-f1e7-4583-a713-363fb8ecae1b
    [2023-07-04 19:13:07.184][Trace][262728][v9856x87] source url=/live/parent_4aeeff5ded82a0bf_f05b7577-f1e7-4583-a713-363fb8ecae1b, ip=35.204.239.70, cache=0, is_edge=0, source_id=/
    [2023-07-04 19:13:07.416][Trace][262728][v9856x87] new source, stream_url=/live/parent_4aeeff5ded82a0bf_f05b7577-f1e7-4583-a713-363fb8ecae1b
    [2023-07-04 19:13:07.416][Trace][262728][v9856x87] RTC bridge from RTMP, rtmp2rtc=1, keep_bframe=0, merge_nalus=0
    [2023-07-04 19:13:07.416][Trace][262728][v9856x87] hls: win=60000ms, frag=2000ms, prefix=, path=/mnt/hls, m3u8=[stream].m3u8, ts=[stream]-[timestamp].ts, aof=2.00, floor=0, clean=1, waitk=1, dispose=0ms, dts_directly=1
    [2023-07-04 19:13:07.416][Trace][262728][v9856x87] ignore disabled exec for vhost=__defaultVhost__
    [2023-07-04 19:13:07.416][Trace][262728][v9856x87] http: mount flv stream for sid=/live/parent_4aeeff5ded82a0bf_f05b7577-f1e7-4583-a713-363fb8ecae1b, mount=/live/parent_4aeeff5ded82a0bf_f05b7577-f1e7-4583-a713-363fb8ecae1b.flv
    [2023-07-04 19:13:07.416][Trace][262728][v9856x87] start publish mr=0/350, p1stpt=20000, pnt=5000, tcp_nodelay=0
    [2023-07-04 19:13:07.604][Trace][262728][v9856x87] got metadata, width=1920, height=1080, vcodec=7, acodec=10
    [2023-07-04 19:13:07.604][Trace][262728][v9856x87] 45B video sh,  codec(7, profile=High, level=Other, 1920x1080, 0kbps, 0.0fps, 0.0s)
    [2023-07-04 19:13:07.604][Trace][262728][v9856x87] 7B audio sh, codec(10, profile=LC, 1channels, 0kbps, 44100HZ), flv(16bits, 2channels, 44100HZ)
    [2023-07-04 19:13:09.913][Trace][262728][60081xy6] Hybrid cpu=25.97%,31MB, cid=6,1, timer=62,0,0, clock=0,45,3,0,0,0,0,0,0, objs=(pkt:952,raw:365,fua:584,msg:1651,oth:1,buf:193)
    [2023-07-04 19:13:13.460][Trace][262728][f6680765] HTTP #0 141.98.10.48:47776 GET http://10.0.0.187/manager/html, content-length=-1
    [2023-07-04 19:13:13.460][Warn][262728][f6680765][2] http miss file=/mnt/hls/manager/html, pattern=/, upath=/manager/html
    [2023-07-04 19:13:13.461][Trace][262728][f6680765] TCP: before dispose resource(HttpStream)(0x55f50bbed330), conns=5, zombies=0, ign=0, inz=0, ind=0
    [2023-07-04 19:13:13.461][Trace][262728][f6680765] client finished.
    [2023-07-04 19:13:13.461][Trace][262728][443l8r4r] TCP: clear zombies=1 resources, conns=5, removing=0, unsubs=0
    [2023-07-04 19:13:13.461][Trace][262728][f6680765] TCP: disposing #0 resource(HttpStream)(0x55f50bbed330), conns=5, disposing=1, zombies=0
    [2023-07-04 19:13:14.913][Trace][262728][60081xy6] Hybrid cpu=10.99%,31MB, cid=6,1, timer=62,0,0, clock=0,45,3,0,0,0,0,0,0, objs=(pkt:952,raw:365,fua:584,msg:1651,oth:1,buf:193)
    [2023-07-04 19:13:14.913][Trace][262728][v9856x87] -> HLS time=7724207ms, sno=5, ts=parent_4aeeff5ded82a0bf_f05b7577-f1e7-4583-a713-363fb8ecae1b-1688487192136.ts, dur=2599ms, dva=0p
    [2023-07-04 19:13:19.918][Trace][262728][60081xy6] Hybrid cpu=11.00%,31MB, cid=3,1, timer=61,0,0, clock=0,43,5,0,0,0,0,0,0, free=1, objs=(pkt:1207,raw:393,fua:811,msg:1936,oth:1,buf:201)
    [2023-07-04 19:13:24.915][Trace][262728][60081xy6] Hybrid cpu=10.99%,31MB, cid=3,1, timer=61,0,0, clock=0,43,5,0,0,0,0,0,0, free=1, objs=(pkt:1207,raw:393,fua:811,msg:1936,oth:1,buf:201)
    [2023-07-04 19:13:25.434][Trace][262728][1n188271] <- CPB time=59999589, okbps=0,0,0, ikbps=1428,1626,0, mr=0/350, p1stpt=20000, pnt=5000
    [2023-07-04 19:13:27.463][Trace][262728][1796k464] -> HLS time=20969043ms, sno=10, ts=parent_a9b8f2c71902c50e_0f7347a0-a421-4029-ab67-c5f386199417-1688487206345.ts, dur=960ms, dva=0p
    [2023-07-04 19:13:29.915][Trace][262728][60081xy6] Hybrid cpu=12.00%,32MB, cid=3,1, timer=61,0,0, clock=0,43,5,0,0,0,0,0,0, free=1, objs=(pkt:1207,raw:393,fua:811,msg:1936,oth:1,buf:201)
    [2023-07-04 19:13:30.348][Trace][262728][48937a38] <- CPB time=64994516, okbps=0,0,0, ikbps=1446,1669,0, mr=0/350, p1stpt=20000, pnt=5000
    [2023-07-04 19:13:34.916][Trace][262728][60081xy6] Hybrid cpu=8.00%,31MB, cid=3,0, timer=62,0,0, clock=0,44,4,0,0,0,0,0,0, objs=(pkt:1458,raw:383,fua:1072,msg:2186,oth:2,buf:201)
  3. SRS Config:

    
    listen              1936;
    max_connections     1000;
    daemon              off;
    srs_log_tank        file;
    srs_log_file        ./objs/logs/srs.log;

http_api {

whether http api is enabled.

# default: off
enabled         on;
# the http api listen entry is <[ip:]port>
# for example, 192.168.1.100:1985
# where the ip is optional, default to 0.0.0.0, that is 1985 equals to 0.0.0.0:1985
# default: 1985
listen          1985;
# whether enable crossdomain request.
# default: on
crossdomain     on;
# the HTTP RAW API is more powerful api to change srs state and reload.
raw_api {
    # whether enable the HTTP RAW API.
    # default: off
    enabled             off;
    # whether enable rpc reload.
    # default: off
    allow_reload        off;
    # whether enable rpc query.
    # default: off
    allow_query         off;
    # whether enable rpc update.
    # default: off
    allow_update        off;
}
# For https_api or HTTPS API.
https {
    # Whether enable HTTPS API.
    # default: off
    enabled on;
    # The listen endpoint for HTTPS API.
    # default: 1986
    listen 1986;
    # The SSL private key file, generated by:
    #       openssl genrsa -out server.key 2048
    # default: ./conf/server.key
    key ./conf/wildcard_playsight_com.key;
    # The SSL public cert file, generated by:
    #       openssl req -new -x509 -key server.key -out server.crt -days 3650 -subj "/C=CN/ST=Beijing/L=Beijing/O=Me/OU=Me/CN=ossrs.net"
    # default: ./conf/server.crt
}

}

http_server { enabled on; listen 8080; dir /mnt/hls; }

srt_server { enabled on; listen 9401; maxbw 1000000000; connect_timeout 4000; peerlatency 2500; recvlatency 2500; latency 2500; }

rtc_server { enabled on;

Listen at udp://8000

listen 8000;
#
# The $CANDIDATE means fetch from env, if not configed, use * as default.
#
# The * means retrieving server IP automatically, from all network interfaces,
# @see https://github.com/ossrs/srs/wiki/v4_CN_RTCWiki#config-candidate
# candidate 0.0.0.0;

}

@doc https://github.com/ossrs/srs/issues/1147#issuecomment-577607026

vhost defaultVhost { min_latency off; tcp_nodelay off; time_jitter off; mix_correct off;

play {
    gop_cache off;
    queue_length 10;
    mw_latency 100;
}
rtc {
    enabled     on;
    rtmp_to_rtc on;
    rtc_to_rtmp on;
}
http_remux {
    enabled     on;
    mount       [vhost]/[app]/[stream].flv;
}
 hls {
    # whether the hls is enabled.
    # if off, do not write hls(ts and m3u8) when publish.
    # default: off
    enabled         on;
    # the hls fragment in seconds, the duration of a piece of ts.
    # default: 10
    hls_fragment    2;
    # the hls m3u8 target duration ratio,
    #   EXT-X-TARGETDURATION = hls_td_ratio * hls_fragment // init
    #   EXT-X-TARGETDURATION = max(ts_duration, EXT-X-TARGETDURATION) // for each ts
    # @see https://github.com/ossrs/srs/issues/304#issuecomment-74000081
    # default: 1.5
    #hls_td_ratio    1;
    # the audio overflow ratio.
    # for pure audio, the duration to reap the segment.
    # for example, the hls_fragment is 10s, hls_aof_ratio is 2.0,
    # the segment will reap to 20s for pure audio.
    # default: 2.0
    hls_aof_ratio   2.0;
    # the hls window in seconds, the number of ts in m3u8.
    # default: 60
    hls_window      60;
    # the error strategy. can be:
    #       ignore, disable the hls.
    #       disconnect, require encoder republish.
    #       continue, ignore failed try to continue output hls.
    # @see https://github.com/ossrs/srs/issues/264
    # default: continue
    hls_on_error    continue;
    # the hls output path.
    # the m3u8 file is configured by hls_path/hls_m3u8_file, the default is:
    #       ./objs/nginx/html/[app]/[stream].m3u8
    # the ts file is configured by hls_path/hls_ts_file, the default is:
    #       ./objs/nginx/html/[app]/[stream]-[seq].ts
    # @remark the hls_path is compatible with srs v1 config.
    # default: ./objs/nginx/html
    hls_path        /mnt/hls;
    # the hls m3u8 file name.
    # we supports some variables to generate the filename.
    #       [vhost], the vhost of stream.
    #       [app], the app of stream.
    #       [stream], the stream name of stream.
    # default: [app]/[stream].m3u8
    hls_m3u8_file   [stream].m3u8;
    # the hls ts file name.
    # we supports some variables to generate the filename.
    #       [vhost], the vhost of stream.
    #       [app], the app of stream.
    #       [stream], the stream name of stream.
    #       [2006], replace this const to current year.
    #       [01], replace this const to current month.
    #       [02], replace this const to current date.
    #       [15], replace this const to current hour.
    #       [04], replace this const to current minute.
    #       [05], replace this const to current second.
    #       [999], replace this const to current millisecond.
    #       [timestamp],replace this const to current UNIX timestamp in ms.
    #       [seq], the sequence number of ts.
    #       [duration], replace this const to current ts duration.
    # @see https://github.com/ossrs/srs/wiki/v2_CN_DVR#custom-path
    # @see https://github.com/ossrs/srs/wiki/v2_CN_DeliveryHLS#hls-config
    # default: [app]/[stream]-[seq].ts
    hls_ts_file     [stream]-[timestamp].ts;
    # whether use floor for the hls_ts_file path generation.
    # if on, use floor(timestamp/hls_fragment) as the variable [timestamp],
    #       and use enhanced algorithm to calc deviation for segment.
    # @remark when floor on, recommend the hls_segment>=2*gop.
    # default: off
    hls_ts_floor    off;
    # the hls entry prefix, which is base url of ts url.
    # for example, the prefix is:
    #         http://your-server/
    # then, the ts path in m3u8 will be like:
    #         http://your-server/live/livestream-0.ts
    #         http://your-server/live/livestream-1.ts
    #         ...
    # optional, default to empty string.
    #hls_entry_prefix http://your-server;
    # the default audio codec of hls.
    # when codec changed, write the PAT/PMT table, but maybe ok util next ts.
    # so user can set the default codec for mp3.
    # the available audio codec:
    #       aac, mp3, an
    # default: aac
    #hls_acodec      aac;
    # the default video codec of hls.
    # when codec changed, write the PAT/PMT table, but maybe ok util next ts.
    # so user can set the default codec for pure audio(without video) to vn.
    # the available video codec:
    #       h264, vn
    # default: h264
    #hls_vcodec      h264;
    # whether cleanup the old expired ts files.
    # default: on
    hls_cleanup     on;
    # If there is no incoming packets, dispose HLS in this timeout in seconds,
    # which removes all HLS files including m3u8 and ts files.
    # @remark 0 to disable dispose for publisher.
    # @remark apply for publisher timeout only, while "etc/init.d/srs stop" always dispose hls.
    # default: 0
    hls_dispose     0;
    # the max size to notify hls,
    # to read max bytes from ts of specified cdn network,
    # @remark only used when on_hls_notify is config.
    # default: 64
    hls_nb_notify   64;
    # whether wait keyframe to reap segment,
    # if off, reap segment when duration exceed the fragment,
    # if on, reap segment when duration exceed and got keyframe.
    # default: on
    hls_wait_keyframe       on;

    # whether using AES encryption.
    # default: off
    hls_keys        off; 
    # the number of clear ts which one key can encrypt.
    # default: 5
    hls_fragments_per_key 5;
    # the hls key file name.
    # we supports some variables to generate the filename.
    #       [vhost], the vhost of stream.
    #       [app], the app of stream.
    #       [stream], the stream name of stream.
    #       [seq], the sequence number of key corresponding to the ts.
    hls_key_file     [app]/[stream]-[seq].key;
    # the key output path.
    # the key file is configed by hls_path/hls_key_file, the default is:
    # ./objs/nginx/html/[app]/[stream]-[seq].key
    hls_key_file_path    ./objs/nginx/html;
    # the key root URL, use this can support https.
    # @remark It's optional.
    hls_key_url       https://localhost:8080;

    # Special control controls.
    ###########################################
    # Whether calculate the DTS of audio frame directly.
    # If on, guess the specific DTS by AAC samples, please read https://github.com/ossrs/srs/issues/547#issuecomment-294350544
    # If off, directly turn the FLV timestamp to DTS, which might cause corrupt audio stream.
    # @remark Recommend to set to off, unless your audio stream sample-rate and timestamp is not correct.
    # Default: on
    hls_dts_directly on;

    # on_hls, never config in here, should config in http_hooks.
    # for the hls http callback, @see http_hooks.on_hls of vhost hooks.callback.srs.com
    # @read https://github.com/ossrs/srs/wiki/v2_CN_DeliveryHLS#http-callback
    # @read https://github.com/ossrs/srs/wiki/v2_EN_DeliveryHLS#http-callback

    # on_hls_notify, never config in here, should config in http_hooks.
    # we support the variables to generate the notify url:
    #       [app], replace with the app.
    #       [stream], replace with the stream.
    #       [param], replace with the param.
    #       [ts_url], replace with the ts url.
    # for the hls http callback, @see http_hooks.on_hls_notify of vhost hooks.callback.srs.com
    # @read https://github.com/ossrs/srs/wiki/v2_CN_DeliveryHLS#on-hls-notify
    # @read https://github.com/ossrs/srs/wiki/v2_EN_DeliveryHLS#on-hls-notify
}

}



## Replay
Run Larix Broadcaster with any algorithm. 
Put a bit rate more than you can transmit on your network. 
Larix client will be disconnected from Srs server 4.0.228 immediately and no take connection timeout parameter. 
You can try it with the last 5.0.b1, and the issue doesn't exist. 

## Expect
To the same behavior as 5.0b1 to take into account connection timeout.
xiaozhihong commented 1 year ago

The error is casue by [2023-07-04 19:12:37.704][Error][262728][6tt95789][11] send media data error:code=3041 : demux annexb : annexb start code

You just mentioned when the Larix broadcaster starts to lose packets in any Larix algorithm., Is this error only occurring when the packet is lost?

Can you try to use the develop version and provide the log? Thanks.

EvgIL74 commented 1 year ago

You just mentioned when the Larix broadcaster starts to lose packets in any Larix algorithm., Is this error only occurring when the packet is lost? Yes. Its happening when Larix is loosing packets. Can you try to use the developed version and provide the log? I run the developer version (5..0) also, and it was ok. I will provide logs also.

EvgIL74 commented 1 year ago

I closed the srt_to_rtmp function in 4.0.228, and now no disconnection is happening, but from now it's a permanent memory leak.

srs.log

xiaozhihong commented 1 year ago

I closed the srt_to_rtmp function in 4.0.228, and now no disconnection is happening, but from now it's a permanent memory leak.

srs.log

Version 4.0 srt had unstable and did not provide option srt_to_rtmp, I refactor it in 5.0. Maybe we can try to resolve this issue #3573, and upgrade your version to 5.x?

EvgIL74 commented 1 year ago

Unfortennly, I tried the last 5.X, and the memory leak is happening with the same scenario with Larix.

srs.log

EvgIL74 commented 1 year ago

I think that I figure out the memory leak issue. I checked the 5.0.21 version randomly and found that no memory leak exists on the 5.0.release version. Please look in the previous log and you find out (CasterTsPse)(Invalid ts PSE payload for stream caster) : ts: ts packet decode : ts: demux payload : ts: PES fresh packet length=0, us=0, cc=11 . I think that it's connecting to a memory leak that doesn't exist in 5.0.21 srs.log

winlinvip commented 1 year ago

Version 5.0.21 is outdated; it is recommended to use the latest 5.0 version, such as 5.0.166.

EvgIL74 commented 1 year ago

Version 5.0.21 is outdated; it is recommended to use the latest 5.0 version, such as 5.0.166.

As I mentioned in previous remarks. I can't use the latest 5.0 version because it has a huge memory leak in the latest version and I attached a log. Please look at my two previous posts about it.

xiaozhihong commented 1 year ago

Unfortennly, I tried the last 5.X, and the memory leak is happening with the same scenario with Larix.

srs.log

image What is happening with this log? It appears that a certain SRT client is connecting to the server frequently.

xiaozhihong commented 1 year ago

Unfortennly, I tried the last 5.X, and the memory leak is happening with the same scenario with Larix.

srs.log

I try 5.0.168, no memory leak. Use jemalloc and enable perf_leak, publish srt stream and limit the network to simulate packet loss, try several times, the result shows that no memory leak.

image

I am puzzled as to why I cannot reproduce this issue in my environment.

You can try streaming for an extended period, such as 30 minutes, and observe that the memory tends to stabilize.

EvgIL74 commented 1 year ago

Did you see (CasterTsPse)(Invalid ts PSE payload for stream caster) : ts: ts packet decode : ts: demux payload : ts: PES fresh packet length=0, us=0, cc=11 in your test? I think that it is connected to a memory leak.

Best, Evgeni

xiaozhihong commented 1 year ago

Did you see (CasterTsPse)(Invalid ts PSE payload for stream caster) : ts: ts packet decode : ts: demux payload : ts: PES fresh packet length=0, us=0, cc=11 in your test? I think that it is connected to a memory leak.

Best, Evgeni

Yes, I simulate a 10% loss of my network, (CasterTsPse)(Invalid ts PSE payload for stream caster) : ts: ts packet decode : ts: demux payload : ts: PES fresh packet length=0, us=0, cc=11 This log appears very frequently. But still no memory leak.

winlinvip commented 4 months ago

I need to close this issue for disconnecting issue has been fixed in SRS 5.0. However, you can reopen it by reproducing the memory leaking issue.