WebRTC: aiortc pushes to SRS, pulling stream flickers and blurs

[robin2099]

Note: Issues that do not provide the following information will be deleted directly (Please follow the issue template, or we will delete it) ' Make sure to maintain the markdown structure.

Note: For inquiries and discussions, please submit them to the SRS community (Please ask questions at) http://bbs.ossrs.net ' Make sure to maintain the markdown structure.

Description aiortc pushes to SRS, but the pulled stream appears flickering and blurry.

Please describe the issue you are facing here. aiortc works fine for local web streaming, but when pushing to SRS and pulling the stream, the video appears flickering and blurry. I'm not sure if there is a misconfiguration.

1. SRS version: docker ossrs/srs 4
2. SRS logs:

[2021-11-01 09:06:56.454][Trace][1][o094f950] TCP: before dispose resource(HttpConn)(0x1516c50), conns=1, zombies=0, ign=0, inz=0, ind=0
[2021-11-01 09:06:56.454][Trace][1][o094f950] client finished.
[2021-11-01 09:06:56.454][Trace][1][40h31016] TCP: clear zombies=1 resources, conns=1, removing=0, unsubs=0
[2021-11-01 09:06:56.454][Trace][1][o094f950] TCP: disposing #0 resource(HttpConn)(0x1516c50), conns=1, disposing=1, zombies=0
[2021-11-01 09:06:56.491][Trace][1][o094f950] RTC: session address init 116.230.54.223:2453
[2021-11-01 09:06:56.491][Trace][1][o094f950] RTC: session STUN done, waiting DTLS handshake.
[2021-11-01 09:06:56.512][Trace][1][o094f950] DTLS: State Passive RECV, done=0, arq=0/0, r0=1, r1=0, len=301, cnt=22, size=243, hs=1
[2021-11-01 09:06:56.513][Trace][1][o094f950] DTLS: State Passive SEND, done=0, arq=0/0, r0=-1, r1=2, len=679, cnt=22, size=82, hs=2
[2021-11-01 09:06:56.527][Trace][1][o094f950] DTLS: State Passive RECV, done=0, arq=0/0, r0=1, r1=0, len=667, cnt=22, size=243, hs=11
[2021-11-01 09:06:56.528][Trace][1][o094f950] DTLS: State Passive SEND, done=1, arq=0/0, r0=1, r1=0, len=618, cnt=22, size=530, hs=4
[2021-11-01 09:06:56.528][Trace][1][o094f950] RTC: DTLS handshake done.
[2021-11-01 09:06:56.528][Trace][1][o094f950] RTC: session pub=1, sub=0, to=30000ms connection established
[2021-11-01 09:06:56.528][Trace][1][o094f950] RTC: Publisher url=/live/livestream established
[2021-11-01 09:06:58.445][Trace][1][o094f950] NACK: update seq=5166, nack range [5165, 5166]
[2021-11-01 09:06:59.015][Trace][1][v374mj74] Hybrid cpu=1.00%,17MB, cid=1,2, timer=62,8,36, clock=0,47,1,0,0,0,0,0,0, free=1, objs=(pkt:87,raw:0,fua:0,msg:87,oth:87,buf:87)
[2021-11-01 09:06:59.015][Trace][1][v374mj74] RTC: Server conns=1, rpkts=(84,rtp:83,stun:1,rtcp:1), spkts=(3,rtp:0,stun:1,rtcp:2), rtcp=(pli:0,twcc:0,rr:1), snk=(67,a:33,v:33,h:1), fid=(id:1,fid:84,ffid:0,addr:1,faddr:84)
[2021-11-01 09:07:00.248][Trace][1][79sf4v65] <- RTC RECV #10, udp 507, pps 4/50, schedule 507
[2021-11-01 09:07:04.015][Trace][1][v374mj74] Hybrid cpu=1.00%,17MB, cid=1,2, timer=62,8,36, clock=0,47,1,0,0,0,0,0,0, free=1, objs=(pkt:87,raw:0,fua:0,msg:87,oth:87,buf:87)
[2021-11-01 09:07:04.015][Trace][1][v374mj74] RTC: Server conns=1, rpkts=(84,rtp:83,stun:1,rtcp:1), spkts=(3,rtp:0,stun:1,rtcp:2), rtcp=(pli:0,twcc:0,rr:1), snk=(67,a:33,v:33,h:1), fid=(id:1,fid:84,ffid:0,addr:1,faddr:84)
[2021-11-01 09:07:09.015][Trace][1][v374mj74] Hybrid cpu=0.00%,17MB, cid=2,2, timer=62,9,41, clock=0,47,2,0,0,0,0,0,0, free=1, objs=(pkt:103,raw:3,fua:1,msg:103,oth:99,buf:103)
[2021-11-01 09:07:09.015][Trace][1][v374mj74] RTC: Server conns=1, rpkts=(123,rtp:120,stun:1,rtcp:2), spkts=(4,rtp:0,stun:1,rtcp:3), rtcp=(pli:0,twcc:0,rr:1), snk=(97,a:48,v:48,h:0), fid=(id:0,fid:123,ffid:0,addr:1,faddr:123)
[2021-11-01 09:07:10.247][Trace][1][79sf4v65] <- RTC RECV #10, udp 1230, pps 4/122, schedule 1230
[2021-11-01 09:07:12.719][Trace][1][o094f950] NACK: update seq=6900, nack range [6898, 6900]
[2021-11-01 09:07:12.770][Warn][1][79sf4v65][11] handle udp pkt, count=32/30, err: code=5015 : size=997, data=[80 65 1a f2 e0 20 ff 7a] : marker=0, pt=101, seq=6898, ts=3760258938, ssrc=3803280376, pad=0, payload=985B : rtp unprotect r0=9
thread [1][79sf4v65]: cycle() [src/app/srs_app_listener.cpp:630][errno=11]
thread [1][o094f950]: on_rtp() [src/app/srs_app_rtc_conn.cpp:1380][errno=11]
thread [1][o094f950]: unprotect_rtp() [src/app/srs_app_rtc_dtls.cpp:1149][errno=11]
[2021-11-01 09:07:14.015][Trace][1][v374mj74] Hybrid cpu=1.00%,17MB, cid=2,2, timer=62,9,41, clock=0,47,2,0,0,0,0,0,0, free=1, objs=(pkt:103,raw:3,fua:1,msg:103,oth:99,buf:103)
[2021-11-01 09:07:14.015][Trace][1][v374mj74] RTC: Server conns=1, rpkts=(123,rtp:120,stun:1,rtcp:2), spkts=(4,rtp:0,stun:1,rtcp:3), rtcp=(pli:0,twcc:0,rr:1), snk=(97,a:48,v:48,h:0), fid=(id:0,fid:123,ffid:0,addr:1,faddr:123)

1. SRS configuration:

# main config for srs.
# @see full.conf for detail config.

listen              1935;
max_connections     1000;
#srs_log_tank        file;
#srs_log_file        ./objs/srs.log;
daemon              on;
http_api {
    enabled         on;
    listen          1985;
}
http_server {
    enabled         on;
    listen          8080;
    dir             ./objs/nginx/html;
}
rtc_server {
    enabled on;
    listen 8000;
    # @see https://github.com/ossrs/srs/wiki/v4_CN_WebRTC#config-candidate
    candidate $CANDIDATE;
}
vhost __defaultVhost__ {
    hls {
        enabled         on;
    }
    http_remux {
        enabled     on;
        mount       [vhost]/[app]/[stream].flv;
    }
    rtc {
        enabled     on;
        # @see https://github.com/ossrs/srs/wiki/v4_CN_WebRTC#rtmp-to-rtc
        rtmp_to_rtc off;
        # @see https://github.com/ossrs/srs/wiki/v4_CN_WebRTC#rtc-to-rtmp
        rtc_to_rtmp off;
    }
}

Replay

How to replay bug?

Steps to reproduce the bug

How to replay bug?

1. Install aiortc using Python and obtain rtc objects.
2. Use addTransceiver to add audio and video with sendonly.
3. Use aiortc MediaPlayer to obtain the video track and add it to RTCPeerConnection.
4. Use createOffer to set the local description.
5. Send a request to srs /rtc/v1/publish.
6. After receiving a successful response, generate an answer and set the remote description.
7. Play the web rtc player, but the video appears blurry.

Expected behavior

Expected behavior

Please describe your expectation. I expect the video to play clearly without any flickering.

TRANS_BY_GPT3

[xiaozhihong]

Take a screenshot. What is a "blob of paste"? Is it screen distortion or low bitrate blurring? Is it blurry playback or some other phenomenon?

TRANS_BY_GPT3

[xiaozhihong]

@robin2099 Can you conveniently provide the py script for this test?

TRANS_BY_GPT3

[robin2099]

@xiaozhihong This is the situation, I don't know if it can be defined as screen flickering.

TRANS_BY_GPT3

[robin2099]

@xiaozhihong client_srs.py.zip Python script, using local docker srs4 for testing

TRANS_BY_GPT3

[xiaozhihong]

@xiaozhihong client_srs.py.zip python script, using local docker srs4 for testing

@robin2099 I checked this script and it's really good and useful. I tested it and the performance is consistent with yours. This is not screen flickering, it's caused by low bitrate. How can I control the bitrate with the python av library? I tried changing the options to...

options = {"framerate": "30", "video_size": "1280x720", "bitrate": "8000000"}

However, it didn't work. If you know how to modify the encoding parameters and bitrate of libx264 in python, please test it and share it with us.

TRANS_BY_GPT3

[robin2099]

@xiaozhihong Okay, thank you for helping to locate the problem 🙏

TRANS_BY_GPT3

[CodeIsWorld]

@xiaozhihong Okay, thanks for helping to locate the problem. Pray

Have you solved the problem? I am also trying to use aiortc for streaming, referring to your solution.

TRANS_BY_GPT3

[robin2099]

@CodeIsWorld Not yet, I have tried many methods but still couldn't achieve a clear picture by modifying the bitrate.

TRANS_BY_GPT3

[qiubc]

Hello, do you have an example of using aiortc to stream and then displaying it with opencv? I tried to stream using aiortc, but it keeps failing o(╥﹏╥)o.

TRANS_BY_GPT3

[CodeIsWorld]

@CodeIsWorld I haven't found a way yet. I've tried many methods but still couldn't achieve a clear picture by modifying the bitrate.

I fixed this issue based on the answer to this problem https://github.com/aiortc/aiortc/issues/170. The bitrate is calculated by the server based on the available bandwidth and sent to the client. It seems that there is a mismatch in the negotiation between SRS and aiortc, resulting in a low resolution set on the client side. I modified the source code of aiortc for RTP sending in aiortc/src/aiortc/rtcrtpsender.py. async def _next_encoded_frame(self, codec: RTCRtpCodecParameters):

    frame = await self.__track.recv()

    if self.__encoder is None:
        self.__encoder = get_encoder(codec)
    force_keyframe = self.__force_keyframe
    self.__force_keyframe = False
    return await self.__loop.run_in_executor(
        None, self.__encoder.encode, frame, force_keyframe
    )

The bitrate is mainly set in self.encoder. The decoder type is set based on the codec type in the file src/aiortc/codecs/init__.py, line 144. I noticed that my encoder type is H264Encoder(). The most crucial encoding function is in the file src/aiortc/codecs/h264.py, line 268, in the function _encode_frame(). The minimum bitrate is set at line 17. DEFAULT_BITRATE = 1000000 # 1 Mbps MIN_BITRATE = 500000 # 500 kbps MAX_BITRATE = 3000000 # 3 Mbps

MAX_FRAME_RATE = 30 PACKET_MAX = 1300 If the bitrate sent by the server is lower than this minimum value, it will use this minimum value. So, directly increase the MIN_BITRATE minimum value, then compile the source code and install it (remember to first uninstall the previously installed aiortc with pip), sudo python3 setup.py install, and then run the client's source code to see if the bitrate can be increased.

*****update* After testing, there is still a problem. The line 282 if self.codec is None: is only None for the first time, and then it keeps using self.codec, causing issues with the bitrate. So, my solution is to force self.codec = None before line 282, and then set self.target_bitrate. Each frame that comes in will create a new encoder, ultimately resolving the issue as follows.

def _encode_frame( self, frame: av.VideoFrame, force_keyframe: bool ) -> Iterator[bytes]: if self.codec and ( frame.width != self.codec.width or frame.height != self.codec.height

we only adjust bitrate if it changes by over 10%

        or abs(self.target_bitrate - self.codec.bit_rate) / self.codec.bit_rate
        > 0.1
    ):
        self.buffer_data = b""
        self.buffer_pts = None
        self.codec = None

'''*Add these two lines*** self.codec = None self.target_bitrate = 500000 '''****''' if self.codec is None: try: self.codec, self.codec_buffering = create_encoder_context( "h264_omx", frame.width, frame.height, bitrate=self.target_bitrate ) except Exception: self.codec, self.codec_buffering = create_encoder_context( "libx264", frame.width, frame.height, bitrate=self.target_bitrate, )

TRANS_BY_GPT3

[robin2099]

@xiaozhihong I tried another method, which is converting the pixel format, resetting the pts and time_base. Although I'm not quite clear about the principle, it does work. I'm not sure if this approach has any issues. The test script has been uploaded. srs_client.zip 。' translates to '.' in English.

TRANS_BY_GPT3

[robin2099]

@qiubc you can refer to this solution on Zhihu: https://zhuanlan.zhihu.com/p/387772163. It might be helpful for you.

TRANS_BY_GPT3

[robin2099]

@CodeIsWorld Okay, I will try your solution and then come back.

TRANS_BY_GPT3

[winlinvip]

Thank you all for the investigation and analysis. It seems that defining it as insufficient clarity is more appropriate, and it requires adjusting the streaming parameters.

TRANS_BY_GPT3