Lengthy blocking in outputStream.write at RtpSocketTcp sendTCP

[eon-yogiwp]

We're using RtspDisplay to stream video (no audio) from mobile devices to local Wowza streaming engine server. Occasionally outputStream.write would seemingly block for a lengthy period of time, causing hiccups during streaming. We do not notice this issue when using Wowza's goCoder SDK, so my guess is it's not a problem on the server.

Wondering what may cause this and how do we resolve it?

[pedroSG94]

device model and API version? I need check it and try reproduce. For now, try with different resolutions and video bitrate also test other dpi.

[eon-yogiwp]

It seem to occur on all devices I've tested: HTC One A9, Motorola Nexus 6, Galaxy S9 Edge, and Vuzix m300. Our target is only the Vuzix m300 smart glasses, the other devices are just for sanity checks.

minSdkVersion 23 targetSdkVersion 27

Lowering resolution and bitrate (much, much lower) does seem to reduce the issue. Minor hiccups seem to occur still but I didn't see similar blocking outputStream.write when profiling.

[pedroSG94]

Maybe it isn't your problem but remember MediaProjection doesn't render surface all time only when any change in the screen happens. Read this issue and emulate solution(use notification code in it): https://github.com/pedroSG94/rtmp-rtsp-stream-client-java/issues/63

Also other solution could be use UDP stream, anyway I will check it.

[eon-yogiwp]

We have a camera preview (and some extra graphics on top) so MediaProjection does render all the time.

I did try UDP stream and it doesn't seem to work, but it could be server configuration problem. I';ll need to look into that. Just to be clear, does RTSP + UDP is expected to work with Wowza streaming engine? (Probably a simple question but I'm new to all of this).

[pedroSG94]

Yes, UDP should work. I did a test and it is working with wowza server. Please let me know parameters used in audio and video to reproduce your case more easy, I will do it in the next days.

[eon-yogiwp]

The parameters are as follows. We do not use audio. Thanks for looking into this.

mRtspDisplay.disableAudio(); mRtspDisplay.setForce(CodecUtil.Force.HARDWARE, CodecUtil.Force.FIRST_COMPATIBLE_FOUND);

mRtspDisplay.prepareVideo(1920, 1080, 30, 8192000, 0, 320) OR: mRtspDisplay.prepareVideo(1280, 720, 30, 3640889, 0, 320)

[pedroSG94]

I was doing multiple test and no problem in sendTCP method. I did tests with 4k resolution in RTSP and 10Mb bitrate in local server working(with a S7 using camera2). Of course I tested your config with rtsp display mode and I see low framerate (fps). I think it is your problem, right? Maybe you mean other thing with "hiccups" but I understand low fps. If not let me know it. My conclusion with this result is that the problem is MediaProjection performance because sendTCP is working with more bitrate and resolution. outputstream.write block is the time that your device need to send a frame to server, this depend of your network card, cpu, bandwith and frame size (size increase with bitrate), bu in this case I think "hiccups" is because MediaProjection can't render surface with a good frame rate.

For now, I can recommend you this:

• You are using high bitrate. If you want use 1080p and 720p this should be good: 1280x720(resolution) -> 800 1024(bitrate) 1920x1080(resolution) -> 2500 1024(bitrate)

• MediaProjection maybe is bad with high resolution so test with 480p and contemplate if this quality is good for you. In this resolution you can use bitrate like: 600 * 1024 with good results.

• Increase dpi to get more quality. I tested with 720 dpi and you can see so much difference (of course it depend of you device screen dpi).

I will investigate more about MediaProjection and performance of it to know if I can do something.

[eon-yogiwp]

Thanks for looking into this. I've just shared a googledrive video showing the hiccup problem to your email address. It's not low framerate, the framerate is okay but we get occasional freezes. I am currently doing more investigations and test your recommendations, I'll update you soon.

[pedroSG94]

I saw the video. I will try emulate your activity and debug this case.

[eon-yogiwp]

Today I found out that the problem may be with this one particular server that we use. (I mentioned in the original post that it's a local server, but I meant local EC2 server). This server is shared with our other lightweight services but there may be some conflicts happening. For some reason we could never get UDP to work on this server. The blocking TCP may be caused by it not being able to consume the stream for some reason.

We tried another EC2 server (non-local, we're in Singapore and this server is in Oregon) and UDP works but everything seem to be lagging/slower in general thus the test was inconclusive.

Finally we set up a new Wowza server on local network, and with this UDP works and we're not getting the freezes so far. We seem to get much less freezes with TCP, but I have not done much test in this configuration. I'll do more test tomorrow and see whether blocking TCP issue is a general problem or just a problem with a particular server.