Video compression support while streaming H.264 video

[ghost]

Hi,

Currently, I stream an H.264 video from my Android device (running OS ver 4.4 or higher) to a Wowza media server using libstreaming.

In conditions where the network strength is low (on a 3G or 2G network), I want to compress the video on the fly and send the compressed stream. In an ideal situation, for example a 4G network, my streaming works fine. However, the issue is when the network on the mobile device is weak.

So what can be done in such situations? Does libstreaming include some video compression support that automatically compresses the video in low network conditions? If not, how can we include any video compression algorithm in the latest libstreaming source code.

Regards, Karan

[sbaar]

RTSP does not support compressing any more. What you can do is adjust the bitrate lower when network conditions are bad and vica versa. Here is a modified rtpsocket.java and bitrate controller to accomplish that https://gist.github.com/sbaar/c6dc432c7d175225f53b4c7c9119e02a

[ghost]

Hi Scott,

Does libstreaming currently supports the following features?

1) If the network condition is bad, does the resolution automatically reduce along with the quality? 2) Are you using H.264 AVC or H.264 SVC. Since the latter is the more preferable in conditions where network is bad since most established video streaming products in the market are using H.264 SVC

Regarding the bitrate, how does setting the bitrate going to help carry out streaming in bad network conditions. Can you explain how this works?

Regards. Karan

[sbaar]

1) No. The resolution is constant and out of the box libstreaming the bitrate is constant too. 2) According to android supported media formats only avc is supported, not svc encoding.

When the network conditions are bad, the phone can't send as many bits out as the encoder is producing. The BitrateController.java and modified RtpSocket.java lower the bitrate of the encoder on the fly if it senses that the buffer is getting full. The resolution is constant, but the quality drops because there are less bits to construct the picture with.

[ghost]

Hi Scott,

Thanks for the quick reply.

I had just one quick doubt. Consider the following scenario.

I am using libstreaming (out of the box) in one of my projects right now. The streaming works fine on 4G and WiFi. When the network switches to 3G, the streaming continues, however I observe that there is a delay between the sending of frames. It's like if a move my phone in one direction, I see a frame and then may be after a considerable / noticeable amount of delay I see the second frame. Here by frame I mean the picture being constructed.

So to solve the above problem, do you want me to write some more code in the BitrateController.java and RtpSocket.java? or do you want me to use the existing code itself and based on my network conditions appropriately set the bitrate?

Can you answer these questions for me?

Thanks

[sbaar]

First verify that the issue you see is caused by too high of a bitrate. Try changing the bitrate in your videoconfig to 1/5 of what it is then try to reproduce the problem. If the problem went away, then it is because your bitrate is too high for 3g, and you should activate the bitrate controller after streaming starts using the above code.

If the problem does not go away then it means there is likely another problem. If your setup uses TCP then this could cause the delay so try switching to UDP. If you run into trouble here, search the issues because a lot has been said about how to get UDP working.

[ghost]

Hi Scott, Can you also prove the Utils.java file since I am getting an error for the clamp method.

import com.vg.hangwith.Utils;

int adjustedBitrate = Utils.clamp(newBitrate,MIN_BITRATE,MAX_BITRATE);

I am trying to implement the changes suggested by you.

[sbaar]

Clamp is just force within that range. Debug is just log On Apr 1, 2016 10:24 PM, "Karan Balkar" notifications@github.com wrote:

Hi Scott, Can you also prove the Utils.java file since I am getting an error for the clamp method.

import com.vg.hangwith.Utils;

int adjustedBitrate = Utils.clamp(newBitrate,MIN_BITRATE,MAX_BITRATE);

I am trying to implement the changes suggested by you.

— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/fyhertz/libstreaming/issues/196#issuecomment-204649193

[ghost]

Hi,

I fixed the Debug part.

"Clamp is just force within that range"

I did not understand this. How is the clamp function defined?

[sbaar]

Sorry about that. Clamp(a,b,c) is math.max(b,math.min(a,c) ) to ensure a is within the range b and c. On Apr 1, 2016 10:36 PM, "Karan Balkar" notifications@github.com wrote:

Hi,

I fixed the Debug part.

"Clamp is just force within that range"

I did not understand this. How is the clamp function defined?

— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/fyhertz/libstreaming/issues/196#issuecomment-204650246

[ghost]

Ok thanks a lot Scott.

Here is my final code. It will be really helpful if u can review the same.

PS : I don't want to use a Seekbar or any UI widget. So I am passing null currently

MainActivity.java file BitrateController.txt MainActivity.txt RtpSocket.txt

Suggest changes if any so that I can go ahead and test this in my environment

[sbaar]

Bitrate Controller is not clear enough on it's own, I should have added comments. Don't bother with the network broadcast receiver. Just checking if you're on 2g or 3g or 4g is insufficient, because you can be on 3g with 1 bar and have a bad connection. Instead, create the bitrate controller in onSessionStarted and it will take care of measuring the connection itself. Get the MediaCodec out of the session.getVideoStream() and get the rtp socket the same way. You will have to make the rtpSocket and MediaCodec public in VideoStream or h264 stream or write methods to expose them.

Also, your bitrate currently is way too high. Looking at the supported media format page shows SD quality supported on all devices to be 480x360 30 fps 500000 bps. I would start with that since it's guaranteed to work on all devices. Even good devices like the nexus 7 have trouble with 2Mbps bitrates in my experience.

[ghost]

Hi,

I have removed the network broadcast receiver and initialized the bitrate controller in onsessionStarted.

However I did not get any getVideoStream() defined in the VideoStream or H.264Stream java files. Also, I did not find any MediaCodec variable used in VideoStream or H264. So how do I make it public and expose them?

[sbaar]

The Session has a VideoStream and the VideoStream (actually the base class Media Stream) has the mediacodec and a packetizer. The packetizer has the rtp socket you're looking for. Go ahead and make what you need public to get all that.

[ghost]

Hi,

Please check below modified code as per your comments.

MainActivity.txt

this is how i fetch the MediaCodec. wrote this in VideoStream.java

/* * Created by karan * This function is used to get the current mediacodec used * @return / public MediaCodec getMediaCodec(){ try{ EncoderDebugger debugger = EncoderDebugger.debug(mSettings, mQuality.resX, mQuality.resY); mMediaCodec = MediaCodec.createByCodecName(debugger.getEncoderName()); }catch(Exception ex){ ex.printStackTrace(); } return mMediaCodec; }

[sbaar]

There is no need for that. You want the media codec object that is created when the session starts, not to create your own.. Look in MediaStream.java and get the mMediaCodec object that is already there, along with the packetizer which has the rtpsocket you want.

[ghost]

Yes you are right. I was actually creating a new object which is wrong. So now I have got it in place with the following line

mBitrateController = new BitrateController(mSession.getVideoTrack().mMediaCodec, 500000 , mSession.getVideoTrack().getPacketizer().getRtpSocket(), null);

So what do you reckon should I go ahead and test this now?

[ghost]

Hi Scott,

I tested the same on a Motorola G device.(1st generation) I am getting an alert popup saying setParameters failed.

Here is how my session is configured

mSession = SessionBuilder.getInstance() .setContext(getApplicationContext()) .setAudioEncoder(SessionBuilder.AUDIO_AAC) .setAudioQuality(new AudioQuality(8000, 16000))
.setVideoEncoder(SessionBuilder.VIDEO_H264) .setVideoQuality(new VideoQuality(480, 360, 30, 500000)) .setCamera(CameraInfo.CAMERA_FACING_BACK) .setSurfaceView(mSurfaceView).setPreviewOrientation(90) .setCallback(this).build();

[ghost]

Hi,

I managed to solve that my changing the VideoQuality

.setVideoQuality(new VideoQuality(640, 480, 30, 500000))

This solved the popup issue. However, when the network was bad, I could still see a large amount of delay in the playback of the video stream on Wowza Server.

Is the bitrate code that we added actually making any effect?

[donghenglong]

Hi Scott,

I need to pack H264 into PS streams （MPEG2-PS）。 Does H264 streams are packaged into PS streams (MPEG2-PS) ? I read about H264 packaged PS implemented in C++, If I use your library to pack H264 into PS streams, can I achieve it?

Regards. Charlie

[sbaar]

@donghenglong This is not my library and is beyond the scope of libstreaming. If you email me I might be able to give some advice on what to do to do that server side.