Stability of RTMP Session

[usman3d]

Hi,

I have observed that VideoCore occasionally terminates the session because it receives kStreamEnd status .(RTMP status 11 is also often printed in the log before termination ). How to reproduce? When I run the recording session for longer time (such as 30+ or 40+ minutes), then recording session often terminates without giving any meaningful warning.

I believe this issue occurs because of: 1- Poor wifi connection In this case we don't get meaningful warning, just RTMP status number 11, and then termination 2- Memory leaks I believe memory leaks cause the crash of the recording session/app. At crash it shows me the call stack : parseCurrentData() -> handleInvoke() -> get_string() But I'm not sure if this is the exact part of code that is causing memory leak.

Can others verify if these issues can be reproduced at their end?

Thanks.

[jgh-]

I will attempt to reproduce - in the mean time do you have an instruments trace or anything showing the memory leak?

[jgh-]

or a stack trace would be useful too

[usman3d]

I'll try taking snapshot with instruments. Meanwhile please see attached stack trace

Thanks. Best, Usman

On Tue, Jul 8, 2014 at 1:45 PM, James Hurley notifications@github.com wrote:

or a stack trace would be useful too

— Reply to this email directly or view it on GitHub https://github.com/jamesghurley/VideoCore/issues/33#issuecomment-48271685 .

[jgh-]

oh your image didn't show up in the post here.

[usman3d]

Updated my comment above.

[germanSancho]

Hi,

I'm also encountering some RTMPStatus 11 and 10 errors when I stream at high bitrate (~2Mbps, over a wifi connection with fiber optics). Often they happen when I move the phone quickly so I guess they are due to dropped network packets...

[jgh-]

Yeah could be something funky going on with the network connection. A quickly changing scene doesn't compress as well as a relatively static scene so the network usage would go up. I'll do some tests to see if I can find the source of the trouble.

[jgh-]

OK So based on a few tests with the network limiter it looks like if you're trying to push down more data than the network will support for whatever reason you'll eventually be disconnected by the server with an RTMP status of 10 or 11.

Luckily it's fairly easy to measure the growth of the send buffer, so we could use that growth rate to sort of guesstimate the supported bitrate and pass that back to the session to adjust the encoders accordingly. I'll start off with doing it the rtcp way which is basically just "bandwidth up / bandwidth down" and then we can do something fancier down the road a bit.

ALSO it should be noted that if the buffer is growing due to a mismatch between how much the network will support and how much it's attempting to send there is currently no ceiling on buffer growth, unfortunately. So it will manifest itself similar to what a memory leak looks like - but it's all in the buffer.

[usman3d]

Hi, Was there any update on this issue? What kind of network limiter are you using? I might give it a try too. I have noticed there is an RTMP "Window Acknowledgement Size" of 2,500,000 (~2.4 MBytes) sent from the server, and then there is another win_size at TCP level that is 65,000 (~64KBytes). I thought 65,000 is the buffer size and 2,500,000 is the bandwidth limit. I'll do some experimentation on the buffer size and will see how does it effect the session.

[jgh-]

I haven't gotten around to doing the bandwidth adaption yet, but it is next on the list of features on my plate. That's the plan for solving this issue.

(as for the network limiter, you can use the one provided in iOS or the one in OS X if you use an ad hoc network)

[usman3d]

Hi James,

Was there any update to this issue? I'm trying to diagnose what bug causes the server to send the stream error or session disconnection status (11).

One thing I have noticed that the chunk size information we receive from the server is 512 bytes but our client is always using the fixed size chunk of 128 bytes. When I try to change the value from 128 to any other value, the client (RTMPSession) fails to send the 'connection' packet and just becomes idle. Then after few seconds the server pings the client and then disconnects.

What is the reason of choosing chunk size as 128 bytes? and is there any way to change it to make the performance better? (if overall bitrate stays much below the peer bandwidth received from the server). Any other parameters that I can change to tune the performance? (have tried changing the idle frequency at the server but did not see any difference).

Thanks, Usman

[usman3d]

Just to update. I have noticed when network is low bandwidth (I use a wireless egg device from telecom operator) then I notice too many "unknown packets" in the WireShark at the server side. Maybe when the network is unable to handle too many packets then the stream or the ring buffer at the client side corrupts the data?

[jgh-]

That is certainly possible, requires a bit more investigation though. I will hopefully be digging into the bitrate adaptation this week so I'll try to investigate this at the same time.

[usman3d]

I'm also wondering what was the reason to write our own code for RTMP streaming instead of using an existing implementation such as libRTMP?

[jgh-]

Initially my plan was to make some commercial projects with it, but then I decided to open source it instead since there isn't really much in the way of open source ios video streaming stuff (that takes advantage of the hardware, anyway, of course there's ffmpeg and such that will do software encode/decode perfectly well). So I ended up writing most of what's here now and then open sourcing it.

That said there's no reason that the RTMP session couldn't be swapped out with a librtmp implementation since the licenses don't conflict now.

[jgh-]

@usman3d I have merged in a simple adaptive bitrate method. Please enable this via the useAdaptiveBitrate property in VCSimpleSession (if that's what you're using, otherwise see VCSimpleSession.mm for an example of its use)

[usman3d]

Awesome. I'll check it soon. Thanks.

On Tue, Aug 19, 2014 at 6:26 AM, james h notifications@github.com wrote:

@usman3d https://github.com/usman3d I have merged in a simple adaptive bitrate method. Please enable this via the useAdaptiveBitrate property in VCSimpleSession (if that's what you're using, otherwise see VCSimpleSession.mm for an example of its use)

— Reply to this email directly or view it on GitHub https://github.com/jgh-/VideoCore/issues/33#issuecomment-52557741.

[usman3d]

Though one thing I was wondering is, what were the reasons to chose RTMP over other protocols such as HLS?

On Tue, Aug 19, 2014 at 12:47 PM, Usman virtualeye@gmail.com wrote:

Awesome. I'll check it soon. Thanks.

On Tue, Aug 19, 2014 at 6:26 AM, james h notifications@github.com wrote:

@usman3d https://github.com/usman3d I have merged in a simple adaptive bitrate method. Please enable this via the useAdaptiveBitrate property in VCSimpleSession (if that's what you're using, otherwise see VCSimpleSession.mm for an example of its use)

— Reply to this email directly or view it on GitHub https://github.com/jgh-/VideoCore/issues/33#issuecomment-52557741.

[jgh-]

@usman3d HLS doesn't provide a means to push a stream to a server. We could also use something like RTSP which is a popular stream publishing protocol as well. The idea is that this project is modular, so it would be a matter of creating a new OutputSession that uses whatever protocol is desired.

Have you managed to reproduce this issue? I have streamed for quite a long time without seeing this problem popping up, so I'm wondering if this is safe to close

[jgh-]

I found an additional bug in the RingBuffer code that may have caused this issue. Please retest with 0.1.10.3 @usman3d

[usman3d]

Aye. Got busy in another project. Will get back to this soon and test it. Thanks :)

On Wed, Sep 10, 2014 at 9:05 AM, james h notifications@github.com wrote:

I found an additional bug in the RingBuffer code that may have caused this issue. Please retest with 0.1.10.3 @usman3d https://github.com/usman3d

— Reply to this email directly or view it on GitHub https://github.com/jgh-/VideoCore/issues/33#issuecomment-55054537.

[jgh-]

Closing this for now. Changes that have affected this issue:

• Adaptive Bitrate is much more stable
• Fixed a bug in the RingBuffer the RTMP session is using.
• Session now properly sets the outgoing chunk size and correctly handles an incoming chunk size change. Currently set to 1536, this value seems to work well.

[chinnuios]

@usman3d Hi, Did you solved issue ? if yes, please tell me