Integrate G.711 and H.264 available with Flash Player 11

[GoogleCodeExporter]

Flash Player 11 (in beta) has new audio and video codecs support from 
microphone and camera captured data. Integrated this with siprtmp/VideoPhone 
and Flash-VideoIO.

Original issue reported on code.google.com by kundan10 on 27 Jul 2011 at 4:53

[GoogleCodeExporter]

Reported by Saul Diaz ncripito@gmail.com on Jul 26, 2011
---

I have being using  SIPRTMP with some success in some of my
applications.

Flash 11 include G.711 codec. you guys plan to modify the gw to take
this in considerancion?

or have any recomendation what needs 2 be done to make it work?
---

Thanks for the kind words...

We will certainly look into integrating G.711 (and H.264) available in
Flash 11...

If you wish to try out adding G.711, here are the suggested steps:

1. Find out how to enable G.711 from Microphone in ActionScript? Is it
as simple as Microphone's codec="pcmu"?
http://help.adobe.com/en_US/FlashPlatform/beta/reference/actionscript/3/flash/me
dia/SoundCodec.html
Modify VideoPhone client application to use G.711.

2. Detect what first byte of audio payload is when Flash Player sends
PCMU to rtmplite. In siprtmp.py's rtmp_data function, the first byte
message.data[1] is assumed to be '\xb2' for speex. Assuming the format
is similar to previous ones except that the first byte is different,
you can just extract the payload and use that.

3. In the reverse media path, use sip_data function to put the correct
first byte instead of '\xb2'.

4. Add the payload type correctly in the _get_sdp_streams function so
that it uses G.711.

5. If you wish to use both Speex and G.711, then need to add more code
in the negotiation, and select the appropriate payload format after
negotiation.
---

Thank a lot for the steps.

I got it working

the first byte is 0x82

the changes really minors. i use rtmplite 6.0

line 471.
self._audio, self._video = format(pt=-1, name='PCMU', rate=8000),
format(pt=-1, name='x-flv', rate=90000)

line 645
 payload = '\x82' + p.payload

regards
saul

Original comment by kundan10 on 27 Jul 2011 at 4:54

[GoogleCodeExporter]

The G.711 and H.264 should work with rtmp.py (Just the RTMP server). It does 
not yet work with siprtmp.py (SIP-RTMP gateway). G.711 is possible using the 
previous workaround, or using the external py-audio module (see 
code.google.com/p/py-audio for details). 

I attempted integrating H.264 in siprtmp.py using Ekiga and/or Bria 3 on SIP 
side and Flash Player 11 beta (using VideoIO) on web side, but could not get it 
to work well. I could get one way video from Flash to SIP (Bria 3) but the 
video was very jerky indicating that SIP side was able to decode only the intra 
frames. With Ekiga, the decoded video was corrupted. I couldn't get Flash to 
display the video from any of SIP side. The main challenge is in converting the 
RTP/payload format of SIP side to/from Flash Player's custom formatting and 
signaling of parameter sets.

If anyone is interested in continuing the work, I will be happy to give out the 
existing modifications that I have made in siprtmp.py. Send me an email to the 
support group.

Original comment by kundan10 on 6 Sep 2011 at 7:28

[GoogleCodeExporter]

Issue 38 has been merged into this issue.

Original comment by kundan10 on 6 Sep 2011 at 7:29

[GoogleCodeExporter]

After changing to
   self._video = format(pt=-1, name='PCMU', rate=8000),
and
   payload = '\x82' + p.payload

Sound with PCMU works great when connecting from flash player 11, but how to 
use video with h.264?

Also when I try to attach to video remote netStream with h.264 video in 
rtmplite console i see message:
   "format not found for pt= 99 Invalid RTP parse error 'NoneType' object has no attribute 'name'"

To use h.264 in flashplayer 11 you need to add this line:
var h264Settings:H264VideoStreamSettings = new H264VideoStreamSettings();
h264Settings.setProfileLevel(H264Profile.BASELINE, H264Level.LEVEL_2);
netStream.videoStreamSettings = h264Settings; //local netStream

Original comment by kko...@gmail.com on 7 Sep 2011 at 3:56

[GoogleCodeExporter]

To avoid the NoneType exception, can you please change in siprtmp.py to 
following. Or accordingly for siprtmp_gevent.py if you use that version.

     def received(self, media, fmt, packet): # an RTP packet is received. Hand over to sip_data.
-        multitask.add(self.sip_data(fmt, packet))
+        if fmt is not None:
+            multitask.add(self.sip_data(fmt, packet))

Your client side changes look good.

I am also attaching the full siprtmp.py (not checked in yet) which is 
modification on top of svn r106 to enable H.264 between SIP and Flash sides by 
changing between Flash's formatting and SIP's RTP payload format for H.264 as 
per RFC 3984. Most of the changes are in h264avc_ functions. It uses the 
version# of the Flash side, and if >= 11, it adds H.264 in SDP. In SIP to Flash 
direction, it caches the SPS and PPS packets, and sends the AVC seq packet to 
Flash when both are received. It avoids sending any media slide packet until 
AVC seq is sent. It also adds appropriate headers to the NALU when sending to 
Flash side. In Flash to SIP direction, it strips off the Flash headers from 
NALU and if needed adds the fragmentation NALs because sending large UDP 
packets to SIP fails in socket.sendto call.

As indicated in my previous post, it doesn't work well in my test. I get 
jittery video playback in SIP/Bria 3 and no video playback in Flash. There is 
some commented out code which enables loopback of video, and works fine.

These information will help anyone who wishes to tackle the problem. 
Unfortunately, due to my other work, I don't get to spend a lot on this 
project. And this problem/issue requires long dedicated effort to fix.

Original comment by kundan10 on 7 Sep 2011 at 6:09

Attachments:

• siprtmp.py

[GoogleCodeExporter]

After playing around with the RTP marker bit sent to the SIP side, I was able 
to get continuous Flash to SIP video. In particular, the marker is True for all 
packets unless there is fragmentation in which case it is False for fragmented 
packets except for the last for which it is True. 

The reverse direction video is still is mystery.

Original comment by kundan10 on 8 Sep 2011 at 5:29

[GoogleCodeExporter]

I test new siprtmp.py you attached, but there are now some problems. I don't 
hear anything now while receiving stream "remote". When I connect with other 
phone (hardware video sip) i that second phone I hear only noise and crackle. I 
see video send from flash11 but it's quality is very poor, and when I change 
resolution in flash to 720p video freezes. Logs from asterisk 1.8 in attachment.

Thank you for your help.

Original comment by kko...@gmail.com on 8 Sep 2011 at 7:06

Attachments:

• asterisk_log.txt

[GoogleCodeExporter]

Sorry, I did not change sound codec "self._audio, self._video = format(pt=-1, 
name='PCMU', rate=8000)".

Now Asterisk warning dissapeared, and on second hard phone I can see video and 
hear voice (bad quality, byt in some way it works). Now in flash I hear noise 
and crackle (before I hear nothing). I also include screen from siprtmp window. 
It was showing this message before also.

Original comment by kko...@gmail.com on 8 Sep 2011 at 7:27

Attachments:

• rtmplite_log.png

[GoogleCodeExporter]

Attaching another siprtmp.py that fixes video quality in Flash to SIP 
(Bria/Ekiga) direction as mentioned in my previous comment. The reverse 
direction SIP to Flash video is still a mystery. Please understand that this is 
work in progress, and haven't tested audio sound correctly with this. It might 
be better to use speex/16000 for now instead of PCMU with this attached 
siprtmp.py

Original comment by kundan10 on 8 Sep 2011 at 5:47

Attachments:

• siprtmp.py

[GoogleCodeExporter]

Hi Kundan and others,

as H.264 requests a minimum frames and buffer to start correctly maybee the 
problem comes from that ? Also maybe depend the H264 encoding...

Original comment by pratn...@gmail.com on 8 Sep 2011 at 9:32

[GoogleCodeExporter]

Adding a test file that prints the parsed NAL unit for SPS and PPS. This is 
useful in comparing the attributes of SPS/PPS from Flash Player and from 
Bria/SIP.

Original comment by kundan10 on 14 Sep 2011 at 3:46

Attachments:

• test_h264avc.py

[GoogleCodeExporter]

how to use test_h264avc.py? Receiving sound from hardphone in PCMU works great 
now. Quality from sip to rtmp is very good.

Original comment by g.koc...@systemycallcenter.pl on 20 Sep 2011 at 8:26

[GoogleCodeExporter]

test_h264avc.py is just a test file to parse SPS and PPS NAL units of H.264. I 
used it for debugging/seeing incompatibility between Flash vs SIP side's H.264. 
I haven't checked in the latest version yet. If someone has any idea about why 
SIP (Bria 3) side's H.264 is not playing on Flash Player, feel free to let me 
know. Thanks.

Original comment by kundan10 on 20 Sep 2011 at 5:12

[GoogleCodeExporter]

about siprtmp_gevent with revision 106:
- should it needs to change manually the sdp or siprtmp does an automatic codec 
negotiation ?
- same question for video fmt ...

thanks

Original comment by pratn...@gmail.com on 25 Sep 2011 at 8:06

[GoogleCodeExporter]

Hey, Is bidirectional H.264 working for you? I haven't fixed the Flash to SIP 
side video yet...
For siprtmp_gevent version, you will need to do the same set of changes that 
are suggested in this issue for G.711 and H.264...

Original comment by kundan10 on 26 Sep 2011 at 4:40

[GoogleCodeExporter]

Hi Kundan,
didn't test it yet, I'm actually trying to modify the transcode function for 
test
with FP11 rc1, once it will work I will test the video...

Original comment by pratn...@gmail.com on 26 Sep 2011 at 5:37

[GoogleCodeExporter]

I succeed to make receive stream in pcmu work well, but the flash publised 
stream not.
I got a crappy hashed and slow low tune sound.
I tried to change rtmp_data but no success.
also I don't understand the concept of self._audio.
maybe the best would be to check the first bit of the published stream and 
adapt the right sdp ?

thanks

Original comment by pratn...@gmail.com on 26 Sep 2011 at 8:28

[GoogleCodeExporter]

ok found the prob. flash seems to send default packet frame to 10ms for 1 frame.
I needed to increase framesesperpacket to 2 or divide the audio flow by 500 
rather than 1000 in rtmp_data

Original comment by pratn...@gmail.com on 27 Sep 2011 at 2:05

[GoogleCodeExporter]

Hey,I can receive h264 from sip to flash(version 10,not 11).but the first frame 
is very slow when connection is established(about 20 second).

Original comment by tiny...@gmail.com on 27 Sep 2011 at 2:16

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

tiny,
it sounds like a buffer problem... H264 needs a minimum of frame to start.. I 
guess that's the prob. unless to increase th buffer I really don't know how to 
resolve this with flash

Original comment by pratn...@gmail.com on 27 Sep 2011 at 2:20

[GoogleCodeExporter]

Hi tiny...@gmail.com, Seeing some video is definitely good starting point, so 
that we can work on fixing the problem. Looks like that is some timestamp 
conversion problem from SIP to Flash side. I was thinking that it was AVC 
SPS/PPS conversion problem. Which SIP client are you using for this? 

Hi Pratn...@gmail.com, that is a good find. Many SIP side systems assume 20ms 
G.711 audio. For speex I assumed framesperpacket of 1 gives 20ms data. But if 
it was different for G.711 that would definitely cause problem.

Regarding minimum # of frames to start, it that the case even with Baseline 
profile without B-frames?

Original comment by kundan10 on 27 Sep 2011 at 5:01

[GoogleCodeExporter]

Kundan,

Flash seems to send 20ms frames when you publish a stream as default.
but in case of you specify the property framesPerPack to 1 so it means 10ms and 
not 20ms like speex, thus to (I think) respect the standard rules

Original comment by pratn...@gmail.com on 27 Sep 2011 at 5:17

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

To: kundan10
My SIP client is LinPhone.

(Flash Player10 can work fine,but 11 can not. So I thing Player11 has some 
unknown problem).

Original comment by tiny...@gmail.com on 28 Sep 2011 at 6:49

[GoogleCodeExporter]

to:kundan10
I change the code in h264avc_rtp2rtmp from

payloads = []
first = packets[0]
all = [x for x in packets if x.ts == first.ts]
packets = [x for x in packets if x.ts != first.ts]
nalType, nri = ord(first.payload[0]) & 0x1f, ord(first.payload[0]) & 0x60
tm = int((first.ts - self.avcRtp['startTs']) * 1000 / self._h264avc.rate)

to

payloads = []
first,nalType, nri = None,None,None
for a in packets:
    first = a
    nalType, nri = ord(first.payload[0]) & 0x1f, ord(first.payload[0]) & 0x60
    if nalType == 5 or nalType == 1:
        break
all = [x for x in packets if x.ts == first.ts]
packets = [x for x in packets if x.ts != first.ts]
tm = int((first.ts - self.avcRtp['startTs']) * 1000 / self._h264avc.rate)

and commit the line 557

#if int(self.client.agent.flashVer.split(' ', 1)[1].split(',')[0]) >= 11: # 
version 11+ has H264Avc

now the first frame is fine(from sip to flash player 10 client). 

Original comment by tiny...@gmail.com on 28 Sep 2011 at 8:25

[GoogleCodeExporter]

Thanks a lot. That is very helpful. I will give it a try soon.

Original comment by kundan10 on 28 Sep 2011 at 4:34

[GoogleCodeExporter]

I verified that your changes work with Bria 3 (SIP phone) as well, i.e., Bria 3 
to Flash Player 10 video works with H.264, but Bria 3 to Flash Player 11 video 
does not work. 

In terms of differences, I discovered that Adobe Video File Format 
Specification version 10 says that AVCVIDEODATA can have one or more NALU from 
frames or slices and full frames are not strictly required, but version 10.1 
says that AVCVIDEODATA can only have one of more NALU from full frames and full 
frames are required. Perhaps this makes a difference if Flash Player 11 
implements version 10.1 of the spec whereas previous Flash Player version 
implements version 10 of the spec. (?)

Original comment by kundan10 on 29 Sep 2011 at 7:55

[GoogleCodeExporter]

Some results after more experiment:

1) I dumped the messages sent to Flash Player in an FLV file. The FLV file 
played fine in Flash Player 10.0, but on Flash Player 11 it didn't play first 
time. Once the first play cycle completed, clicking again on play to re-play it 
worked fine. This seems to indicate that the timing data sent by siprtmp to 
Flash Player 11 is probably not understood by Flash Player 11.

2) Flash Player itself sends SEI with pic_timing for every frame. On the other 
hand, Bria doesn't send any SEI. I tried to replicate similar stuff in siprtmp 
by generating and sending SEI (and setting pic_struct_present_flag in SPS) but 
that didn't help.

Original comment by kundan10 on 29 Sep 2011 at 10:33

[GoogleCodeExporter]

Also, that FLV was played fine by VLC on first attempt, so VLC seems to 
understand the timing data correctly.

Original comment by kundan10 on 29 Sep 2011 at 10:36

[GoogleCodeExporter]

I tried with linphone and its H264 plugin but with FP 11 rc1 but no success.
also Kundan I didn't find the code specified by Tiny above in your attached 
siprtmp.py in h264avc_rtp2rtmp function.

Original comment by pratn...@gmail.com on 29 Sep 2011 at 2:42

[GoogleCodeExporter]

Hey, I haven't uploaded the new siprtmp with those changes by tin..@, because 
it doesn't work with FP 11 yet. If you can please send me a reminder on 
kundan10@gmail.com and I will send the code by email. Thanks

Original comment by kundan10 on 29 Sep 2011 at 8:03

Attachments:

• siprtmp.py

[GoogleCodeExporter]

If I change to non-live mode, I can play H.264 from SIP to Flash 11 direction. 
The non-live mode is triggered either by sending user_control message of 
streamIsRecorded from server to client, or by setting bufferTime greater than 0 
on the client's play NetStream. Unfortunately, non-live mode introduces 
inherent delay in the stream of at least 4 seconds, which makes it useless for 
interactive communication.

Given that existing code works in playing back on Flash 10.3, I suspect this is 
a bug in Flash 11. One speculation could be that since Bria 3 has a slice 
instead of full frame per NALU, the Flash Player needs to store multiple NALU 
to construct a frame to display. Without buffering it is not possible, so it 
doesn't play in live mode with bufferTime of 0. Just some guesses...

Original comment by kundan10 on 2 Oct 2011 at 6:10

[GoogleCodeExporter]

it's also what I guessed unfortunately.

Original comment by pratn...@gmail.com on 2 Oct 2011 at 7:40

[GoogleCodeExporter]

seems that FP11 is officially out today (11.01.152)
let's make test with this release version...

Original comment by pratn...@gmail.com on 4 Oct 2011 at 6:26

[GoogleCodeExporter]

I logged a bug with Adobe about this.

Also tested with Ekiga on windows, and seems to work in showing its moving ball 
animation to Flash 11. Ekiga sends one slice NALU per frame (used 
packetization-mode=1 in SDP).

I will check-in the code in the next few days to include the changes as well as 
workaround of buffer-time (with delay).

Original comment by kundan10 on 6 Oct 2011 at 5:07

[GoogleCodeExporter]

ok Kundan, I hope the buffer won't exceed 100ms ! :)

Original comment by pratn...@gmail.com on 6 Oct 2011 at 3:41

[GoogleCodeExporter]

Fixed in svn r115 (gevent/untested) and r114 (multitask).

There is still an issue with Flash Player in case of certain SIP phones such as 
Bria which sends multiple slice NALUs per frame.
https://bugbase.adobe.com/index.cfm?event=bug&id=2991202
Added right click option in VideoPhone to add play buffer which fixes the issue 
albeit with a huge delay.
The VideoPhone application is modified in r112 to allow selecting a list of 
preferred codecs to use in a call. 

Need more testing.

Original comment by kundan10 on 18 Oct 2011 at 11:04

• Changed state: Fixed

[GoogleCodeExporter]

The bug has been fixed in Flash Player 11.2.202.96 (beta) which you can 
download an try out. I also checked in a fix in svn r125 of rtmplite to use the 
correct timestamp offset for multiple media streams.

I just tried the video call between Bria 3 and this Flash Player version, and 
works great!

Original comment by voiprese...@gmail.com on 12 Dec 2011 at 11:25

[GoogleCodeExporter]

excellent and good job Kundan,
very good new :D
will try it as soon as possible

Original comment by pratn...@gmail.com on 13 Dec 2011 at 12:04