Improve performance hit caused by conversion and resampling on iOS

EricssonResearch / openwebrtc

A cross-platform WebRTC client framework based on GStreamer

http://www.openwebrtc.org

BSD 2-Clause "Simplified" License

1.8k stars 537 forks source link

Improve performance hit caused by conversion and resampling on iOS #144

Open ikonst opened 9 years ago

ikonst commented 9 years ago

A loopback test app takes considerable CPU time on iOS (iPhone 5s) due to performance hit caused by audio sample conversion (F32LE to S16LE) and resampling (44,100 to 48,000 for Opus, 44,100 to 8,000 for PCM). Xcode profiler confirms these elements to be the offenders.

I've narrowed it down by transmitting only audio, disabling DTLS and using the PCMA codec with 8000 clock-rate. One would think loopback of 8KHz PCM would be an easy task for modern hardware.

ikonst commented 9 years ago

On iOS, the audiosrc's src pad offers fixed capabilities of:

audio/x-raw
format: F32LE
layout: interleaved rate: 44100 
channels: 2
channel-mask: 0x0000000000000003

From a cursory look at osxaudio (gst-plugins-good/sys/osxaudio/gstosxaudiosrc.c, function gst_osx_audio_src_probe_caps), it returns fixed caps based on the input stream's current format (i.e. querying kAudioUnitProperty_StreamFormat). This is unlike, for example, the ALSA source element which provides a wide range of possible formats.

1) We're using Remote IO, and Apple says we cannot expect it to convert or resample for us. Therefore, is it the Right Thing for osxaudio source not to offer flexible caps?

2) If so, what kind of performance hit is expected from a 44,100 F32LE to 8,000 S16LE conversion? What we're seeing now seems way high to me, but I'm inexperienced with this.

sdroege commented 9 years ago

In latest git master of gst-plugins-bad, it can output many more formats than F32LE. It will IIRC output whatever the audio session is configured to.

So for 1), yes that's correct. For 2), what are the numbers? How expensive is it?

stefanalund commented 9 years ago

@sdroege CPU usage outlined in this thread: https://groups.google.com/forum/#!topic/openwebrtc/bZJfu5CQ33U

ikonst commented 9 years ago

btw, owr_local_media_source_request_source doesn't use the caps for OWR_MEDIA_TYPE_AUDIO at all. Is it supposed to be like this, i.e. are the caps somehow propagated back to the source as "preferred" caps? (I'm a gstreamer newbie, so I really don't know.)

ikonst commented 9 years ago

It appears that osxaudio retrieves zero sampling rate from AU (at least on Remote IO), and so sampling rate ends up being GST_AUDIO_DEF_RATE (= 44100).

Following IRC discussion:

@ford-prefect will implement real probing ("preferred_caps") for Remote IO, based on kAudioSessionProperty_CurrentHardwareSampleRate etc.
It should be documented that OS X and iOS developers should set up AVAudioSession in their app code.
owr should warn when conversion and/or resampling would occur on OS X and iOS, and recommend setting up AVAudioSession.

ikonst commented 9 years ago

@ford-prefect I took care of this -- can you review https://bugzilla.gnome.org/show_bug.cgi?id=743758 ?

This reduces iPhone 5s CPU usage to 14% (PCMA) and 35% (Opus 1.0), which is pretty good :)

ford-prefect commented 9 years ago

Sweet! I'll try to get a review of that done today. A quick look suggests we're still working with F32LE? I guess the next step would be to probe that too.

On 31 January 2015 at 07:52, Ilya Konstantantinov notifications@github.com wrote:

@ford-prefect https://github.com/ford-prefect I took care of this -- can you review https://bugzilla.gnome.org/show_bug.cgi?id=743758 ?

This reduces iPhone 5s CPU usage to 14% (PCMA) and 35% (Opus 1.0), which is pretty good :)

— Reply to this email directly or view it on GitHub https://github.com/EricssonResearch/openwebrtc/issues/144#issuecomment-72300271 .

stefanalund commented 9 years ago

Fantastic!

ijsf commented 9 years ago

I'm available for any help on further optimization for iOS audio. Done a iOS 6.x and 7.x port of WebRTC in the past and have a working setup for an earlier version of WebRTC, so I might be able to provide some pointers.

ikonst commented 9 years ago

This fix should eliminate the other case of resampling (before osxaudiosink): https://bugzilla.gnome.org/show_bug.cgi?id=743925

(This, second, case of resampling only happens sometimes, dependent on a race condition.)

ford-prefect commented 9 years ago

Just as a quick status update, with current gst-plugins-base/good master, we no longer do any resampling on a Bowser <-> FF call. Next step is to look at the rest of @ikonst's caps probing work for the next set of improvements.

superdump commented 9 years ago

Excellent!

superdump commented 9 years ago

@ikonst any news on https://bugzilla.gnome.org/show_bug.cgi?id=743758 ?

ikonst commented 9 years ago

We had the Purim holiday this weekend which really took all my attention. :) I hope to work on it today. On Mar 8, 2015 1:58 PM, "Robert Swain" notifications@github.com wrote:

@ikonst https://github.com/ikonst any news on https://bugzilla.gnome.org/show_bug.cgi?id=743758 ?

— Reply to this email directly or view it on GitHub https://github.com/EricssonResearch/openwebrtc/issues/144#issuecomment-77745115 .

superdump commented 9 years ago

Just to note that these has been progress in https://bugzilla.gnome.org/show_bug.cgi?id=743758 - great work guys! :smile: