Audio processing and silence skips

[jamescridland]

Is your feature request related to a problem? Please describe.

• I'm always frustrated when I can't hear some of the words because I'm in a noisy space.

• Sometimes the podcasts I listen to have lots of spaces in them and I'd like to not waste that time please

Describe the solution you'd like

• A 'skip silences' feature which would examine the audio that is about to play. If it is longer than (xx) miliseconds and is below (yy) audio level, then skip the silent bit.

  • Double points if you always skip any silence at the beginning or end of an audio file. The silence at the top of an audio file is normally only there for technical reasons; and skipping that would make the UI appear snappier.

• A 'voice boost' feature which adds some form of dynamic processing to the audio, artificially compressing the dynamic range to make it easier to hear in loud environments, or to compensate for crappy audio production. Should compress audio's peaks over a certain audio level, then boost it back up again to hit a standard level. Ideally that standard level should be the same level as other notifications on the phone.

  • Double points if this is configurable by someone using this SDK, to allow differences for, say, speech and music.

Describe alternatives you've considered

(n/a)

Additional context

A relatively poor example of 'skip silences' is implemented in Google Podcasts. A much better example (and I don't know what they do differently) is implemented in Pocket Casts, and they have a setting to control the fierceness of it. A cleverer version is in Overcast, which I believe also uses speed control.

Pocket Casts has a nice example of audio compression / "voice boost".

These features are particularly useful for podcast playback; though you could see some benefit for dynamic compression within music playback too. An enterprising app could use the microphone input to understand the noisiness of the environment and automatically add processing to make it easier to hear.

[ryanheise]

This has been on my radar, and there is a relatively simple way of implementing this on Android. What has delayed things is:

1. It is substantially more difficult to implement this on iOS/macOS, and feature parity is important in Flutter plugins.
2. On the API design front, further consideration is needed on whether these audio effects can be generalised (similarly to how they are in the native Android APIs). Backwards compatibility is also an important consideration for Flutter plugins, so I'd like to get the API right.
3. It does take a lot of time to implement everything. Some of the more difficult features were a solid month's work each (or more) to get right, and so the time it takes me to work through the backlog could result in a wait of a year or more without community contributions.

But, if there is enough demand for this feature, I could start thinking about point 2, then point 2, then later point 3.

If anyone stumbles upon this feature request and would like to help, it would certainly be helpful to post links below to sample code and tutorials on the web that implement this on iOS/macOS, preferably AVPlayer/AVQueuePlayer-based implementations. To start things off, here is an article on skipping silence using AVAudioPlayer:

https://blog.breaker.audio/how-we-skip-silences-in-podcasts-with-avaudioplayer-69232b57850a

[ryanheise]

On the iOS side, some information on volume boost: https://stackoverflow.com/questions/30238287/core-audio-audio-unit-to-boost-signal-level

This would depend on #334 .

[ryanheise]

The Android side of "skip silence" is now implemented in dev via PR #234 .

The volume boost feature needs some further thought on the API design front. iOS can represent all sorts of audio effects via AVAudioUnit, while Android can represent these via AudioEffect. I would like to create some similar type of generalisation in the just_audio API after analysing the commonalities between these two underlying frameworks.

[ryanheise]

The plan is now to implement volume boost as part of #398 .

[ryanheise]

398 is now complete, with two initial AudioEffect implementations on the Android side: LoudnessEnhancer (can be used for volume boosting) and Equalizer. There are many other Android audio effects that could be added in the future, in particular DynamicsProcessing but #398 at least sets up the foundations for adding further effects in the future. The iOS work is now dependent on #334 .

[ryanheise]

The Android implementation is now published in release 0.8.0.

[hioshih]

How can we apply LoudnessEnhancer to an AudioPlayer?

[hioshih]

oh nevermind, I think I figured it out:

var _enhancer = AndroidLoudnessEnhancer();
    _enhancer.setTargetGain(10.0);
    _enhancer.setEnabled(true);
    AudioPipeline pipeline = AudioPipeline(androidAudioEffects: [_enhancer]);
    _player = AudioPlayer(audioPipeline: pipeline);

[mhassan772]

Will the iOS be supported soon too?

[ryanheise]

Hi @mhassan772

Audio processing on the iOS side is more complex, but should be made easier on the AVAudioEngine-based implementation (see #784 ). Still, this project is only possible through collaborative effort, and the iOS side is waiting for a contributor.

As such, you may like to consider becoming a contributor yourself, to help implement the features you need and contribute them back to the project.

[grafst]

and feature parity is important in Flutter plugins I just wanted to say that I really like that sentiment 😊