search

igorski / MWEngine

Audio engine and DSP library for Android, written in C++ providing low latency performance within a musical context, while providing a Java/Kotlin API. Supports both OpenSL and AAudio.
MIT License
259 stars 45 forks source link

Sound stutters when app is playing in background #119

Closed harthorst closed 3 years ago

harthorst commented 4 years ago

When the example app goes into background while playing (e.g. by removing onWindowFocusChanged method) the sound begins to stutter. This sounds very similar to issue 114 btw. After returning into foreground mode the sound does not recover. Even starting and stopping the sequencer does not help. I know, I'm a pain in the ass :)

igorski commented 4 years ago

Well it's logical as it attaches to the thread of the foreground application, but it gets less priority when the application is delegated to the background. It is curious though that this isn't raised upon return.

I could solve it by separating the thread from the MWEngine class so it can construct a new one but is it really a valid use case to suspend an application that has urgent audio priority for live synthesis ?

harthorst commented 4 years ago

It is useful if you want to open another app for recording. In my case this is a use case requested by an user. But I'm sure this feature is rarely used so it is not very important.

BTW: Going into background did not have any affect to sound stability in older versions of MWEngine.

igorski commented 4 years ago

Makes sense, just a question, have you tried this without using MWEngine.optimizePerformance() ? (as that ties the engine to the foreground thread)

harthorst commented 4 years ago

Uncommenting MWEngine.optimizePerformance() makes the example app stuttering even in foreground mode.

igorski commented 4 years ago

Hi, could you check out the native thread branch and see what results are ?

I have changed the render thread handling so that it is not maintained by Java and shared by the native layer, but the other way around.

Could you also check for different driver types ?

harthorst commented 4 years ago

Hi,

i get a lot of cpp compile errors.

`FAILED: /home/tf/bin/Android/Sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++ --target=armv7-none-linux-androideabi16 --gcc-toolchain=/home/tf/bin/Android/Sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64 --sysroot=/home/tf/bin/Android/Sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/sysroot -Dmwengine_wrapped_EXPORTS -I../../../../src/main/cpp -g -DANDROID -fdata-sections -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -D_FORTIFY_SOURCE=2 -march=armv7-a -mthumb -Wformat -Werror=format-security -std=c++14 -Werror -Wno-deprecated-declarations -fexceptions -frtti -Oz -DNDEBUG -fPIC -MD -MT CMakeFiles/mwengine_wrapped.dir/src/main/java/nl/igorski/mwengine/core/mwengineJAVA_wrap.cxx.o -MF CMakeFiles/mwengine_wrapped.dir/src/main/java/nl/igorski/mwengine/core/mwengineJAVA_wrap.cxx.o.d -o CMakeFiles/mwengine_wrapped.dir/src/main/java/nl/igorski/mwengine/core/mwengineJAVA_wrap.cxx.o -c /home/tf/tmp/MWEngine/src/main/java/nl/igorski/mwengine/core/mwengineJAVA_wrap.cxx /home/tf/tmp/MWEngine/src/main/java/nl/igorski/mwengine/core/mwengineJAVA_wrap.cxx:9391:8: error: object of type 'std::ndk1::atomic' cannot be assigned because its copy assignment operator is implicitly deleted arg1 = *argp1; ^ /home/tf/bin/Android/Sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include/c++/v1/atomic:1643:7: note: copy assignment operator of 'atomic' is implicitly deleted because base class '__atomic_base' has a deleted copy assignment operator : public atomic_base<_Tp>

...`

igorski commented 4 years ago

Thomas wtf ? :)

Could you tell me what is reported when you run clang -v on the command line ?

Also, you mentioned you update the minSdkVersion to 23 because otherwise compilation won't work which I find kind of curious. Could you also list the Android SDK and NDK versions you are using and what OS ?

harthorst commented 4 years ago

Also, you mentioned you update the minSdkVersion to 23 You are right - I did'nt change that this time. MWEngine is a fresh clone and checkout to branch 'native-thread'.

clang gives me:

Android (5900059 based on r365631c) clang version 9.0.8 (https://android.googlesource.com/toolchain/llvm-project 207d7abc1a2abf3ef8d4301736d6a7ebc224a290) (based on LLVM 9.0.8svn) Target: x86_64-unknown-linux-gnu Thread model: posix InstalledDir: /home/tf/bin/Android/Sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/bin Found candidate GCC installation: /home/tf/bin/Android/Sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/bin/../lib/gcc/i686-linux-android/4.9.x Found candidate GCC installation: /home/tf/bin/Android/Sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/bin/../lib/gcc/x86_64-linux-android/4.9.x Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/7 Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/7.5.0 Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/8 Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/7.5.0 Candidate multilib: .;@m64 Selected multilib: .;@m64

BTW: Building the master branch works.

igorski commented 4 years ago

Well the reason I asked about updating minSdkVersion to 23 is because there is no reason compilation shouldn't work at the level defined in the gradle file... so that's why I was curious about your NDK and SDK version as there is some mismatch between our build environments and I'd like to understand why you are forced to use a newer version. What is the message you are getting when attempting at a lower version?

I have updated the branch to circumvent the issue you experienced in your previous trace.

harthorst commented 4 years ago

Hi Igor,

lint is complaining:

/home/tf/tmp/MWEngine/src/main/java/nl/igorski/example/MWEngineActivity.java:102: Error: Call requires API level 23 (current min is 16): android.app.Activity#requestPermissions [NewApi] requestPermissions( new String[]{ Manifest.permission.RECORD_AUDIO }, PERMISSIONS_CODE );

About the branch: The example app works much better now in background with both drivers. AAudio still shows glitches equally in foreground and background mode. OpenSL sounds clean at the first glance.

igorski commented 4 years ago

Hi Igor,

lint is complaining:

/home/tf/tmp/MWEngine/src/main/java/nl/igorski/example/MWEngineActivity.java:102: Error: Call requires API level 23 (current min is 16): android.app.Activity#requestPermissions [NewApi] requestPermissions( new String[]{ Manifest.permission.RECORD_AUDIO }, PERMISSIONS_CODE );

Ah I should've known, needed to add a missing TargetApi annotation 🤦‍♂️ thanks for pointing that out as my linter settings were more forgiving.

About the branch: The example app works much better now in background with both drivers. AAudio still shows glitches equally in foreground and background mode. OpenSL sounds clean at the first glance.

Yes, I had noticed degraded performance for AAudio, I made some nice discoveries with regards to different thread usage between OpenSL and AAudio, still have to investigate further though it is nice to uncover such long hidden issues :)

By the way, are you still having the issue where the recorded file is 0 bytes in size?

harthorst commented 4 years ago

Hi Igor. Yes, recording works in the native-thread branch and glitches are audible in the wav file.

igorski commented 4 years ago

Well that is one step in the right direction and one backwards :grimacing:

If the glitches are audible in the recording that would imply that something is going wrong with threaded callback - which is actually a good thing in the sense that it makes tracking this down easier -.

Are the glitches audible in both AAudio and OpenSL ? (for me the OpenSL version is clean)

harthorst commented 4 years ago

On my phone it's the same. OpenSL sounds good, AAudio has glitches. Seems that CPU consumption is very different in AAudio (~15%) and OpenSL (~5%).

igorski commented 3 years ago

It's been a while, but looking into these issues again. If you were to comment _PREVENT_CPU_FREQUENCYSCALING in global.h before building the library, would that restore the stability for you?

Just to sum up if I'm correct:

harthorst commented 3 years ago

As far as I remember I was not able to find a satisfying configuration. And according to my latest commend I think your assumptions are correct.

I'm not using MWEngine anymore so I can't tell you any news here but I can support you and verify things on my test devices.

igorski commented 3 years ago

Yeah I can imagine after practically a year passing cough 😊

In the process of doing some cleanups and making the library more easily shareable and was looking at the open issues. I have a wider range of test devices now so it's hopefully easier to verify. Were you testing this on a Xiaomi if I remember correctly ?

harthorst commented 3 years ago

Yes, Xiaomi Pad 4 and Redmi Note 7. I think, the Redmi had more Problems.

igorski commented 3 years ago

Interesting, both use the Snapdragon 660 processor. Could be an arm64-v8a related issue where I think the thread frequency tuning is geared towards Intel specific instructions.

Anyways I thank you for your time and input, I'll create a new issue from your findings and the related one on the app stuttering in the background.