Closed jamescookmd closed 4 years ago
jamescookmd
commented
6 years ago BTW, I'm not wedded to this mic hardware, so if there's some other way to do a remote wireless microphone that works with Android low-latency audio I'd be happy to switch. I just need to be able to support output to headphones, so I'm hoping not to use the headphone jack for input.
dturner
commented
6 years ago Thanks for raising this issue. The logs actually seem fine. The error you see is an attempt to open an AAudio MMAP (memory mapped) stream which is only supported on specific hardware, and definitely won't be supported on a bluetooth headset (for now at least) so Oboe falls back to opening a "legacy" stream which then succeeds.
Are there any errors shown further down the logs when the output stream is opened?
dturner
commented
6 years ago Ignore my comment about the output stream, I can see this is opened successfully with device id 218 before the input stream.
Would you mind detailing the reproduction steps exactly. For example:
It just helps us ensure we're doing the exact same thing when we test.
jamescookmd
commented
6 years ago I'm a Googler in MTV, I can bring the hardware to someone in MTV if that's helpful.
Repro:
Results in no audio in headphones.
I get the same result with a Plantronics M165 Bluetooth headset as the input source. The headset works properly for phone calls.
I added some logging to LiveEffectEngine::onAudioReady(). I see the initial 1/2 second audio drain, the n a bunch of calls mRecordingStream->read() that return framesRead = 96. However, the audioData is always full of 0 after each read().
Logcat shows:
07-10 19:18:42.536 8995-8995/? D/OboeAudio: AAudioLoader(): dlopen(libaaudio.so) returned 0xc86d87f3332faf3b
07-10 19:18:42.536 8995-8995/? V/OboeAudio: AAudioLoader(): dlsym(AAudioStreamBuilder_setChannelCount) succeeded.
AAudioLoader(): dlsym(AAudioStreamBuilder_setBufferCapacityInFrames) succeeded.
AAudioLoader(): dlsym(AAudioStreamBuilder_setDeviceId) succeeded.
AAudioLoader(): dlsym(AAudioStreamBuilder_setDirection) succeeded.
AAudioLoader(): dlsym(AAudioStreamBuilder_setFormat) succeeded.
AAudioLoader(): dlsym(AAudioStreamBuilder_setFramesPerDataCallback) succeeded.
AAudioLoader(): dlsym(AAudioStreamBuilder_setSharingMode) succeeded.
AAudioLoader(): dlsym(AAudioStreamBuilder_setPerformanceMode) succeeded.
AAudioLoader(): dlsym(AAudioStreamBuilder_setSampleRate) succeeded.
07-10 19:18:42.536 8995-8995/? E/OboeAudio: AAudioLoader could not find AAudioStreamBuilder_setUsage
AAudioLoader could not find AAudioStreamBuilder_setContentType
AAudioLoader could not find AAudioStreamBuilder_setInputPreset
AAudioLoader could not find AAudioStreamBuilder_setSessionId
07-10 19:18:42.536 8995-8995/? V/OboeAudio: AAudioLoader(): dlsym(AAudioStreamBuilder_delete) succeeded.
AAudioLoader(): dlsym(AAudioStream_getChannelCount) succeeded.
AAudioLoader(): dlsym(AAudioStream_close) succeeded.
AAudioLoader(): dlsym(AAudioStream_getBufferSizeInFrames) succeeded.
AAudioLoader(): dlsym(AAudioStream_getDeviceId) succeeded.
AAudioLoader(): dlsym(AAudioStream_getDirection) succeeded.
AAudioLoader(): dlsym(AAudioStream_getBufferCapacityInFrames) succeeded.
AAudioLoader(): dlsym(AAudioStream_getFramesPerBurst) succeeded.
AAudioLoader(): dlsym(AAudioStream_getFramesRead) succeeded.
AAudioLoader(): dlsym(AAudioStream_getFramesWritten) succeeded.
AAudioLoader(): dlsym(AAudioStream_getPerformanceMode) succeeded.
AAudioLoader(): dlsym(AAudioStream_getSampleRate) succeeded.
AAudioLoader(): dlsym(AAudioStream_getSharingMode) succeeded.
AAudioLoader(): dlsym(AAudioStream_getState) succeeded.
AAudioLoader(): dlsym(AAudioStream_getXRunCount) succeeded.
AAudioLoader(): dlsym(AAudioStream_requestStart) succeeded.
AAudioLoader(): dlsym(AAudioStream_requestPause) succeeded.
AAudioLoader(): dlsym(AAudioStream_requestFlush) succeeded.
AAudioLoader(): dlsym(AAudioStream_requestStop) succeeded.
AAudioLoader(): dlsym(AAudioStream_setBufferSizeInFrames) succeeded.
AAudioLoader(): dlsym(AAudio_convertResultToText) succeeded.
AAudioLoader(): dlsym(AAudio_convertStreamStateToText) succeeded.
07-10 19:18:42.537 8995-8995/? E/OboeAudio: AAudioLoader could not find AAudioStream_getUsage
AAudioLoader could not find AAudioStream_getContentType
AAudioLoader could not find AAudioStream_getInputPreset
AAudioLoader could not find AAudioStream_getSessionId
07-10 19:18:42.555 8995-9002/? I/zygote64: Do partial code cache collection, code=29KB, data=22KB
07-10 19:18:42.556 8995-9002/? I/zygote64: After code cache collection, code=29KB, data=22KB
Increasing code cache capacity to 128KB
07-10 19:18:42.557 8995-9067/? D/OpenGLRenderer: HWUI GL Pipeline
07-10 19:18:42.641 8995-9067/? I/Adreno: QUALCOMM build : 2941438, I916dfac403
Build Date : 10/03/17
OpenGL ES Shader Compiler Version: EV031.21.02.00
Local Branch : O18A
Remote Branch :
Remote Branch :
Reconstruct Branch :
07-10 19:18:42.646 8995-9067/? I/Adreno: PFP: 0x005ff087, ME: 0x005ff063
07-10 19:18:42.650 8995-9067/? I/zygote64: android::hardware::configstore::V1_0::ISurfaceFlingerConfigs::hasWideColorDisplay retrieved: 0
07-10 19:18:42.650 8995-9067/? I/OpenGLRenderer: Initialized EGL, version 1.4
07-10 19:18:42.650 8995-9067/? D/OpenGLRenderer: Swap behavior 2
07-10 19:18:42.688 8995-9002/? I/zygote64: Do partial code cache collection, code=33KB, data=49KB
After code cache collection, code=33KB, data=49KB
Increasing code cache capacity to 256KB
Compiler allocated 7MB to compile void android.widget.TextView.
Hope that helps!
dturner
commented
6 years ago Thanks James. philburk@ is taking a look at this well who is also in MTV.
I have tried 2 sets of different bluetooth headsets (a Google branded noise-cancelling set and a TaoTronics TT-BH03). I can reproduce the issue on both.
I also tried a recording app. This was unable to record anything using the headset's built-in microphone.
One possible cause is that this is a permissions issue where an app is not allowed to record data from an audio device which is used for telephony. Could be wrong though. Phil will have more info.
jamescookmd
commented
6 years ago I'll bring the HeyMic to work today. Ping me at jamescook@ if it would be useful to have for testing.
stereomatch
commented
6 years ago Bluetooth microphones will have higher latency compared to a wired headphone (to the point of being unusable for full-duplex use - not because the audio will visibly mismatch the lip movement (not significant problem), but if the user has a good ear and are thus hearing both the real-time audio over the air, as well as through the earphones - a latency of 100ms or more will make it unusable (confusing to the user). If the user can only hear perceptibly through the headphones, latency will be less of an issue (for example the latency on Bluetooth headsets is not a significant issue for phone calls).
If the user want to have ability to hear the android device's own output, you can use an equalizer for system sound like Viper4Android (needs root) - and there may be some other apps for that as well. This provides a 10-band equalizer (I think) - so will help in boosting audio for movies, and android audio.
From an android programming perspective, if you want to use Bluetooth, that has to be enabled explicitly in your code - you have to add extra permissions for BLUETOOTH in AndroidManifest.xml, plus in the java code you have to enable Bluetooth and also register some receivers to monitor the availability of Bluetooth device etc. This is covered in many stackoverflow threads - try to find one which is recent.
Bluetooth in general on android is problematic (on some devices it will give issues - on others you won't get higher than 8000 Hz audio quality). This is why Bluetooth support in audio app is so uneven on Android. iOS have forked their own Bluetooth and adapted it to their needs (which has positives and negatives).
If you want to see how the lowest latency audio sounds like on a hearing aid app on Android Oreo 8.1, try the Dectone app on Google Play (not related to us), or search for Hearing Aid for Oreo 8.1 (which is our app - download the free version - let me know and I can send a Promo Code) - it has few downloads, but we already have gotten good feedback from an audiologist user, and some hearing impaired users via e-mail. These will work best with wired headphones or headset.
Also you need to have Oreo 8.1 to benefit from new low latency AAudio (used only by Oboe for Oreo 8.1+). However, if you have an Oreo 8.0 device (and if you are lucky and have the right android device), you may still be able to benefit from lowest latency using our Hearing Aid for Oreo 8.0 app - the reason I say lucky is because half the android Oreo 8.0 devices out there will crash with AAudio (because of a bugfix in Oreo 8.0 which was not picked by most major manufacters like Samsung, HTC, Xperia).
So if you have an Oreo 8.0 device, the only lowest latency app is our Hearing Aid for Oreo 8.0 app - and that too is not guaranteed to work on all Oreo 8.0 devices (there is a reason we created a separate app for Oreo 8.0).
Oreo 8.1 is the first android version where all android devices are guaranteed to work with AAudio - which is why Oboe later had to switch to using AAudio only in Oreo 8.1+ (skipping Oreo 8.0).
Let me know if you have issues - click Help - Contact from with the Hearing Aid for Oreo 8.1 app for help via e-mail.
As feedback, if you see an increase in latency over time - i.e. starting off with very low latency, but a few minutes later the latency is noticeably higher, let me know. This is a known issue covered here:
https://github.com/google/oboe/issues/165 AAudio full-duplex latency increasing over time reason identified and needs fix to dataCallback() API in AAudio #165
antonkovalyov351
commented
5 years ago @philburk @dturner Any updates on this issue? Will Oboe library support BT headset as input? Will it work for both AAudio and OpenSL?
jamescookmd
commented
5 years ago Just FYI, I was able to use a Bluetooth headset for input by doing this:
remoteButton = findViewById(R.id.remoteButton);
remoteButton.setOnClickListener(new View.OnClickListener() {
@Override
public void onClick(View view) {
Log.d(TAG, "JAMES Starting bluetooth SCO");
AudioManager am = (AudioManager) getSystemService(Context.AUDIO_SERVICE);
am.startBluetoothSco();
}
});
registerReceiver(new BroadcastReceiver() {
@Override
public void onReceive(Context context, Intent intent) {
int state = intent.getIntExtra(AudioManager.EXTRA_SCO_AUDIO_STATE, -1);
if (state == AudioManager.SCO_AUDIO_STATE_CONNECTING) {
Log.d(TAG, "JAMES Audio SCO connecting " + state);
} else if (state == AudioManager.SCO_AUDIO_STATE_CONNECTED) {
Log.d(TAG, "JAMES Connected!");
/*
* Now the connection has been established to the bluetooth device.
* Record audio or whatever (on another thread).With AudioRecord you can record with an object created like this:
* new AudioRecord(MediaRecorder.AudioSource.MIC, 8000, AudioFormat.CHANNEL_CONFIGURATION_MONO,
* AudioFormat.ENCODING_PCM_16BIT, audioBufferSize);
*
* After finishing, don't forget to unregister this receiver and
* to stop the bluetooth connection with am.stopBluetoothSco();
*/
unregisterReceiver(this);
} else if (state == AudioManager.SCO_AUDIO_STATE_DISCONNECTED) {
Log.d(TAG, "JAMES Audio SCO disconnected " + state);
}
}
}, new IntentFilter(AudioManager.ACTION_SCO_AUDIO_STATE_UPDATED));However, latency is quite high, just due to the Bluetooth connection. It's not really suitable for live effects.
antonkovalyov351
commented
5 years ago @jamescookmd Thanks for your answer. I'm curious if it possible to use Bluetooth headset as input using Oboe library.
philburk
commented
5 years ago am.startBluetoothSco();
That is the key step.
Do we want to allow the user to control this from a Button? Or should we start/stop it automatically based on the BroadcastReceiver?
This is related to https://github.com/google/oboe/issues/471
philburk
commented
4 years ago I implemented Bluetooth SCO in this PR https://github.com/google/oboe/pull/896
We should add startBluetoothSco() to the samples and docs. See #909 and 910.
I'm new to audio, please excuse me if I'm doing something obviously wrong. I'm trying to write an app to help my son with a hearing condition, so I'm trying to get input from a remote Bluetooth microphone, process it, and output to headphones.
I'm using oobe eb236cc466b73344a9f2e1d15c2dc7751a0e8406 (tip-of-master) building the samples Android Studio 3.1.3. I haven't modified the code.
I'm running on a Pixel 1, OMR1 build OPM4.171019.021.D1 (latest stable build with June 5th security updates).
LiveEffect works fine with input Pixel mic and output Pixel speaker (with annoying feedback). Likewise it works fine with output to wired headphones.
I have this Bluetooth "microphone": https://www.amazon.com/Hey-Mic-Bluetooth-Microphone-Recording/dp/B075K3K8M3/ I think it's actually a headset under the hood (it has an audio output jack!).
After pairing it shows up as an input in LiveEffect, as "HeyMic Bluetooth device typically used for telephony". However, when I tap "Start" and talk into the mic I don't get audio out the headphones.
The mic works with the company's iPhone app, so I don't think the hardware is bad.
Logcat shows AAudio receiving an OPEN_STREAM error below, but then does fallback and appears to succeed.
Should a Bluetooth headset work out-of-the-box as an input? Am I doing something wrong?
07-09 20:08:24.167 7505-7510/com.google.sample.oboe.liveeffect I/zygote64: Compiler allocated 6MB to compile void android.view.ViewRootImpl.performTraversals() 07-09 20:08:26.622 7505-7505/com.google.sample.oboe.liveeffect D/com.google.sample.oboe.liveEffect.MainActivity: Attempting to start 07-09 20:08:26.624 7505-7505/com.google.sample.oboe.liveeffect D/OboeAudio: AudioStreamAAudio() call isSupported() AudioStreamAAudio(): AAudio_createStreamBuilder() AudioStreamAAudio.open() try with deviceId = 218 07-09 20:08:26.624 7505-7505/com.google.sample.oboe.liveeffect D/AAudio: AAudioStreamBuilder_openStream() called ---------------------------------------- AudioStreamBuilder(): mmapPolicy = 2, mapExclusivePolicy = 2 07-09 20:08:26.624 7505-7505/com.google.sample.oboe.liveeffect D/AudioStreamInternal_Client: AudioStreamInternal(): mWakeupDelayNanos = 200000, mMinimumSleepNanos = 200000 07-09 20:08:26.624 7505-7505/com.google.sample.oboe.liveeffect I/AAudioStream: AudioStream::open() rate = 0, channels = 2, format = 1, sharing = EX, dir = OUTPUT AudioStream::open() device = 218, perfMode = 12, callback: ON with frames = 0 07-09 20:08:26.633 7505-7505/com.google.sample.oboe.liveeffect D/AAudio: AAudioStreamBuilder_openStream() returns 0 = AAUDIO_OK for (0x784724e500) ---------------- 07-09 20:08:26.634 7505-7505/com.google.sample.oboe.liveeffect D/OboeAudio: AudioStreamAAudio.open() app format = 1 AudioStreamAAudio.open() native format = 1 AudioStreamAAudio.open() sample rate = 48000 AudioStreamAAudio.open() capacity = 3072 AudioStreamAAudio.open: AAudioStream_Open() returned AAUDIO_OK, mAAudioStream = 0x784724e500 AudioStreamAAudio() call isSupported() AudioStreamAAudio(): AAudio_createStreamBuilder() AudioStreamAAudio.open() try with deviceId = 204 07-09 20:08:26.634 7505-7505/com.google.sample.oboe.liveeffect D/AAudio: AAudioStreamBuilder_openStream() called ---------------------------------------- AudioStreamBuilder(): mmapPolicy = 2, mapExclusivePolicy = 2 07-09 20:08:26.634 7505-7505/com.google.sample.oboe.liveeffect D/AudioStreamInternal_Client: AudioStreamInternal(): mWakeupDelayNanos = 200000, mMinimumSleepNanos = 200000 07-09 20:08:26.634 7505-7505/com.google.sample.oboe.liveeffect I/AAudioStream: AudioStream::open() rate = 48000, channels = 2, format = 1, sharing = EX, dir = INPUT AudioStream::open() device = 204, perfMode = 12, callback: OFF with frames = 0 07-09 20:08:26.646 7505-7505/com.google.sample.oboe.liveeffect E/AAudio: BpAAudioService::client OPEN_STREAM passed stream -889 07-09 20:08:26.646 7505-7505/com.google.sample.oboe.liveeffect E/AudioStreamInternal_Client: AudioStreamInternal::open(): openStream() returned -889 07-09 20:08:26.646 7505-7505/com.google.sample.oboe.liveeffect D/AAudioStream: destroying 0x7847250100, state = AAUDIO_STREAM_STATE_UNINITIALIZED 07-09 20:08:26.646 7505-7505/com.google.sample.oboe.liveeffect D/AAudio: AudioStreamBuilder.build() MMAP stream did not open so try Legacy path 07-09 20:08:26.646 7505-7505/com.google.sample.oboe.liveeffect I/AAudioStream: AudioStream::open() rate = 48000, channels = 2, format = 1, sharing = EX, dir = INPUT AudioStream::open() device = 204, perfMode = 12, callback: OFF with frames = 0 07-09 20:08:26.646 7505-7505/com.google.sample.oboe.liveeffect D/AudioStreamRecord: AudioStreamRecord::open(), request notificationFrames = 0, frameCount = 0 07-09 20:08:26.657 7505-7505/com.google.sample.oboe.liveeffect I/AudioRecord: AUDIO_INPUT_FLAG_FAST successful; frameCount 0 -> 4096 07-09 20:08:26.657 7505-7505/com.google.sample.oboe.liveeffect W/AudioStreamRecord: AudioStreamRecord::open() flags changed from 0x00000005 to 0x00000001 07-09 20:08:26.659 7505-7505/com.google.sample.oboe.liveeffect D/AAudio: AAudioStreamBuilder_openStream() returns 0 = AAUDIO_OK for (0x782fb69980) ---------------- 07-09 20:08:26.659 7505-7505/com.google.sample.oboe.liveeffect D/OboeAudio: AudioStreamAAudio.open() app format = 1 AudioStreamAAudio.open() native format = 1 AudioStreamAAudio.open() sample rate = 48000 AudioStreamAAudio.open() capacity = 4096 AudioStreamAAudio.open: AAudioStream_Open() returned AAUDIO_OK, mAAudioStream = 0x782fb69980 07-09 20:08:26.659 7505-7505/com.google.sample.oboe.liveeffect D/AAudio: AAudioStream_requestStart(0x782fb69980) called -------------- 07-09 20:08:26.663 7505-7505/com.google.sample.oboe.liveeffect D/AAudio: AAudioStream_requestStart(0x782fb69980) returned 0 --------- AAudioStream_requestStart(0x784724e500) called -------------- 07-09 20:08:26.663 7505-7518/com.google.sample.oboe.liveeffect D/AudioStreamLegacy: onAudioDeviceUpdate() deviceId 204 07-09 20:08:26.779 7505-7505/com.google.sample.oboe.liveeffect D/AAudio: AAudioStream_requestStart(0x784724e500) returned 0 --------- 07-09 20:08:26.783 7505-8707/com.google.sample.oboe.liveeffect D/AudioStreamInternal_Client: AudioStreamInternal::onEventFromServer() AAUDIO_SERVICE_EVENT_VOLUME 1.000000 AudioStreamInternal::onEventFromServer() AAUDIO_SERVICE_EVENT_VOLUME 0.023646 AudioStreamInternal::onEventFromServer() got AAUDIO_SERVICE_EVENT_STARTED 07-09 20:08:26.783 7505-8707/com.google.sample.oboe.liveeffect D/AAudio: advanceClientToMatchServerPosition() readN = 2592, writeN = 0, offset = -2592