search

pichillilorenzo / flutter_inappwebview

A Flutter plugin that allows you to add an inline webview, to use a headless webview, and to open an in-app browser window.
https://inappwebview.dev
Apache License 2.0
3.31k stars 1.64k forks source link

How to avoid microphone state change on IOS (stops recording) while media playback is ongoing from the Flutter side #2375

Closed keyur2maru closed 1 month ago

keyur2maru commented 1 month ago

Is there an existing issue for this?

Current Behavior

I am using Headless InAppWebView to run a VAD model that uses browser API to record. Everything works fine until I hit the play button, which starts playing the identified speech segments from the Flutter/Dart layer. From here onwards, the onMicrophoneCaptureStateChanged is triggered, and no new audio is processed until I restart the Headless InAppWebView.

For playing audio on the flutter side, I am using audioplayers package.

I also tried using audio_session to allow AVAudioSessionCategory.playAndRecord but that did not help.

This issue does not appear on Flutter Web. It is observed on IOS only (I haven't tried on Android).

Expected Behavior

It should continue recording even while the media playback is on from the Flutter side.

Steps with code example to reproduce

Steps with code example to reproduce ```dart // vad_handler.dart import 'package:vad/src/vad_handler_base.dart'; import 'vad_handler_web.dart' if (dart.library.io) 'vad_handler_non_web.dart' as implementation; class VadHandler { static VadHandlerBase create() { return implementation.createVadHandler(); } } // vad_handler_base.dart import 'dart:async'; abstract class VadHandlerBase { Stream> get onSpeechEnd; Stream get onSpeechStart; Stream get onVADMisfire; Stream get onError; void startListening({ double positiveSpeechThreshold = 0.5, double negativeSpeechThreshold = 0.35, int preSpeechPadFrames = 1, int redemptionFrames = 8, int frameSamples = 1536, int minSpeechFrames = 3, bool submitUserSpeechOnPause = false }); void stopListening(); void dispose(); } // vad_handler_non_web.dart // vad_handler_non_web.dart import 'dart:async'; import 'dart:convert'; import 'package:flutter/foundation.dart'; import 'package:flutter_inappwebview/flutter_inappwebview.dart'; import 'vad_handler_base.dart'; class VadHandlerNonWeb implements VadHandlerBase { HeadlessInAppWebView? _headlessWebView; final _onSpeechEndController = StreamController>.broadcast(); final _onSpeechStartController = StreamController.broadcast(); final _onVADMisfireController = StreamController.broadcast(); final _onErrorController = StreamController.broadcast(); bool _isInitialized = false; static const String _vadHtml = ''' '''; @override Stream> get onSpeechEnd => _onSpeechEndController.stream; @override Stream get onSpeechStart => _onSpeechStartController.stream; @override Stream get onVADMisfire => _onVADMisfireController.stream; @override Stream get onError => _onErrorController.stream; VadHandlerNonWeb() { _initialize(); } Future _initialize() async { if (_isInitialized) return; final completer = Completer(); _headlessWebView = HeadlessInAppWebView( initialData: InAppWebViewInitialData( data: _vadHtml, mimeType: 'text/html', encoding: 'utf-8', baseUrl: WebUri('https://cdn.jsdelivr.net'), ), onPermissionRequest: (controller, request) async { return PermissionResponse(resources: request.resources, action: PermissionResponseAction.GRANT); }, initialSettings: InAppWebViewSettings( mediaPlaybackRequiresUserGesture: false, javaScriptEnabled: true, isInspectable: kDebugMode, allowsInlineMediaPlayback: true, ), onWebViewCreated: (controller) { controller.addJavaScriptHandler( handlerName: 'onVadInitialized', callback: (args) { _isInitialized = true; completer.complete(); }, ); controller.addJavaScriptHandler( handlerName: 'handleEvent', callback: (args) { if (args.length >= 2) { final eventType = args[0] as String; final payload = args[1] as String; _handleEvent(eventType, payload); } }, ); controller.addJavaScriptHandler( handlerName: 'logMessage', callback: (args) { if (args.isNotEmpty) { debugPrint('VAD Log: ${args.first}'); } }, ); }, onLoadError: (controller, url, code, message) { debugPrint('VAD Load Error: $message'); _onErrorController.add('Failed to load VAD: $message'); }, onConsoleMessage: (controller, consoleMessage) { debugPrint('VAD Console: ${consoleMessage.message}'); }, ); await _headlessWebView?.run(); try { await completer.future.timeout( const Duration(seconds: 10), onTimeout: () { throw TimeoutException('VAD initialization timed out'); }, ); debugPrint('VAD initialized successfully'); } catch (e) { _onErrorController.add(e.toString()); debugPrint('VAD initialization failed: $e'); } } void _handleEvent(String eventType, String payload) { try { Map eventData = payload.isNotEmpty ? json.decode(payload) : {}; switch (eventType) { case 'onError': if (eventData.containsKey('error')) { _onErrorController.add(eventData['error'].toString()); } else { _onErrorController.add(payload); } break; case 'onSpeechEnd': if (eventData.containsKey('audioData')) { final List audioData = (eventData['audioData'] as List) .map((e) => (e as num).toDouble()) .toList(); _onSpeechEndController.add(audioData); } else { debugPrint('Invalid VAD Data received: $eventData'); } break; case 'onSpeechStart': _onSpeechStartController.add(null); break; case 'onVADMisfire': _onVADMisfireController.add(null); break; default: debugPrint("Unknown event: $eventType"); } } catch (e, st) { debugPrint('Error handling event: $e'); debugPrint('Stack Trace: $st'); } } @override void startListening({ double positiveSpeechThreshold = 0.5, double negativeSpeechThreshold = 0.35, int preSpeechPadFrames = 1, int redemptionFrames = 8, int frameSamples = 1536, int minSpeechFrames = 3, bool submitUserSpeechOnPause = false }) async { if (!_isInitialized) { await _initialize(); } await _headlessWebView?.webViewController?.evaluateJavascript( source: ''' startListeningImpl( $positiveSpeechThreshold, $negativeSpeechThreshold, $preSpeechPadFrames, $redemptionFrames, $frameSamples, $minSpeechFrames, $submitUserSpeechOnPause ) ''', ); } @override void stopListening() async { if (!_isInitialized) { _onErrorController.add('VAD not initialized'); return; } await _headlessWebView?.webViewController?.evaluateJavascript( source: 'stopListening()', ); } @override void dispose() { stopListening(); _headlessWebView?.dispose(); _onSpeechEndController.close(); _onSpeechStartController.close(); _onVADMisfireController.close(); _onErrorController.close(); } } VadHandlerBase createVadHandler() => VadHandlerNonWeb(); // main.dart import 'package:flutter/foundation.dart'; import 'package:flutter/material.dart'; import 'package:flutter_inappwebview/flutter_inappwebview.dart'; import 'package:permission_handler/permission_handler.dart'; import 'package:audioplayers/audioplayers.dart'; import 'package:vad/vad.dart'; import 'audio_utils.dart'; void main() async { WidgetsFlutterBinding.ensureInitialized(); if (!kIsWeb && defaultTargetPlatform == TargetPlatform.android) { await InAppWebViewController.setWebContentsDebuggingEnabled(kDebugMode); } runApp(const MyApp()); } class MyApp extends StatelessWidget { const MyApp({super.key}); @override Widget build(BuildContext context) { return MaterialApp( title: 'VAD Example', theme: ThemeData(primarySwatch: Colors.blue), home: const MyHomePage(), ); } } class MyHomePage extends StatefulWidget { const MyHomePage({super.key}); @override State createState() => _MyHomePageState(); } class Recording { final List samples; Recording(this.samples); } class _MyHomePageState extends State { List recordings = []; final AudioPlayer _audioPlayer = AudioPlayer(); final _vadHandler = VadHandler.create(); bool isListening = false; int preSpeechPadFrames = 10; int redemptionFrames = 8; bool submitUserSpeechOnPause = false; @override void initState() { super.initState(); _vadHandler.onSpeechStart.listen((_) { debugPrint('Speech detected.'); }); _vadHandler.onSpeechEnd.listen((List samples) { setState(() { recordings.add(Recording(samples)); }); debugPrint('Speech ended, recording added.'); }); _vadHandler.onVADMisfire.listen((_) { debugPrint('VAD misfire detected.'); }); _vadHandler.onError.listen((String message) { debugPrint('Error: $message'); }); } Future _playRecording(Recording recording) async { try { // Convert to WAV only when needed for playback String uri = AudioUtils.createWavUrl(recording.samples); await _audioPlayer.play(UrlSource(uri)); } catch (e) { debugPrint('Error playing audio: $e'); } } @override void dispose() { _audioPlayer.dispose(); _vadHandler.dispose(); super.dispose(); } Widget _buildRecordingItem(Recording recording, int index) { return ListTile( leading: const Icon(Icons.mic), title: Text('Recording ${index + 1}'), subtitle: Text('${recording.samples.length} samples (${recording.samples.length ~/ 16000} seconds)'), trailing: IconButton( icon: const Icon(Icons.play_arrow), onPressed: () => _playRecording(recording), ), ); } @override Widget build(BuildContext context) { return Scaffold( appBar: AppBar(title: const Text("VAD Example")), body: Center( child: Column( children: [ Expanded( child: ListView.builder( itemCount: recordings.length, itemBuilder: (context, index) { return _buildRecordingItem(recordings[index], index); }, ), ), Padding( padding: const EdgeInsets.all(8.0), child: Column( children: [ ElevatedButton( onPressed: () async { setState(() { if (isListening) { _vadHandler.stopListening(); } else { _vadHandler.startListening( submitUserSpeechOnPause: submitUserSpeechOnPause, preSpeechPadFrames: preSpeechPadFrames, redemptionFrames: redemptionFrames ); } isListening = !isListening; }); }, child: Text(isListening ? "Stop Listening" : "Start Listening"), ), const SizedBox(height: 10), ElevatedButton( onPressed: () async { final status = await Permission.microphone.request(); debugPrint("Microphone permission status: $status"); //logMessage("Microphone permission status: $status"); }, child: const Text("Request Microphone Permission"), ), ], ), ), ], ), ), ); } } // audio_utils.dart import 'dart:convert'; import 'dart:typed_data'; class AudioUtils { static String createWavUrl(List samples) { final wavData = float32ToWav(samples); final base64Wav = base64Encode(wavData); return 'data:audio/wav;base64,$base64Wav'; } static Uint8List float32ToWav(List float32Array) { const int sampleRate = 16000; const int byteRate = sampleRate * 2; // 16-bit = 2 bytes per sample final int totalAudioLen = float32Array.length * 2; final int totalDataLen = totalAudioLen + 36; final ByteData buffer = ByteData(44 + totalAudioLen); // Write WAV header _writeString(buffer, 0, 'RIFF'); buffer.setInt32(4, totalDataLen, Endian.little); _writeString(buffer, 8, 'WAVE'); _writeString(buffer, 12, 'fmt '); buffer.setInt32(16, 16, Endian.little); buffer.setInt16(20, 1, Endian.little); buffer.setInt16(22, 1, Endian.little); buffer.setInt32(24, sampleRate, Endian.little); buffer.setInt32(28, byteRate, Endian.little); buffer.setInt16(32, 2, Endian.little); buffer.setInt16(34, 16, Endian.little); _writeString(buffer, 36, 'data'); buffer.setInt32(40, totalAudioLen, Endian.little); // Convert and write audio data int offset = 44; for (double sample in float32Array) { sample = sample.clamp(-1.0, 1.0); final int pcm = (sample < 0 ? sample * 0x8000 : sample * 0x7FFF).toInt(); buffer.setInt16(offset, pcm, Endian.little); offset += 2; } return buffer.buffer.asUint8List(); } static void _writeString(ByteData view, int offset, String string) { for (int i = 0; i < string.length; i++) { view.setUint8(offset + i, string.codeUnitAt(i)); } } } ```

Stacktrace/Logs

[IOSInAppWebViewController] (iOS) WebView ID 1127219822522392511561692122098150248154100127 calling "onMicrophoneCaptureStateChanged" using {newState: 1, oldState: 0}
[IOSInAppWebViewController] (iOS) WebView ID 1127219822522392511561692122098150248154100127 calling "onConsoleMessage" using {messageLevel: 0, message: [VAD] initializing vad}
flutter: VAD Console: [VAD] initializing vad
flutter: VAD Console: [VAD] vad is initialized
flutter: VAD Console: VAD started successfully
flutter: VAD Log: VAD started successfully
[IOSInAppWebViewController] (iOS) WebView ID 1127219822522392511561692122098150248154100127 calling "onConsoleMessage" using {messageLevel: 0, message: [VAD] vad is initialized}
[IOSInAppWebViewController] (iOS) WebView ID 1127219822522392511561692122098150248154100127 calling "onConsoleMessage" using {messageLevel: 1, message: VAD started successfully}
[IOSInAppWebViewController] (iOS) WebView ID 1127219822522392511561692122098150248154100127 calling "logMessage" using [VAD started successfully]
flutter: Speech detected.
flutter: Speech ended, recording added.
[IOSInAppWebViewController] (iOS) WebView ID 1127219822522392511561692122098150248154100127 calling "handleEvent" using [onSpeechEnd, {"audioData":[0.00022896744485478848,0.0002394227631157264,0.0002058379031950608,0.00012442556908354163,0.000043014733819290996,-0.00003522740371408872,-0.00007867643580539152,0.00006905733607709408,0.00018232467118650675,0.00024367480364162475,0.00029036117484793067,0.0002749766281340271,0.00021486073092...
[IOSInAppWebViewController] (iOS) WebView ID 1127219822522392511561692122098150248154100127 calling "onMicrophoneCaptureStateChanged" using {oldState: 1, newState: 2}

Flutter version

v3.24.3

Operating System, Device-specific and/or Tool

iOS 18.1 Beta 2

Plugin version

v6.1.5

Additional information

No response

Self grab

keyur2maru commented 1 month ago

The issue was resolved by listening for onMicrophoneCaptureStateChanged and changing the value again to active and setting AVAudioSessionOptions.mixWithOthers for the audio session which in my case for audioplayers was using setAudioContext.

await _audioPlayer.setAudioContext(audioplayers.AudioContext(iOS: audioplayers.AudioContextIOS(options: const {audioplayers.AVAudioSessionOptions.mixWithOthers})));

      onMicrophoneCaptureStateChanged: (controller, state, _) {
        if (state == MediaCaptureState.ACTIVE) {
          debugPrint('Microphone capture is no longer recording, starting recording again');
          _headlessWebView?.webViewController?.setMicrophoneCaptureState(state: MediaCaptureState.ACTIVE);
        }
        return Future.value();
      },
github-actions[bot] commented 2 weeks ago

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug and a minimal reproduction of the issue.