iOS issue with Background service plugin / outputRawScores

[bobosette]

Hi @Caldarie. I'm testing the app on iOS but the package doesn't work. I have followed the guidelines for the implementation but it still doesn't work. This is the exception:

`Unhandled Exception: MissingPluginException(No implementation found for method loadModel on channel tflite_audio)

0 MethodChannel._invokeMethod (package:flutter/src/services/platform_channel.dart:165:7)

══╡ EXCEPTION CAUGHT BY SERVICES LIBRARY ╞══════════════════════════════════════════════════════════ The following MissingPluginException was thrown while activating platform stream on channel AudioRecognitionStream: MissingPluginException(No implementation found for method listen on channel AudioRecognitionStream) When the exception was thrown, this was the stack: #0 MethodChannel._invokeMethod (package:flutter/src/services/platform_channel.dart:165:7) #1 EventChannel.receiveBroadcastStream. (package:flutter/src/services/platform_channel.dart:506:9) ════════════════════════════════════════════════════════════════════════════════════════════════════` How can I fix it??? Thank you.

[Caldarie]

Hi @bobosette

thanks for reporting a potential bug. I will get back to you once I have investigated the issue.

[bobosette]

I did more test and I noticed that your example work, so maybe in my app there's a configuration problem. I'll let you know

[Caldarie]

Please do. Also i have updated the instructions to be clearer on readme to help you with the installation.

Edit: I also added more pictures as well.

[bobosette]

I followed those. But it didn't work. And your example work. Now I have to investigate to find the differences

Il Gio 20 Gen 2022, 08:47 Michael Nguyen @.***> ha scritto:

Please do. Also i have made the instructions clearer on readme to help you with the installation

— Reply to this email directly, view it on GitHub https://github.com/Caldarie/flutter_tflite_audio/issues/30#issuecomment-1017201366, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADEWB4OUIJX4ESGD6JMK5MTUW64Z7ANCNFSM5MKLXLCA . You are receiving this because you were mentioned.Message ID: @.***>

[Caldarie]

What I meant was that I added new instructions just now.

I had tested the instructions on the readme, and fixed some outdated information.

[bobosette]

Ah ok 🤣

Il Gio 20 Gen 2022, 09:07 Michael Nguyen @.***> ha scritto:

What I meant was that I added new instructions just now.

I had tested the instructions on the readme, and fixed some outdated information.

— Reply to this email directly, view it on GitHub https://github.com/Caldarie/flutter_tflite_audio/issues/30#issuecomment-1017213623, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADEWB4M7SSEVP3MSG2VIR5LUW67DNANCNFSM5MKLXLCA . You are receiving this because you were mentioned.Message ID: @.***>

[bobosette]

Still doesn't work. But I can believe it, the configuration is exactly the same of your example app

[bobosette]

It's like the platform channel doesn't find all tflite method: load Model, startAudioRecognition, stopAudioRecognition

[Caldarie]

If you're willing to share your project with me, I can take a look for you.

[bobosette]

I'm doing a final test. The app runs the recognizer as a background service cause it has to work both in foreground and in background. To do this I'm using flutter_background_service. On Android it works perfectly, but maybe on iOS throws this exception.

Il Gio 20 Gen 2022, 11:47 Michael Nguyen @.***> ha scritto:

If you're willing to share your project with me, I can take a look for you.

— Reply to this email directly, view it on GitHub https://github.com/Caldarie/flutter_tflite_audio/issues/30#issuecomment-1017354594, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADEWB4NCJL3GADOVHXA6LDDUW7R4TANCNFSM5MKLXLCA . You are receiving this because you were mentioned.Message ID: @.***>

[Caldarie]

In that case can you try this for me? I believe that iOS has not properly installed the plugin. In that case you can try on the terminal:

1. flutter pub add tflite_audio
2. Run flutter pub get
3. CD into project iOS folder and run pod install
4. Run flutter clean
5. Run flutter run

This should rebuild it

[bobosette]

I tried this: in my app I added an example page with the code of your example app: and it's work, so the problem is between tflite-audio and the background service, that running into an isolate. But I can't understand why on Android everything works fine

Il Gio 20 Gen 2022, 11:56 Michael Nguyen @.***> ha scritto:

In that case can you try this for me? I believe that iOS has not properly installed the plugin. In that case you can try on the terminal:

1. flutter pub add tflite_audio
2. Run flutter pub get
3. CD into project iOS folder and run pod install
4. Run flutter clean
5. Run flutter run

This should rebuild it

— Reply to this email directly, view it on GitHub https://github.com/Caldarie/flutter_tflite_audio/issues/30#issuecomment-1017367303, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADEWB4KN2TIKVUCQTPV43E3UW7S63ANCNFSM5MKLXLCA . You are receiving this because you were mentioned.Message ID: @.***>

[bobosette]

Ok...I found something. If I put outputRawScores = true the app crashes. If I put false it works (only foreground, not background).

Il Gio 20 Gen 2022, 12:42 Roberto Bobo Prandin @.***> ha scritto:

I tried this: in my app I added an example page with the code of your example app: and it's work, so the problem is between tflite-audio and the background service, that running into an isolate. But I can't understand why on Android everything works fine

Il Gio 20 Gen 2022, 11:56 Michael Nguyen @.***> ha scritto:

In that case can you try this for me? I believe that iOS has not properly installed the plugin. In that case you can try on the terminal:

1. flutter pub add tflite_audio
2. Run flutter pub get
3. CD into project iOS folder and run pod install
4. Run flutter clean
5. Run flutter run

This should rebuild it

— Reply to this email directly, view it on GitHub https://github.com/Caldarie/flutter_tflite_audio/issues/30#issuecomment-1017367303, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADEWB4KN2TIKVUCQTPV43E3UW7S63ANCNFSM5MKLXLCA . You are receiving this because you were mentioned.Message ID: @.***>

[Caldarie]

Hi, thanks for pointing out the bug. It seems this release has a lot of bugs due to how complex this plugin has become. Very difficult to find as well.

as for your dilemma, maybe you’re experiencing something like this?

[Caldarie]

Hi, when you say it crashes, do you mean it crashes when you use background service? Or it crashes without the background service?

[bobosette]

Without background service: if I put outputRawScores = true THE APP CRASHES

With background service: I have missing plugin implementation

For the background service I have to implement something like that but I'm using flutter_background_service and on android it works very well.

Do you think you can fix the outputRawScores bug? Cause for the logic of my app I need percentages!!!

Then we have to find a way to fix the background bug. Maybe I can provide an example for u

Il Gio 20 Gen 2022, 14:27 Michael Nguyen @.***> ha scritto:

Hi, when you say it crashes, do you mean it crashes when you use background service? Or it crashes without the background service?

— Reply to this email directly, view it on GitHub https://github.com/Caldarie/flutter_tflite_audio/issues/30#issuecomment-1017503640, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADEWB4NW2DNHPMKNK7XQXCLUXAETVANCNFSM5MKLXLCA . You are receiving this because you were mentioned.Message ID: @.***>

[Caldarie]

To help me pinpoint the issue, can you post the logs for me?

[bobosette]

For the outputRawScores bug I can't give you a log, cause when I debug and I start tflite audio it doesn't thrown an exception, it says LOST CONNECTION TO DEVICE. EXITED SIGTERM

Il Gio 20 Gen 2022, 14:45 Michael Nguyen @.***> ha scritto:

To help me pinpoint the issue, can you post the logs for me?

— Reply to this email directly, view it on GitHub https://github.com/Caldarie/flutter_tflite_audio/issues/30#issuecomment-1017519151, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADEWB4MQT2CJXJ7H4TI246DUXAGZPANCNFSM5MKLXLCA . You are receiving this because you were mentioned.Message ID: @.***>

[Caldarie]

Usually, you can get the logs if you run it on xcode.

[bobosette]

Mmm ok I can try but unfortunately not now cause I don't have a Mac at home, only at work. Do u know what it could be happened?

Il Gio 20 Gen 2022, 15:37 Michael Nguyen @.***> ha scritto:

Usually, you can get the logs if you run it on xcode.

— Reply to this email directly, view it on GitHub https://github.com/Caldarie/flutter_tflite_audio/issues/30#issuecomment-1017574635, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADEWB4OII6P5RK3ONF2K42TUXAM27ANCNFSM5MKLXLCA . You are receiving this because you were mentioned.Message ID: @.***>

[Caldarie]

Just making sure, you are using this plugin correct?

If yes, than I may know the reason why it does not work with iOS. According to the description A plugin to keep flutter apps running in the background. Currently only works with Android.

[bobosette]

No. I'm using this one: https://pub.dev/packages/flutter_background_service.

But the outputRawScores bug happen also in normal condition, without background service. For this bug, what could be happened?

[Caldarie]

Hmm, it’s difficult for me to locate the bug. If you do get the logs on Xcode, please let me know.

[bobosette]

Hi @Caldarie . Even if I can't give you more details about the outputRawScores bug, I found that bug also in your example, so you can easily reproduce it. Indeed, if you run your example app with outputRawScores = true the app crashes.

[Caldarie]

Hi @bobosette,

I had run tests on the example and on a new app for IOS, and am unable to reproduce the bug when using outputRawScores = true. You can find my logs below:

If you can get the logs for me somehow, I may be able to pinpoint as to why the problem occurs for you. Let me know if this is possible

[bobosette]

Mmm ok. Is it possible that the problem was the device? Currently i can test only on an iphone 6!!! Maybe is it too old???

[Caldarie]

Difficult to say. I’m running the tests on an iPhone 7, just one generation above iPhone 6.

I honestly am not sure where the problem lies. It could be an asynchronous operation, event could be null, incorrect configuration, many things,

[bobosette]

Unfortunately I can't give you a feedback from xcode cause I'm not able to run the app on xcode. Using vscode no problem, but if I try to run it on xcode the build failed. I tried everything, I don't know.

Il Sab 22 Gen 2022, 08:31 Michael Nguyen @.***> ha scritto:

Difficult to say. I’m running the tests on an iPhone 7, just one generation above iPhone 6.

I honestly am not sure where the problem lies. It could be an asynchronous operation, event could be null, incorrect configuration, many things,

— Reply to this email directly, view it on GitHub https://github.com/Caldarie/flutter_tflite_audio/issues/30#issuecomment-1019084267, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADEWB4L5ZZ7LYRONVY3VUTTUXJMOPANCNFSM5MKLXLCA . You are receiving this because you were mentioned.Message ID: @.***>

[Caldarie]

Understandable. No problems at all.

as for the background service, I think the reason it’s not working with this plugin is due to this function here: Enable background_fetch capability in xcode (optional).

From what I gather, iOS is very strict with background processes. Perhaps enabling this function will work. I can test this and get back to you.

[bobosette]

I've already enabled this function. flutter_background_service works with an isolate. I have to put all the code i want to work also in background inside this onStart() function and when flutter_background_service is called this function start. On Android works fine, on ios I get this strange error => missing plugin implementation but I don't understand why it should throw this error

[Caldarie]

If you dont mind, can you provide a snippet code on how you implemented flutter_background_service with tflite_audio? I would like to test this problem.

[bobosette]

This is the onStart() function. You can use this one like the example here https://pub.dev/packages/flutter_background_service/example

//isolate for background service=========================================
void onStart() {
  int count = 0;

  WidgetsFlutterBinding.ensureInitialized();

  //background service initialization

  FlutterBackgroundService().onDataReceived.listen((event) async {
    if (event!["action"] == "setAsForeground") {
      FlutterBackgroundService().setForegroundMode(true);
      return;
    }

    if (event["action"] == "setAsBackground") {
      FlutterBackgroundService().setForegroundMode(false);
    }

    //when recording stops
    if (event["action"] == "stopService") {
      TfliteAudio.stopAudioRecognition();

      FlutterBackgroundService().sendData(
        {"isStopped": count},
      );
      await Future.delayed(const Duration(milliseconds: 1000), () {});
      FlutterBackgroundService().stopBackgroundService();
    }

    //when user force recording to stop
    if (event["action"] == "stopServiceWithoutResult") {
      TfliteAudio.stopAudioRecognition();

      FlutterBackgroundService().sendData(
        {"isStoppedWithoutResult": count},
      );
      await Future.delayed(const Duration(milliseconds: 1000), () {});
      FlutterBackgroundService().stopBackgroundService();
    }
  });

  //model initialization
  TfliteAudio.loadModel(
      inputType: 'rawAudio', //'decodedWav', //'rawAudio',
      numThreads: 1,
      outputRawScores: true,
      isAsset: true,
      model:
          'assets/cough_model.tflite', 
      label:
          'assets/cough_label.txt' 
      );

  // bring to foreground
  FlutterBackgroundService().setForegroundMode(true);

  Stream<Map<dynamic, dynamic>>? result;

  result = TfliteAudio.startAudioRecognition(
    sampleRate: 44100, 
    //recordingLength: 44032, //16000, //44032,
    bufferSize: 11008, //to get 2-3 secs of analisys instead of 1
    numOfInferences: 99999,
    detectionThreshold: 0.7,
    averageWindowDuration: 1500,
    //minimumTimeBetweenSamples: this.minimumTimeBetweenSamples,
    //suppressionTime: this.suppressionTime,
  );

  //notification
  FlutterBackgroundService().setNotificationInfo(
    title: "Background",
    content: "$count",
  );

  ///Logs the results and assigns false when stream is finished.
  result.listen((audioEvent) {
    var scores = audioEvent['recognitionResult'];
    try {
      //handle Unexpected character error
      scores = jsonDecode(scores.replaceAll('"', ''));
    } catch (e) {
      Exception(e.toString());
    }

    double last = 0;
    double max = 1;
    if (scores is List) {
      List<double> l = scores.cast<double>();
      last = l.last;
      max = l.reduce(math.max);
    }

    //if cough has the higher percentage and this is higher than 0.72 > IS COUGH
    if (last == max && max > 0.72) {
      count = count + 1;
      FlutterBackgroundService().setNotificationInfo(
        title: "Background",
        content: "$count",
      );
      FlutterBackgroundService().sendData(
        {"result": count},
      );

    }
  }).onDone(() {});
}

[bobosette]

Hi @Caldarie . I have some news regarding the outputrawscores bug. I was able to run the app on Xcode and this is the stack trace:

2022-01-24 23:38:48.027009+0100 Runner[1500:344782] Metal API Validation Enabled 2022-01-24 23:38:49.703683+0100 Runner[1500:345332] flutter: Observatory listening on http://127.0.0.1:53245/x8B8jQDppKw=/ 2022-01-24 23:38:50.686115+0100 Runner[1500:344782] Initialized TensorFlow Lite runtime. INFO: Initialized TensorFlow Lite runtime. Input shape: [1, 44032] Input size 44032 Input type rawAudio labels: ["background_noise_snippets", "no", "yes", "go", "stop", "cough"] Permission granted start microphone recordingBuffer length: 11008 | inferenceCount: 1/99999 recordingBuffer length: 22016 | inferenceCount: 1/99999 recordingBuffer length: 33024 | inferenceCount: 1/99999 recordingBuffer length: 44032 | inferenceCount: 1/99999 reached threshold Running model 0.3 Raw Label Scores: ▿ 6 elements

• nan
• nan
• nan
• nan
• nan
• nan 2022-01-24 23:39:05.331440+0100 Runner[1500:345285] Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: 'Invalid number value (NaN) in JSON write' First throw call stack: (0x216a2b180 0x215c039f8 0x2169353a0 0x217531a48 0x21752f948 0x217532b84 0x216a0a944 0x21753233c 0x2173ee58c 0x2173ee2b4 0x104adae68 0x104ad7afc 0x104ae22e4 0x104ad0fa4 0x10e8236f4 0x10e824c78 0x10e82cbf4 0x10e82d8b4 0x10e83777c 0x21664a114 0x21664ccd4) libc++abi.dylib: terminating with uncaught exception of type NSException Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: 'Invalid number value (NaN) in JSON write' terminating with uncaught exception of type NSException Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: 'Invalid number value (NaN) in JSON write' terminating with uncaught exception of type NSException (lldb)

The configuration is this:

` TfliteAudio.loadModel( // numThreads: this.numThreads, // isAsset: this.isAsset, inputType: "rawAudio", outputRawScores: true, model: "assets/cough_model.tflite", label: "assets/cough_label.txt", );

TfliteAudio.startAudioRecognition( sampleRate: 44100, bufferSize: 22016, numOfInferences: 99999, // detectionThreshold: this.detectionThreshold, // averageWindowDuration: this.averageWindowDuration, // minimumTimeBetweenSamples: this.minimumTimeBetweenSamples, // suppressionTime: this.suppressionTime, ); `

[Caldarie]

Hi, sorry for the late reply.

Bit busy with work lately. I will get back to you when I have some time.

[bobosette]

Hi @Caldarie . Do you have news???

[Caldarie]

hi @bobosette , i will take a look at your problem sometime around next week. Unfortunately, this week has been very busy with work

Is your project urgent by the way?

[bobosette]

Should be urgent 😅! But I can understand if you're busy, job is job.

[Caldarie]

Hey, sorry for the long wait.

Can you tell me where you got this math.max from?

[bobosette]

Is a function from dart math to get the max number in a list.

Il Mar 1 Feb 2022, 12:15 Michael Nguyen @.***> ha scritto:

Hey, sorry for the long wait.

Can you tell me where you got this math.max from?

— Reply to this email directly, view it on GitHub https://github.com/Caldarie/flutter_tflite_audio/issues/30#issuecomment-1026732586, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADEWB4LQVHIIVQG2RG5WMF3UY66ERANCNFSM5MKLXLCA . You are receiving this because you were mentioned.Message ID: @.***>

[Caldarie]

I have used dart math, but got this error instead.

 The argument type 'double' can't be assigned to the parameter type 'double Function(double, double)'.
lib/main.dart:144
            max = l.reduce(max);

[bobosette]

That part of code works. I'm sure cause with android. With ios i can't arrive to thst point, the app crash before

Il Mar 1 Feb 2022, 12:28 Michael Nguyen @.***> ha scritto:

Reopened #30 https://github.com/Caldarie/flutter_tflite_audio/issues/30.

— Reply to this email directly, view it on GitHub https://github.com/Caldarie/flutter_tflite_audio/issues/30#event-5987491128, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADEWB4K2UVGW54MRMDDNYHLUY67WJANCNFSM5MKLXLCA . You are receiving this because you were mentioned.Message ID: @.***>

[Caldarie]

Oh ok, no problems. Ill let you know how my investigation will go

[bobosette]

Thank you

Il Mar 1 Feb 2022, 12:44 Michael Nguyen @.***> ha scritto:

Oh ok, no problems. Ill let you know how my investigation will go

— Reply to this email directly, view it on GitHub https://github.com/Caldarie/flutter_tflite_audio/issues/30#issuecomment-1026755213, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADEWB4J655BKJJ2DGBYDQ53UY7BQBANCNFSM5MKLXLCA . You are receiving this because you were mentioned.Message ID: @.***>

[Caldarie]

Anyway you could share your main.dart? Unfortunately, i do not have much time to try and understand the background_service plugin and implement it.

[bobosette]

` void main() async { WidgetsFlutterBinding.ensureInitialized(); initializeService();

runApp(const MyApp());

}

// to ensure this executed // run app from xcode, then from xcode menu, select Simulate Background Fetch void onIosBackground() { WidgetsFlutterBinding.ensureInitialized(); }

Future initializeService() async { final service = FlutterBackgroundService(); await service.configure( androidConfiguration: AndroidConfiguration( // this will executed when app is in foreground or background in separated isolate onStart: onStart,

  // auto start service
  autoStart: false,
  isForegroundMode: true,
),
iosConfiguration: IosConfiguration(
  // auto start service
  autoStart: false,

  // this will executed when app is in foreground in separated isolate
  onForeground: onStart,

  // you have to enable background fetch capability on xcode project
  onBackground: onIosBackground,
),

); } `

But can you provide for me a working example on ios with outputrawscores = true. Not with flutter background service, a normal one, cause i have problem also without background service

[Caldarie]

Hi, the code you have given me above is NOT from your main.dart. That code is from the background_service example.

I understand if you cannot share the code; though, I will be unable to quickly identify the issue considering that you said it was urgent.

As for the outputrawscores, I cannot seem to reproduce the problem. Can you tell me your tflite_audio version, your parameters, as well as post the logs from flutter doctor -v

[bobosette]

on my main I use that code....

I use the ^0.2.2+2 , for the doctor I will tell you as soon as I get in front of my mac

[Caldarie]

Hi, can you try to update to 0.2.2+3? Maybe that will fix the issue.

As for the code you have given me, I would like to know where you called onstart(). Also, I would like to see your imports and widgets as well. Hence why I have asked for your main.dart.

[bobosette]

import 'dart:async';

import 'package:flutter/material.dart'; import 'package:flutter_background_service/flutter_background_service.dart';

void main() { WidgetsFlutterBinding.ensureInitialized(); initializeService(); runApp(MyApp()); }

Future initializeService() async { final service = FlutterBackgroundService(); await service.configure( androidConfiguration: AndroidConfiguration( // this will executed when app is in foreground or background in separated isolate onStart: onStart,

  // auto start service
  autoStart: true,
  isForegroundMode: true,
),
iosConfiguration: IosConfiguration(
  // auto start service
  autoStart: true,

  // this will executed when app is in foreground in separated isolate
  onForeground: onStart,

  // you have to enable background fetch capability on xcode project
  onBackground: onIosBackground,
),

); }

// to ensure this executed // run app from xcode, then from xcode menu, select Simulate Background Fetch void onIosBackground() { WidgetsFlutterBinding.ensureInitialized(); print('FLUTTER BACKGROUND FETCH'); }

class MyApp extends StatefulWidget { @override _MyAppState createState() => _MyAppState(); }

class _MyAppState extends State { String text = "Stop Service"; @override Widget build(BuildContext context) { return MaterialApp( home: Scaffold( appBar: AppBar( title: const Text('Service App'), ), body: Column( children: [ StreamBuilder<Map<String, dynamic>?>( stream: FlutterBackgroundService().onDataReceived, builder: (context, snapshot) { if (!snapshot.hasData) { return Center( child: CircularProgressIndicator(), ); }

            final data = snapshot.data!;
            DateTime? date = DateTime.tryParse(data["current_date"]);
            return Text(date.toString());
          },
        ),

        ElevatedButton(
          child: Text(text),
          onPressed: () async {
            final service = FlutterBackgroundService();
            var isRunning = await service.isServiceRunning();
            if (isRunning) {
              service.sendData(
                {"action": "stopService"},
              );
            } else {
              service.start();
            }

            if (!isRunning) {
              text = 'Stop Service';
            } else {
              text = 'Start Service';
            }
            setState(() {});
          },
        ),
      ],
    ),
    floatingActionButton: FloatingActionButton(
      onPressed: () {
        FlutterBackgroundService().sendData({
          "hello": "world",
        });
      },
      child: Icon(Icons.play_arrow),
    ),
  ),
);

} }

You can use this logic with the onstart() function that you already has. The most important thing is to understand why tflite_audio doesn't start, once we understand that it's just a good point.

For the outputRawScores bug i will try to update the package

[Caldarie]

I think you best talk to the creator of background_service about this issue. As I have almost no knowledge about background service, he or she may have an insight to your problem.