How can I load model in isolate?

[sejun2]

I have done loadModel and getImagePrediction in isolate but it send a exception ' Exception: Null check operator used on a null value'. Any solution for this?

[tsubauaaa]

@sejun2

'Exception: Null check operator used on a null value'

It's not Null (maybe!). But in fact, it seems to be null. If you don't mind, could you please share the code you wrote? Also, are you trying to do this with a Live camera? If so, I haven't verified that Live works yet.

[sejun2]

@tsubauaaa Thank you for your kind reply. Sure I'll get it to you soon. And also yeah, I tried it in preview streaming mode. This is logic what I wrote.

1. Make a Uint8List file.
2. Run ML model in isolate
3. Then it send such exception.

Actually, I don't think that it is a fault of ML loading but isolate. Anyway repeatedly I'll get my code to you soon.

ps. Because of the exception, Im trying to make it possible in android native code. Could you please any method to fulfill my purpose in android native?

[tsubauaaa]

@sejun2 This app has a Live mode, so it might work! https://github.com/pytorch/android-demo-app/tree/master/D2Go

[sejun2]

@tsubauaaa Here is my full code. Even if it is a lot of mess, you can check detect method and doMLWork in compute method

_I used camerawesome, Get, pathprovider and image lib.

import 'dart:async';
import 'dart:io';
import 'dart:isolate';
import 'dart:typed_data';
import 'package:camerawesome/camerapreview.dart';
import 'package:camerawesome/camerawesome_plugin.dart';
import 'package:flutter/foundation.dart';
import 'package:flutter/material.dart';
import 'package:flutter/services.dart';
import 'package:flutter_d2go/flutter_d2go.dart';
import 'package:get/get.dart';
import 'package:image/image.dart' as ImageLib;
import 'package:path_provider/path_provider.dart';
import 'package:permission_handler/permission_handler.dart';

typedef IsolateFunction(message);

doMLWork(Uint8List _data) async {
  print('doMLWork called... ');
  try {
    return await FlutterD2go.getImagePredictionByData(
        data: _data, minScore: 0.3);
  } catch (e) {
    print(e);
  } finally {}
}

class CameraPage extends StatefulWidget {
  CameraPage({Key? key}) : super(key: key);

  @override
  _CameraPageState createState() => _CameraPageState();
}

class _CameraPageState extends State<CameraPage> {
  final ValueNotifier<CameraFlashes> _switchFlash =
      ValueNotifier(CameraFlashes.NONE);
  final ValueNotifier<Sensors> _sensor = ValueNotifier(Sensors.BACK);
  final ValueNotifier<CaptureModes> _captureMode =
      ValueNotifier(CaptureModes.PHOTO);
  final ValueNotifier<Size> _size = ValueNotifier(const Size(1920, 1080));

  // Controllers
  final PictureController _pictureController = PictureController();
  final VideoController _videoController = VideoController();

  String detectionModel = '';

  ///Load ML model
  Future loadModel() async {
    print('loadModel called...');

    String modelPath = 'assets/models/d2go.pt';
    String labelPath = 'assets/models/classes.txt';
    try {
      await FlutterD2go.loadModel(
        modelPath: modelPath,
        labelPath: labelPath,
      );
      setState(() {});
    } on PlatformException {
      debugPrint('only supported for android so far');
    }
  }

  ///do Image detect process
  Future detect(Uint8List _imageData) async {
    print('detect called...');

    await loadModel();

    //make file
    //File _file = await writeImageFile(_imageData);
    //print('_file : $_file');
    //do ML in isolate
    //var predictions = await doMLWork(_imageData);
    await compute(doMLWork, _imageData);
    //print('predictions : ${predictions.toString()}');

/*
    final predictions = await FlutterD2go.getImagePrediction(
      image: await getImageFile(),
      minScore: 0.7,
    );
*/

/*
    print(predictions);
    setState(() {
      if (predictions.asMap()['detectedClass'] != null) {
        detectionModel = predictions.asMap()['detectedClass'];
      }
    });
*/
    //await deleteImageFile();
  }

  ///Streaming mutex
  bool streamMutex = true;

  late Stream<Uint8List> previewDataStream;
  late StreamController<Uint8List> previewDataStreamController;

  ///Request permission
  _requestPermission() async {
    print('_requestPermission called...');

    //permission list
    Map<Permission, PermissionStatus> permissionMap =
        await [Permission.manageExternalStorage, Permission.storage].request();
    for (var permission in permissionMap.values) {
      if (permission.isPermanentlyDenied) {
        openAppSettings();
      }
    }
  }

  _initResources() async {
    /* final imageFileStream = await getImageFileStream();
    imageFileStream.listen((event) {
      print('imageFileStream event occurred..');
      print('${event.toString()}');
    });*/

    previewDataStreamController = StreamController();
    previewDataStream = previewDataStreamController.stream;
  }

  _releaseResources() async {
    previewDataStreamController.close();
    CameraAwesomeState().dispose();
  }

  @override
  void dispose() {
    print('dispose called...');
    _size.dispose();
    _captureMode.dispose();
    _releaseResources();
    super.dispose();
  }

  @override
  void initState() {
    //_requestPermission();
    _initResources();
    super.initState();
  }

  ///open IOSink
  getIOSink() async {
    final tempDirPath = await getTempPath();
    final _file = File(tempDirPath);
    return _file.openWrite();
  }

  ///Get temporary directory path
  Future<String> getTempPath() async {
    final tempDir = await getTemporaryDirectory();
    return tempDir.path;
  }

  ///Get image file
  Future<File> getImageFile() async {
    print('getImageFile...');
    final tempPath = await getTempPath();
    return File(
      '$tempPath/tempImageFile',
    );
  }

  ///Get image file stream
  Future<Stream> getImageFileStream() async {
    final _file = await getImageFile();
    return _file.openRead();
  }

  ///Write image bytes to file
  Future<File> writeImageFile(Uint8List _uint8List) async {
    print('writeImageFile...');
    final File _file = await getImageFile();
    await _file.writeAsBytes(_uint8List, flush: true);
    return _file;
  }

  ///Read image file
  Future<Uint8List?> readImageFile() async {
    print('readImageFile...');
    Uint8List? result;
    try {
      final File? _file = await getImageFile();
      result = await _file?.readAsBytes();
      print('result : ${result.toString()}');
    } catch (e) {
      print('readImageFile exception :: $e');
    } finally {
      print('readImageFile finally...');
    }
    return result;
  }

  ///Read image file by stream
  Future<Stream?> readImageFileByStream() async {
    print('readImageFileByStream...');
    Stream? _stream;
    try {
      final File? _file = await getImageFile();
      if (_file != null) {
        _stream = _file.openRead();
      }
    } catch (e) {
      print(e);
    } finally {
      if (_stream != null) {}
    }
    return _stream;
  }

  ///Delete image file
  deleteImageFile() async {
    final File _file = await getImageFile();
    await _file.delete();
    print('deleteImageFile...');
  }

  ///Return image file
  convertUint8ListToPng(Uint8List _uint8List) {
    return Image.memory(_uint8List);
  }

  ///Run on isolate function
  runOnIsolate(IsolateFunction run, Uint8List data) {
    print('runOnIsolate called...');
    compute(run, data);
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      body: SingleChildScrollView(
        child: Stack(children: [
          Column(
            crossAxisAlignment: CrossAxisAlignment.center,
            children: [
              SizedBox(
                height: 500,
                child: CameraAwesome(

                    ///Image_stream function
                    imagesStreamBuilder: (imageStream) {
                      imageStream!.listen((data) async {
                        if (streamMutex) {
                          print('imageStream working...');
                          streamMutex = false;
                          //logic here...
                          //await writeImageFile(data);

                          //add image data to stream
                          if (!previewDataStreamController.isClosed) {
                            previewDataStreamController.add(data);
                          }
                          //runOnIsolate(detect, data);
                          detect(data);
                          await Future.delayed(
                              const Duration(milliseconds: 500), () {
                            streamMutex = true;
                          }); //await time - 700 milliseconds
                        }
                      });
                    },
                    photoSize: _size,
                    sensor: _sensor,
                    captureMode: _captureMode),
              ),
              Column(
                children: [
                  Wrap(
                    alignment: WrapAlignment.center,
                    children: [
                      TextButton(
                        onPressed: () async {
                          await readImageFile();
                        },
                        child: const Text('ReadFile'),
                      ),
                      TextButton(
                        child: const Text('buildImageWidget'),
                        onPressed: () {},
                      ),
                      TextButton(
                        child: const Text('dep'),
                        onPressed: () {},
                      ),
                      Text(
                        'Detected : $detectionModel',
                        textAlign: TextAlign.center,
                        style: const TextStyle(
                            color: Colors.black, fontWeight: FontWeight.bold),
                      ),
                      StreamBuilder(
                          builder: (context, snapshot) {
                            var result = snapshot.data;
                            //print('snapshot.data : $result');

                            if (!snapshot.hasData || snapshot.hasError) {
                              return const Text('something\'s wrong');
                            } else {
                              return Image.memory(
                                result as Uint8List,
                              );
                            }
                          },
                          stream: previewDataStream),
                    ],
                  ),
                ],
              ), //end of stack's first child
            ],
          ),
        ]),
      ),
    );
  }
}

[sejun2]

@sejun2 This app has a Live mode, so it might work! https://github.com/pytorch/android-demo-app/tree/master/D2Go

I also can see running ml model in preview streaming mode, but it consume many resources. That's why I wanna make it isolate. It works perfectly without isolate.

[tsubauaaa]

@sejun2 Thank you for sharing this code. First of all, I will implement the inference function of the camera streaming image. After that, I'll try the isolated implementation.

[tsubauaaa]

@sejun2 I've implemented a streaming image inference feature, can you give it a try with the latest flutter_d2go?

[sejun2]

Thank you for reply. Yeah It works!.

Im trying to make it work in android native with coroutines cause flutter isolate couldn't succeded for that. What I am doing now is...

1. Attach android activity which is doing camera and ML things to flutter.
2. Use coroutines for running model.