Long latency using Yolo tflite model

Hi all,

I exported a generic (and trained) Yolov8n model into tflite format and loaded in a react native app (no expo).

After I have understood the output format, I have been trying to execute a real time inference but I have been facing 2 issues :

I have a very long latency despite the fact my model weight is only 6mo. The inference time is about 200ms (which is long but explained by the image size of 640 I guess) but what is the weirdest part is that the camera is freezing during much more than that time. For comparison, I have been using also the efficientDet model from the example and it worked fine in real time with very low latency. I actually have no idea what could cause that issue.
Sorry if this is not completely related to this repo but it might be. My confidence score from the outputs are very very low (0.0000123) and consequently not exploitable. I suspect a wrong input frame during the inference frame which could explain this low score as i'm pretty confident about what I record with my camera. Any insights about what I could possibly do wrong in that case ?

Here is the code :

...
  const model  = useTensorflowModel(require('./android/app/src/main/assets/yolov8n_float16_googleColab.tflite'))//model path
  const actualModel = model.state === 'loaded' ? model.model : undefined

...

const frameProcessor = useFrameProcessor(
    (frame) => {
      'worklet'

        const resized = resize(frame, {
          scale: {
            width: 640 ,
            height: 640 ,
          },
          pixelFormat: 'rgb',
          dataType: 'float32',

        });

        const start = performance.now();

        // Run model with given input buffer synchronously
        const outputs = actualModel.runSync([resized])
        const end = performance.now();
        const executionTime = end - start;
        console.log(`Inference time : ${executionTime.toFixed(2)} ms`);

    const x = valeurs.slice(0,8400)
    const y = valeurs.slice(8400,8400*2)
    const width = valeurs.slice(8400*2,8400*3)
    const height = valeurs.slice(8400*3,8400*4)
    const class1 = valeurs.slice(8400*4,8400*5)
    const class2 = valeurs.slice(8400*5,8400*6)

    const reshaped = []
    for( let i = 0; i < 8400; i++) {
        const detection = []
        detection.push(x[i])
        detection.push(y[i])
        detection.push(width[i])
        detection.push(height[i])
        detection.push(class1[i])
        detection.push(class2[i])
        reshaped.push(detection)
    }

console.log('X', x.slice(0,10))
console.log('Y', y.slice(0,10))
console.log('Width', width.slice(0,10))
console.log('Height', height.slice(0,10))
console.log('Class1', class1.slice(0,10))
console.log('Class2', class2.slice(0,10))

              });

My return jsx :

 <View style={{ width: '100%', aspectRatio: 3/4}}>
  <Camera
    device={device}
    style={StyleSheet.absoluteFill}
    isActive={true}
    frameProcessor={frameProcessor}
    pixelFormat="yuv"
  />
</View>

Thanks for the help!

mrousavy / react-native-fast-tflite

Long latency using Yolo tflite model #96