Long latency using Yolo tflite model

[manu13008]

Hi all,

I exported a generic (and trained) Yolov8n model into tflite format and loaded in a react native app (no expo).

After I have understood the output format, I have been trying to execute a real time inference but I have been facing 2 issues :

• I have a very long latency despite the fact my model weight is only 6mo. The inference time is about 200ms (which is long but explained by the image size of 640 I guess) but what is the weirdest part is that the camera is freezing during much more than that time. For comparison, I have been using also the efficientDet model from the example and it worked fine in real time with very low latency. I actually have no idea what could cause that issue.

• Sorry if this is not completely related to this repo but it might be. My confidence score from the outputs are very very low (0.0000123) and consequently not exploitable. I suspect a wrong input frame during the inference frame which could explain this low score as i'm pretty confident about what I record with my camera. Any insights about what I could possibly do wrong in that case ?

Here is the code :

...
  const model  = useTensorflowModel(require('./android/app/src/main/assets/yolov8n_float16_googleColab.tflite'))//model path
  const actualModel = model.state === 'loaded' ? model.model : undefined

...

const frameProcessor = useFrameProcessor(
    (frame) => {
      'worklet'

        const resized = resize(frame, {
          scale: {
            width: 640 ,
            height: 640 ,
          },
          pixelFormat: 'rgb',
          dataType: 'float32',

        });

        const start = performance.now();

        // Run model with given input buffer synchronously
        const outputs = actualModel.runSync([resized])
        const end = performance.now();
        const executionTime = end - start;
        console.log(`Inference time : ${executionTime.toFixed(2)} ms`);

    const x = valeurs.slice(0,8400)
    const y = valeurs.slice(8400,8400*2)
    const width = valeurs.slice(8400*2,8400*3)
    const height = valeurs.slice(8400*3,8400*4)
    const class1 = valeurs.slice(8400*4,8400*5)
    const class2 = valeurs.slice(8400*5,8400*6)

    const reshaped = []
    for( let i = 0; i < 8400; i++) {
        const detection = []
        detection.push(x[i])
        detection.push(y[i])
        detection.push(width[i])
        detection.push(height[i])
        detection.push(class1[i])
        detection.push(class2[i])
        reshaped.push(detection)
    }

console.log('X', x.slice(0,10))
console.log('Y', y.slice(0,10))
console.log('Width', width.slice(0,10))
console.log('Height', height.slice(0,10))
console.log('Class1', class1.slice(0,10))
console.log('Class2', class2.slice(0,10))

              });

My return jsx :

 <View style={{ width: '100%', aspectRatio: 3/4}}>
  <Camera
    device={device}
    style={StyleSheet.absoluteFill}
    isActive={true}
    frameProcessor={frameProcessor}
    pixelFormat="yuv"
  />
</View>

Thanks for the help!

[maintenance-hans[bot]]

Guten Tag, Hans here! 🍻

Thanks for reporting your issue. It sounds like you have a few challenges with your Yolo model's latency and low confidence scores. However, I notice that you haven't included any logs that would help mrousavy diagnose the problem further.

For troubleshooting, please provide logs from your React Native app. You can gather logs on Android using adb logcat or from Xcode for iOS. This information is crucial for pinpointing the issue accurately.

Keep us updated, and we look forward to your logs!

Note: If you think I made a mistake, please ping @mrousavy to take a look.

[francesco-clementi-92]

You can try to export the model with quantization and changing the datatype to uint8, but I don't think this library works with quantised model

[manu13008]

@francesco-clementi-92 Quantization didn't work fro yolo model.

The best I could do was to decrease image size down to 320*256 or something similar but inference's still around 57 ms per frame, which isn't enough fast for real time analysis.

My phone is not top of the market as well (Xiaomi Poco X5 Pro) so I guess with a better phone, this could go faster.

I solved my low inference score problem. It was due to the rotation of the image. By default, the image of the camera is rorated 90° and my model can only see objects in a specific rotation angle.

[francesco-clementi-92]

@manu13008 what do you mean for "by default, the image of the camera is rotated 90 degree"?

I was going to implement this package with the resize plugin, but the resize is not going to work async so I stopped.

Be aware that with quantised model usually you have to dequantized the input by multiply it with the quantization scale.

[manu13008]

@francesco-clementi-92

To give you an example :

What I see on my phone while recording a video :

What the frame sees :

So I had to use the rotation argument in the resize function as follow :

   const dimensions = {width : 320 , height : 320}
        const resized = resize(frame, {
          scale: {
            width: dimensions.width ,
            height: dimensions.height ,
          },
          pixelFormat: 'rgb',
          // dataType: 'int8',
          dataType: 'float32',
          **rotation : '90deg',**

        });

I'm still a newbie using these kind of tools, so it may sound very obvious for certain people but I hope I could help some others :)

I used the resized plugin with async and it worked. Took around 2ms to resize each frame. Problem went later with inference time.