🐛 frame processing error frame not send to the flask server through axios

[niazikhan001]

What's happening?

const sendFrameToBackend = Worklets.createRunInJsFn(async (frame) => { console.log(Running inference on ${frame}); try {

  const response = await axios.post('http://192.168.1.3:5000/predict', {
    frame ,
  });
  console.log('Frame sent successfully:', response.data);
} catch (error) {
  console.error('Error sending frame to backend:', error);
}

})

const frameProcessor = useFrameProcessor((frame) => { 'worklet' // console.log(Frame: ${frame.width}x${frame.height} (${frame.pixelFormat})) // console.log(Running inference on ${frame});

// Send the frame to the backend when captured
sendFrameToBackend(frame)
})

**errors are**

 ERROR  Error sending frame to backend: [Error: Exception in HostObject::getPropertyNames: com.mrousavy.camera.core.FrameInvalidError: [capture/frame-invalid] Trying to access an already closed Frame! Are you trying to access the Image data outside of a Frame Processor's lifetime?

• If you want to use console.log(frame), use console.log(frame.toString()) instead.
• If you want to do async processing, use runAsync(...) instead.
• If you want to use runOnJS, increment it's ref-count: frame.incrementRefCount()]

Reproduceable Code

import React, { useState, useEffect } from 'react';
import {
  StyleSheet,
  Text,
  View,
} from 'react-native';
import {
  useCameraPermission,
  useMicrophonePermission,
  useCameraDevice,
  Camera,
  useFrameProcessor,runAtTargetFps,runAsync
} from 'react-native-vision-camera';
import axios from 'axios'; // Import Axios for making HTTP requests

function App() {
  const {
    hasPermission: cameraPermission,
    requestPermission: requestCameraPermission
  } = useCameraPermission();
  const {
    hasPermission: microphonePermission,
    requestPermission: requestMicrophonePermission,
  } = useMicrophonePermission();
  const cameraDevice = useCameraDevice('back');
  const [loading, setLoading] = useState(true);

  useEffect(() => {
    if (!cameraPermission) {
      requestCameraPermission();
    }
    if (!microphonePermission) {
      requestMicrophonePermission();
    }
  }, [cameraPermission, microphonePermission]);

  useEffect(() => {
    if (cameraPermission && microphonePermission) {
      setLoading(false);
    }
  }, [cameraPermission, microphonePermission]);

  const sendFrameToBackend = Worklets.createRunInJsFn(async (frame) => {
    console.log(`Running inference on ${frame}`);
    try {

      const response = await axios.post('http://192.168.1.3:5000/predict', {
        frame ,
      });
      console.log('Frame sent successfully:', response.data);
    } catch (error) {
      console.error('Error sending frame to backend:', error);
    }

  })
  const frameProcessor = useFrameProcessor((frame) => {
    'worklet'
    // console.log(`Frame: ${frame.width}x${frame.height} (${frame.pixelFormat})`)
    // console.log(`Running inference on ${frame}`);

    // Send the frame to the backend when captured
     runAsync(frame, () => {
      'worklet'
      runAtTargetFps(5,  () => {
        'worklet'
        try {
          console.log('Frame sent successfully:', frame);

          const response =   axios.post('http://192.168.53.39:5000/predict', {
            frame
          });
          console.log('Frame sent successfully:', response.data);
        } catch (error) {
          console.error('Error sending frame to backend:', error);
        }
      })
    })

  }, []);

  return (
    <View style={styles.container}>
      {loading ? (
        <Text style={styles.text}>Loading...</Text>
      ) : !cameraPermission || !microphonePermission ? (
        <Text style={styles.text}>
          Please grant camera and microphone permissions to use the app.
        </Text>
      ) : !cameraDevice ? (
        <Text style={styles.text}>No camera device available.</Text>
      ) : (
        <Camera
          frameProcessor={frameProcessor}
          style={styles.camera}
          device={cameraDevice}
          isActive={true}
        />
      )}
    </View>
  );
}

const styles = StyleSheet.create({
  container: {
    flex: 1,
    justifyContent: 'center',
    alignItems: 'center',
  },
  text: {
    fontSize: 24,
    fontWeight: 'bold',
    textAlign: 'center',
  },
  camera: {
    flex: 1,
    width: '100%',
  },
});

export default App;

Relevant log output

ERROR  Error sending frame to backend: [Error: Exception in HostObject::getPropertyNames: com.mrousavy.camera.core.FrameInvalidError: [capture/frame-invalid] Trying to access an already closed Frame! Are you trying to access the Image data outside of a Frame Processor's lifetime?
- If you want to use `console.log(frame)`, use `console.log(frame.toString())` instead.
- If you want to do async processing, use `runAsync(...)` instead.
- If you want to use runOnJS, increment it's ref-count: `frame.incrementRefCount()`]

Camera Device

<Camera
          frameProcessor={frameProcessor}
          style={styles.camera}
          device={cameraDevice}
          isActive={true}
        />

Device

samsung a10s

VisionCamera Version

"^3.9.0",

Can you reproduce this issue in the VisionCamera Example app?

No, I cannot reproduce the issue in the Example app

Additional information

• [ ] I am using Expo
• [X] I have enabled Frame Processors (react-native-worklets-core)
• [X] I have read the Troubleshooting Guide
• [X] I agree to follow this project's Code of Conduct
• [X] I searched for similar issues in this repository and found none.

[Soban71]

Yes exactly the same issue i am facing

[mrousavy]

Multiple things:

1. Format your code properly. It is impossible to read it this way.
2. The error message tells you exactly what your problem is:

   [capture/frame-invalid] Trying to access an already closed Frame! Are you trying to access the Image data outside of a Frame Processor's lifetime?
   - If you want to use `console.log(frame)`, use `console.log(frame.toString())` instead.
   - If you want to do async processing, use `runAsync(...)` instead.
   - If you want to use runOnJS, increment it's ref-count: `frame.incrementRefCount()`

   what is not clear about this?

3. Uploading a Frame like this directly doesn't just work. You'd need to properly convert it to some format before, e.g. to binary (frame.toArrayBuffer()) or encoding it to a smaller size using a native frame processor plugin (eg video encoder).

Overall, this is a pretty complex thing to do, uploading all frames to an axios backend at 30 FPS will probably explode your backend unless you do it properly.