React Native Expo 48 cameraWithTensors, TypeError: undefined is not a function at createProgramObjects

[mnai01]

System information

• Windows 10
• Pixel 4 API 30 Android 11, Pixel 3 API 30 Android 11

Describe the current behavior

On start the camera view will display as completely black and a undefined createProgramObjects error is displayed. I assume this is coming from \node_modules\@tensorflow\tfjs-react-native\src\camera\camera_webgl_util.ts as that is the only place I can find the createProgramObjects function, but do not know why its happening. If you comment out TensorCamera the app doesn't produce an error

Describe the expected behavior

This code should produce a simple front-camera view using tfjs-react-native

Standalone code to reproduce the issue

App.tsx

import React, { useEffect, useRef, useState } from 'react';
import { StyleSheet, Text, View } from 'react-native';
import * as posedetection from '@tensorflow-models/pose-detection';
import { Camera, CameraType } from 'expo-camera';
import { cameraWithTensors } from '@tensorflow/tfjs-react-native';
import * as tf from '@tensorflow/tfjs';
import { ExpoWebGLRenderingContext } from 'expo-gl';

const TensorCamera = cameraWithTensors(Camera);

export default function App() {
    const [tfReady, setTfReady] = useState(false);
    const [cameraType, setCameraType] = useState<CameraType>(CameraType.front);

    const handleCameraStream = (images: IterableIterator<tf.Tensor3D>, updatePreview: () => void, gl: ExpoWebGLRenderingContext) => {
        const loop = async () => {
            const nextImageTensor = images.next().value;
            requestAnimationFrame(loop);
        };
        loop();
    };

    useEffect(() => {
        (async () => {
            await Camera.requestCameraPermissionsAsync();
            await tf.ready();
            setTfReady(true);
        })();
    }, []);

    return (
        <View style={styles.container}>
            <Text>Camera Below</Text>
            {tfReady && (
                <TensorCamera
                    style={styles.camera}
                    type={cameraType}
                    useCustomShadersToResize={false}
                    cameraTextureHeight={1920}
                    cameraTextureWidth={1080}
                    resizeHeight={200}
                    resizeWidth={152}
                    resizeDepth={3}
                    autorender={true}
                    onReady={handleCameraStream}
                />
            )}
        </View>
    );
}

const styles = StyleSheet.create({
    container: {
        flex: 1,
        backgroundColor: '#fff',
        alignItems: 'center',
        justifyContent: 'center',
    },
    camera: { width: '50%', height: '50%', zIndex: 1 },
});

Package.json

{
    "name": "name",
    "version": "1.0.0",
    "main": "node_modules/expo/AppEntry.js",
    "scripts": {
        "start": "expo start",
        "android": "expo start -c --android",
        "ios": "expo start --ios",
        "web": "expo start --web"
    },
    "dependencies": {
        "@mediapipe/pose": "^0.5.1675469404",
        "@react-native-async-storage/async-storage": "^1.17.11",
        "@tensorflow-models/pose-detection": "^2.1.0",
        "@tensorflow/tfjs": "4.4.0",
        "@tensorflow/tfjs-react-native": "^0.8.0",
        "expo": "~48.0.10",
        "expo-camera": "~13.2.1",
        "expo-file-system": "~15.2.2",
        "expo-gl": "~12.4.0",
        "expo-gl-cpp": "~11.4.0",
        "expo-screen-orientation": "~5.1.1",
        "react": "18.2.0",
        "react-dom": "18.2.0",
        "react-native": "0.71.6",
        "react-native-fs": "~2.20.0",
        "react-native-gesture-handler": "~2.9.0",
        "react-native-svg": "13.4.0"
    },
    "devDependencies": {
        "@babel/core": "^7.21.4",
        "@expo/webpack-config": "^18.0.3",
        "@types/react": "~18.0.33",
        "@types/react-native": "~0.71.5",
        "babel-preset-expo": "~9.3.2",
        "jest-expo": "~48.0.2",
        "typescript": "~5.0.3"
    },
    "private": true
}

Other info / logs

 WARN  Possible Unhandled Promise Rejection (id: 0):
TypeError: undefined is not a function
TypeError: undefined is not a function
    at createProgramObjects (http://192.168.1.24:19000/node_modules%5Cexpo%5CAppEntry.bundle?platform=android&dev=true&hot=false:159776:35)
    at drawTextureProgram (http://192.168.1.24:19000/node_modules%5Cexpo%5CAppEntry.bundle?platform=android&dev=true&hot=false:159738:48)
    at drawTexture (http://192.168.1.24:19000/node_modules%5Cexpo%5CAppEntry.bundle?platform=android&dev=true&hot=false:159620:49)
    at renderToGLView (http://192.168.1.24:19000/node_modules%5Cexpo%5CAppEntry.bundle?platform=android&dev=true&hot=false:159526:40)
    at renderFunc (http://192.168.1.24:19000/node_modules%5Cexpo%5CAppEntry.bundle?platform=android&dev=true&hot=false:160106:40)
    at renderLoop (http://192.168.1.24:19000/node_modules%5Cexpo%5CAppEntry.bundle?platform=android&dev=true&hot=false:160057:30)
    at ?anon_0_ (http://192.168.1.24:19000/node_modules%5Cexpo%5CAppEntry.bundle?platform=android&dev=true&hot=false:160061:25)
    at next (native)
    at asyncGeneratorStep (http://192.168.1.24:19000/node_modules%5Cexpo%5CAppEntry.bundle?platform=android&dev=true&hot=false:4641:26)
    at _next (http://192.168.1.24:19000/node_modules%5Cexpo%5CAppEntry.bundle?platform=android&dev=true&hot=false:4660:29)
    at tryCallOne (/Users/kudo/01_Work/Repos/expo/expo/android/versioned-react-native/ReactAndroid/hermes-engine/.cxx/MinSizeRel/5a4n485h/x86/lib/InternalBytecode/InternalBytecode.js:53:16)
    at anonymous (/Users/kudo/01_Work/Repos/expo/expo/android/versioned-react-native/ReactAndroid/hermes-engine/.cxx/MinSizeRel/5a4n485h/x86/lib/InternalBytecode/InternalBytecode.js:139:27)
    at apply (native)
    at anonymous (http://192.168.1.24:19000/node_modules%5Cexpo%5CAppEntry.bundle?platform=android&dev=true&hot=false:29651:26)
    at _callTimer (http://192.168.1.24:19000/node_modules%5Cexpo%5CAppEntry.bundle?platform=android&dev=true&hot=false:29570:17)
    at _callReactNativeMicrotasksPass (http://192.168.1.24:19000/node_modules%5Cexpo%5CAppEntry.bundle?platform=android&dev=true&hot=false:29600:17)
    at callReactNativeMicrotasks (http://192.168.1.24:19000/node_modules%5Cexpo%5CAppEntry.bundle?platform=android&dev=true&hot=false:29763:44)
    at __callReactNativeMicrotasks (http://192.168.1.24:19000/node_modules%5Cexpo%5CAppEntry.bundle?platform=android&dev=true&hot=false:2793:46)
    at anonymous (http://192.168.1.24:19000/node_modules%5Cexpo%5CAppEntry.bundle?platform=android&dev=true&hot=false:2605:45)
    at __guard (http://192.168.1.24:19000/node_modules%5Cexpo%5CAppEntry.bundle?platform=android&dev=true&hot=false:2777:15)
    at flushedQueue (http://192.168.1.24:19000/node_modules%5Cexpo%5CAppEntry.bundle?platform=android&dev=true&hot=false:2604:21)

[julian-hecker]

Had this issue too. Was caused by Android Emulator not supporting WebGL2. You can update that in the emulator's advanced setting.

However, that was not the main issue for me as Expo Camera is not supported on emulators in the first place.

[gaikwadrahul8]

Hi, @mnai01

Aplogize for the delayed response and It seems like there is some package dependancy issue between React Nativeand Expo latest versions and if possible could you please try to downgrade the versions of React Native andExpo to previous stable version and see whether is it resolving your issue or not ? till then I'll try it from my end also and you can refer this comment It seems like concerned team is looking into this issue and we will update you soon. Thank you!

[mnai01]

Hey @gaikwadrahul8 thanks for getting back to me. I have tried messing with the deps and have had no luck yet.

What worked for me was adding these extra dependencies that I've seen other TS project use.

    - "@tensorflow/tfjs-backend-webgpu": "^0.0.1-alpha.19",
    - "react-native-unimodules": "^0.14.10"

and running npm i --legacy-peer-deps idk if it's the --legacy-peer-deps or the 2 extra modules or both that fixed the problem but I can test it out and check later.

It does have many conflicting peer dependency problems though and it performs very poorly. In terms of model performance and FPS it's very slow and I am hoping that is just from the weird dependency situation and not the actual code

Edit: Another big thing is from @julian-hecker (thank you). expo-camera doesn't support emulators per the docs so that was a big solution to the createProgramObjects problem to just use a device, but then it led to trying to get TS to work with everything. Once that was up now it's the performance problems.

[gaikwadrahul8]

Hi, @mnai01

Thank you for your detailed analysis and I really appreciate your efforts and time you invested in this issue and as I mentioned in the previous comment our concerned team is working on this issue and they'll look into your analysis which you have done, we'll update you soon. Thank you!