Support for Expo and Web

[AyushExel]

Area Select

react-native-pytorch-core (core package)

Description

First of all, I want to say that this is one of the best projects I've come across during the holidays. This exactly fits my use case of rapid ML application prototyping. can't wait for the stable release! :)

Unfortunately, I'm running a mac m1 so I faced some complications in the automated cli setup. I tried to hack together a solution with expo cli and it worked for Android, although now I can't remember what I did exactly :( This brings me to my questions/ feature requests.

1. Will the expo cli be officially supported? It'll probably ease the setup process. I don't remember how, but I was able to get it working in expo cli by installing it via expo install react-native-pytorch-core and ran the app using yarn android and it worked! I did some troubleshooting by googling the errors but I don't remember it now. Currently, when I try to build the app after importing and using say Camera module from PyTorch core, I get this error:

Earlier I was able to solve it. I know that expo isn't officially supported so this isn't really a bug but It'd be great if it could be supported.

2. Will the pytorch react native core components work with Web? When I was able to run the camera module on the android emulator successfully, I tried to bundle the app for web. But I was getting errors like "export 'requireNativeComponent' was not found in 'react-native-web/dist/index' and the app didn't load on web. So, I was wondering if this library is only meant for mobile app development or is it intended to be the pytorch port for react-native that supports all platforms. TLDR: Can I use react-native-pytorch-core components on web and mobile alike?

[raedle]

Hi @AyushExel,

Thanks for leaving the encouraging comment! It is one of our goals to make PyTorch Live the best-in-class ML tool for rapid prototyping, which is why it is important to get feedback from the community (like your feedback) to help us achieve this goal!

There are some issues with M1 chipsets with Android, React Native, Python, and PyTorch. We have solved a few already and will be solving issues as they are reported.

1. Thanks for the callout on Expo. Expo is on our radar, so it's great to hear that there is interest there. Please let us know how to reproduce the issue that you are facing, so we can have a closer look at it or submit a PR if you find fix ;)

2. PyTorch Live uses PyTorch Mobile to run on-device inference. Unfortunately, PyTorch (Mobile) doesn't have a JavaScript runtime (or WASM runtime) to load TorchScript models in the browser and run inference on them. There are a few paths to make this probably work (e.g., expose libTorch via WASM or run via local Flask/Python server), but that's currently out of the scope for PyTorch Live project.

I'll keep this issue open until we either fixed the Expo issue or identified it as a #wontfix!

Thanks again, and don't hesitate to reach out!

[AyushExel]

@raedle thanks for the info. for the expo reproduction, here are the steps:

• Initialize a project using expo init <project_name>
• Install react-native-pytorch-core expo install react-native-pytorch-core
• Use any module from pytorch core in App.js. e.g, the ` module
• Running this app on emulator or even the physical device throws this error, but without pytorch-core components the app runs fine

Ohh and btw, I was able to get it working with react-native manual installation, so expo support is not a blocker for me right now. I'm able to experiment without it. I'd be happy to write the steps for setting it up on m1 mac if needed along with the common troubles users might face

[raedle]

Thanks, @AyushExel! I could confirm the "Invariant Violation" error on my end with the blank Expo template.

FWIW, ejecting the Expo app (expo eject) and running the React Native app with yarn android works.

@byCedric, do you have an idea? :)

[chrisklaiber]

Hi @AyushExel - I wanted to circle back. There have been many updates and major API changes in our 0.2.0 series. One specific change with the just-released 0.2.1 version is a working Expo config plugin to help with configuring a new Expo project. See my comment here, I hope it helps: https://github.com/facebookresearch/playtorch/issues/73#issuecomment-1217099999

[AyushExel]

@chrisklaiber hey thanks for the update. I kinda stopped playing with pytorch after this attempt. But the new release looks promising! I'll start exploring it again! btw I see you guys have added support for YOLOv5. I'm one of the maintainers of the repo, feel free to tag me if there are any issues related to yolov5. Thanks!

[chrisklaiber]

Thanks for the followup, and thanks for your work on YOLOv5! I'll close this issue out for now, since it appears there is nothing specific left to address. Also, happily, there is now a YOLOv5 tutorial available: https://playtorch.dev/docs/tutorials/snacks/yolov5/

[AyushExel]

@chrisklaiber hey! I decided to give it another shot with expo. I upgraded to the latest version of react-native-pytorch-core. Here is my package.json

  "dependencies": {
    "expo": "~47.0.6",
    "expo-status-bar": "~1.4.2",
    "react": "18.1.0",
    "react-dom": "18.1.0",
    "react-native": "0.70.5",
    "react-native-web": "~0.18.9",
    "react-native-pytorch-core": "0.2.2",
    "react-native-safe-area-context": "3.3.2"
  },

And App.ts

import { View } from 'react-native';
import {Camera} from 'react-native-pytorch-core';
export default function App() {

  return (
      <View>
        <Camera />
      </View>
    );

On doing yarn android via expo, I get this error:

 ERROR  Error: PlayTorchJSIModule not found. Maybe try rebuilding the app.
 ERROR  Invariant Violation: Failed to call into JavaScript module method AppRegistry.runApplication(). Module has not been registered as callable. Registered callable JavaScript modules (n = 11): Systrace, JSTimers, HeapCapture, SamplingProfiler, RCTLog, RCTDeviceEventEmitter, RCTNativeAppEventEmitter, GlobalPerformanceLogger, JSDevSupportModule, HMRClient, RCTEventEmitter.
        A frequent cause of the error is that the application entry file path is incorrect. This can also happen when the JS bundle is corrupt or there is an early initialization error when loading React Native.
 ERROR  Invariant Violation: Failed to call into JavaScript module method AppRegistry.runApplication(). Module has not been registered as callable. Registered callable JavaScript modules (n = 11): Systrace, JSTimers, HeapCapture, SamplingProfiler, RCTLog, RCTDeviceEventEmitter, RCTNativeAppEventEmitter, GlobalPerformanceLogger, JSDevSupportModule, HMRClient, RCTEventEmitter.
        A frequent cause of the error is that the application entry file path is incorrect. This can also happen when the JS bundle is corrupt or there is an early initialization error when loading React Native.

[raedle]

Hey @AyushExel, there are a few notable changes since the launch from last year. There are three options now. I listed them below in the order of increased involvement (or my subjective rating of difficulty).

1. Use PlayTorch app

no CLI commands required

Use the PlayTorch app available in Play Store and App Store and build ML demos on the Expo Snack platform (this is what the YOLOv5 tutorial describes step by step: https://playtorch.dev/docs/next/tutorials/snacks/yolov5/).

2. Create Standalone App with Expo

requires CLI commands

1. Create Expo app with npx create-expo-app my-app
2. Navigate to app directory cd my-app
3. Install PlayTorch SDK with npx expo install react-native-pytorch-core (it needs to use npx expo install because it makes sure to run the PlayTorch config plugin in the next step; i.e., yarn add react-native-pytorch-core or npm install react-native-pytorch-core will result in an error similar to what you reported)
4. Change App.js (see code below)

import { StatusBar } from 'expo-status-bar';
import { StyleSheet, Text, View } from 'react-native';
import { Camera } from 'react-native-pytorch-core';

export default function App() {
  return (
    <View style={styles.container}>
      <Camera style={StyleSheet.absoluteFill} />
    </View>
  );
}

const styles = StyleSheet.create({
  container: {
    flex: 1,
    backgroundColor: '#fff',
    alignItems: 'center',
    justifyContent: 'center',
  },
});

Note: style={StyleSheet.absoluteFill} makes sure the camera is full width/height (otherwise it might end up 0,0 width/height.

5. If .watchmanconfig is missing in the app directory, run echo "{}" > .watchmanconfig (otherwise the next command will err with Error: watchman::CommandValidationError:)
6. For iOS, we just realized that it is now necessary to add NSCameraUsageDescription to the Info.plist. Add the following to the ios/myapp/Info.plist:

<key>NSCameraUsageDescription</key>
<string>Allow $(PRODUCT_NAME) to use the camera as input for machine learning models.</string>

7. Run app on Android with npx expo run:android or on iOS with npx expo run:ios (note: use the commands provided here to build the native app from scratch, otherwise Expo will use the Expo Go app, which does not ship with the PlayTorch SDK native code)

3. Add PlayTorch SDK to Existing RN App

Instructions for this are on the website in the Add Package to Existing App tutorial

[AyushExel]

@raedle thanks! I was probably doing yarn add. Your suggestion fixed the problem!