search

Shopify / react-native-skia

High-performance React Native Graphics using Skia
https://shopify.github.io/react-native-skia
MIT License
6.8k stars 436 forks source link

Not Compatible with Babylon React Native #1690

Closed nmongiya closed 1 year ago

nmongiya commented 1 year ago

Description

I am working on React Native Project where I use Babylon React Native (https://github.com/BabylonJS/BabylonReactNative) for 3d stuff and skia as well. Skia does not seem be compatible with that, I can use either of the library. I had this problem before but one of the skia version (0.1.157) was working fine with Rn 0.70.10 But since I have decided to update update the RN to 0.71 that skia version((0.1.157)) is not supported and updating the skia version brings the same problem that either skia or babylon can be used.

Version

0.1.197

Steps to reproduce

Create a New RN 0.71.0 project Add react-native-skia and react-native-babylon that comes with following dependencies "@babylonjs/core": "5.42.2", "@babylonjs/loaders": "5.42.2", "@babylonjs/materials": "5.42.2", "@babylonjs/react-native": "1.6.1", "@babylonjs/react-native-iosandroid-0-71": "1.6.1", "@babylonjs/react-native-windows-0-71": "1.6.1",

Snack, code example, screenshot, or link to a repository

<Group> <Group transform={ [{ scale: SampleScale }, { rotate: rotation }] }> <Path path={ SamplePath } color={ SampleColor } style={ style } strokeWidth={ strokeWidth }/> </Group> <Text text={ SampleText() } color={ textColor } font={ font } x={ SamplePosition.x } y={ SamplePosition.y } /> </Group>

and add some Babylon scene as well

chrfalch commented 1 year ago

Hi, @nmongiya - could you please provide some more information about errors etc?

nmongiya commented 1 year ago

Hi @chrfalch I am having this issue on Android only. NO issues on ios. Couple of interesting things:

When Metro is connected : Skia component is available but Babylon is not When Metro is NOT connected: Babylon is visible on screen but skia is not I thought it could be a race condition to get the canvas and I tried to load Babylon component after couple of seconds of timeout and both works fine in that case even when metro is connected, and here is what I can see in android logcat

I/SurfaceView: onWindowVisibilityChanged(0) true android.view.SurfaceView{f45f322 V.E...... ......I. 0,0-0,0} of ViewRootImpl@9e07a49[MainActivity] I/SurfaceView: onWindowVisibilityChanged(0) false android.view.SurfaceView{f2911b3 I.E...... ......I. 0,0-0,0} of ViewRootImpl@9e07a49[MainActivity] I/SurfaceControl: nativeRelease nativeObject s[526794639040] I/SurfaceControl: nativeRelease nativeObject e[526794639040] I/SurfaceControl: assignNativeObject: nativeObject = 0 Surface(name=null)/@0x8c0dc8b / android.view.SurfaceControl.readFromParcel:1117 android.view.IWindowSession$Stub$Proxy.relayout:1820 android.view.ViewRootImpl.relayoutWindow:9005 android.view.ViewRootImpl.performTraversals:3360 android.view.ViewRootImpl.doTraversal:2618 android.view.ViewRootImpl$TraversalRunnable.run:9971 android.view.Choreographer$CallbackRecord.run:1010 android.view.Choreographer.doCallbacks:809 android.view.Choreographer.doFrame:744 android.view.Choreographer$FrameDisplayEventReceiver.run:995 I/SurfaceControl: nativeRelease nativeObject s[526792490368] I/SurfaceControl: nativeRelease nativeObject e[526792490368] I/SurfaceControl: nativeRelease nativeObject s[526793921536] I/SurfaceControl: nativeRelease nativeObject e[526793921536] I/ViewRootImpl@9e07a49[MainActivity]: Relayout returned: old=(0,0,1920,1200) new=(0,0,1920,1200) req=(1920,1200)0 dur=11 res=0x1 s={true 529446535168} ch=false fn=194 I/SurfaceView: surfaceCreated 1 https://github.com/BabylonJS/BabylonReactNative/pull/8 android.view.SurfaceView{f45f322 V.E...... ......ID 0,0-1917,1002} I/SurfaceView: surfaceChanged (1917,1002) 1 https://github.com/BabylonJS/BabylonReactNative/pull/8 android.view.SurfaceView{f45f322 V.E...... ......ID 0,0-1917,1002} I/SurfaceView: applySurfaceTransforms: t = android.view.SurfaceControl$Transaction@b534370 surfaceControl = Surface(name=SurfaceView - com.emesent/com.emesent.MainActivity@f45f322@0)/@0x9a006e9 frame = 194 I/ReactNativeJS: BJS - [13:49:01]: Babylon Native (v5.42.2) launched

wcandillon commented 1 year ago

@nmongiya thanks for bringing this to our attention, I was as unaware of the this project as they were of ours. I reached out to discuss it offline.

nmongiya commented 1 year ago

Please find attached a screen recording illustrating how Skia and Babylon Native are fighting for (canvas?) resources. The foreground red square in the top left corner is the Skia component and the Babylon 3D scene is in the background. As you'll see in the video, when I don't interact with the Skia component, Babylon just works fine - I can rotate around the scene in Babylon and start the XR (camera video stream). However as soon as I tap the Skia component, the Babylon component freezes. I need to toggle the Babylon window to restore it. Another interesting thing is, when the Skia component takes control and I attempt to start the XR, it opens the camera video stream in the Skia component (instead of Babylon) until I reset it back by toggling the Babylon view.

https://github.com/Shopify/react-native-skia/assets/24381573/e9c441b6-c1b9-45fb-8e79-cbd81455a651

nmongiya commented 1 year ago

Please find the sample code here for the above mentioned video. RnSkiaBabylon.zip This is what my package.json looks like "dependencies": { "@babylonjs/core": "5.42.2", "@babylonjs/loaders": "5.42.2", "@babylonjs/react-native": "1.6.1", "@babylonjs/react-native-iosandroid-0-71": "1.6.1", "@babylonjs/react-native-windows-0-71": "1.6.1", "@react-native-community/slider": "^4.2.2", "@shopify/react-native-skia": "^0.1.197", "react": "18.2.0", "react-native": "0.71.7", "react-native-permissions": "^3.6.1", "react-native-windows": "0.71.7" }, "devDependencies": { "@babel/core": "^7.12.9", "@babel/preset-env": "^7.14.0", "@babel/runtime": "^7.12.5", "@react-native-community/eslint-config": "^3.0.0", "@tsconfig/react-native": "^2.0.2", "@types/jest": "^29.2.1", "@types/react": "^18.0.24", "@types/react-test-renderer": "^18.0.0", "babel-jest": "^29.2.1", "eslint": "^8.19.0", "jest": "^29.2.1", "metro-config": "^0.73.7", "metro-react-native-babel-preset": "0.73.5", "prettier": "^2.4.1", "react-test-renderer": "18.2.0", "typescript": "4.8.4" }

nmongiya commented 1 year ago

@chrfalch/ @wcandillon Let me know if you need more information on this.

chrfalch commented 1 year ago

Hi, thanks again for the input. This looks like a problem with Babylon using SurfaceView on Android. It is a view that is less suitable to combine with other views and uses a special form of rendering on Android built for performance - but not for interoperability with other views. RNSkia uses TextureViews on Android - both is based on OpenGL rendering.

What happens if you use multiple Babylon views at the same time?

chrfalch commented 1 year ago

Did you try the transparency trick mentioned in the issue you posted? (https://github.com/BabylonJS/BabylonReactNative/issues/585).

nmongiya commented 1 year ago

Thank @chrfalch for having a look, yes I have tried setting the transparency and setting the androidView as TextureView as well but issues remains the same.

chrfalch commented 1 year ago

I've tried to create a reproduction here, but it is a bit hard due to some issues setting up and getting a reliable Babylon.js example up and running.

In addition I've read the limitations described in the Babylon.js React Native docs here that says that only single views are supported - this makes me believe that the problem lies in the Babylon.js tool and not RN Skia?

I created another test to see if RN Skia plays nice with other OpenGL based libraries - installed Expo-gl and made a test that shows that we can display both Expo-GL views and RN Skia views at the same time without any issues. (repo can be found here: https://github.com/chrfalch/ExpoGLAndSkia)

I'll close this one for now since Babylon.js clearly states that it only supports a single view and RN Skia works well with other OpenGL views.

A test you can do is to see if Expo-GL works well with Babylon.js.