[Feature] Make three.js a first class citizen — Exposing API (Camera, Surface, Light, Measurement/Scale Metrics)

[D1no]

It is pretty impressive how far this project has come in such a short amount of time. However, it would be great to have one imperative way of constructing 3D scenes. Three.js has been around for quite a while and with projects such as react-native-webgl that mimic a webGL webview API but stream directly to the GraphicLayer ES API, it is possible to create incredible performant 3D experiences in RN.

The expo team as done a very simple and straight forward ARKit implementation by translating the phones AR-Position to the camera position within three js. See here: https://blog.expo.io/introducing-expo-ar-mobile-augmented-reality-with-javascript-powered-by-arkit-b0d5a02ff23

Try the snack here with the expo app for performance testing: https://snack.expo.io/Hk1C_YqjW (don't forget: you need an iPhone SE, IPhone 6S or higher)

While the declarative way of adding shapes and models is great, as soon we would wish to create more abstract scenes (animations, interactions, physics etc.) this project would face a lot of replication without gaining much more surface area.

Proposal

Provide an API/Container that can hold a three.js instance which scene is feed with

• Camera Position
• Surface(es) Position
• Light Source
• Measurement/Scale Metrics (i.e. 0.1 = 10 cm)

Opportinuty

Doing so would make scene creation much simpler (three.js has years worth of tutorials) while dropping sources of bugs in the library and making i.e. the integrating of ARCore and other solutions much simpler. Lastly, this allows developers to re-use the scene code (basically three.js) on the web and none AR enabled devices (fallback).

Performance is of course a challenge — but as the GL-GPU layer is directly tapped — I wanted to put this out here.

[macrozone]

yeah, expo's arkit demo is pretty cool and using threejs has surely some advantages.

react-native-arkit is based scenekit and arkit directly. So it is both an abstraction layer over scenekit (to specify 3d content) and over arkit (that controls the 3d camera and adds some api).

Threejs would be a replacement for scenekit and would require a rewrite. (It could also be possible that both lived side-by-side).

you would also want to remove the declarative bindings for objects that we have and instead just work with threejs directly.

I am not sure if threejs can be implemented seamlessly here, but i would love to see a PR showing a proof-of-concept!

[macrozone]

Some more thoughts:

Three.js could replace our scenekit integration. The cool thing about our implemenation is, that its declarative and in react-style.

You could map that to threejs, similarly how we do it. So an ARBox would create the corresponding THREE.Mesh and place it on the scene. This would make it easier to integrate android. You could just use ARCore and ARKit for controlling camera and detecting planes and use THREE.js to display the content.

The main advantage about threejs however is, that you could use threejs api directly and use existing code. But then, you would loose the decralative style of defining 3d content. Or you find a way, how both can co-exist.

we even could keep our existing api. So an ARBox might work identical (minus some scenekit-specific features and properties like lightModel)

shaders

looks pretty similar to scenekit: view-source:https://threejs.org/examples/webgl_shader2.html

We could define shaders on the material like we do. They have also multiple entry point, but the naming is a bit different.

What needs to be done

• [ ] replace scenekit with threejs
• [ ] write mapping code from our primitives (ARBox, ARSphere, ARModel, etc.) to threejs
  • [ ] create new mesh when a element is mounted , attach geometry and material and place it on the scene
  • [ ] update existing objects when properties change
  • [ ] remove objects from the scene, when its react-element is unmounted
• [ ] wire the camera with arkit (the expo team has done that) and arcore

and later:

• [ ] check how the user can work with threejs directly (alongside declarative api?)

[D1no]

I believe that a full translation of the declarative api to three js is probably a shoot over the top. Also, in animation - where time is a state - it proves pretty difficult to come up with great declarative ways (as https://github.com/toxicFork/react-three-renderer/blob/master/README.md proves to be).

Having a container element that allows to to take a three js scene and provide an instance for camera position and light source would already open up many possibilities. Especially if one considers to overlay SceneKit and three.js.

[birkir]

Sample repo with ARKit + THREE.js (without expo) https://github.com/birkir/react-native-webgl-arkit

[D1no]

@birkir I already stared it as I found it by chance. On the weekend I will play around with it – ps: you seem to be quite talented (!)

[hvardhan617]

Hi @birkir @macrozone Does the Expo and three js ARKIT work with ARCore supported android devices as well? or is it only restricted to apple phones