Dynamic surfaces

[dijana-sagit]

Hi, my question is related to the surface rendering (particularly flat). According to the example MultiRoot, you render all the surfaces from the client.js. Once rendered, is there a way to update the location of the surface from the code? Or, is there a way to create and render a surface on the go from the code? I'm looking into implementing a 2D popup when I click on a VrButton. Thanks!

[andrewimm]

Yes, the same setAngle methods you use to initially position your flat surface can be called anytime to reposition them. The same is true for resizing or re-shaping a surface. This needs to be handled from the client.js side, but you can easily implement your behavior with a Native Module that either creates a surface on demand, or (recommended) previously creates a surface with no contents, and repositions it when you need it.

As a team, we are working on developing an official API for repositioning or generating surfaces from React, but we want to make sure that it is flexible enough to support all potential use cases we have in mind – we don't want to change APIs and break code later. In the meantime, we recommend the path of creating your own native module methods.

[flyandi]

how do u access the actual Instance from a NativeModule? All I am getting is the context that doesn't seem to expose access to the actual r360 instance.

[andrewimm]

You can't send the value through the native module – you can only transfer primitives and objects that follow the structured clone algorithm. What you can do is expose a method to React that updates a surface instance on the client side. Building a specific module for your application is easier; building a generalized API that supports every possible case today, and doesn't prevent future behavior tomorrow, is going to be a little trickier.

As we look forward at possible APIs, we are considering the ability to register each surface with a unique identifier. This will be used to reference it from the React side – a common pattern in React Native setups like this. From there, you could trigger a variety of reshaping and repositioning effects, as long as they were exposed through the module.

const {SurfaceModule} = NativeModules;
SurfaceModule.reposition('mysurface', {angleHoriz: -0.5, angleVert: 0.3});

[flyandi]

So if I am understanding this right, all surfaces are created in client.js only during initialization and although we can modify a surfaces behavior, there is no way of creating dynamic surfaces at this point since that would require re-running client.js.. correct?

[andrewimm]

No, surfaces can be created anytime. If you look at the the code in the MultiRoot sample, those surfaces are actually being created after React has initialized. At any point in app initialization, a surface can be generated and used with renderToSurface() to create a new React root.

While it's possible to generate Surfaces on the fly, I do recommend you pre-allocate and reuse them in situations where you know the number of surfaces ahead of time. For instance, to show a display every time a button is clicked, you could send a message via Native Module to generate a new popup each time – but that's wasteful of resources. It'd be better to preallocate a popup surface, and then reposition and reshape it when needed. Surfaces are cheap to create today, but we have some optimizations we're looking at in the near future that would dramatically increase the visual quality of surface content, similar to how the Gear VR re-release in early 2017 effectively doubled the pixel density with only a software update. These optimizations, while great for app quality, would make it more expensive to generate surfaces.

I also want to highlight a new Sample I just published that shows surfaces being smoothly repositioned each frame: https://github.com/facebook/react-360/tree/master/Samples/HeadlockedSurfaces

[flyandi]

Yeah I think the pre-allocation seems to be the best strategy if you want to place dynamic content. Now from your experience, what would you say is a max amount of surfaces you could run at any given time?

For example, I want to add hotspot points in an overlay of an video that show additional information in a popup or change the scene. Some scenes could contain more than 5 hotspots. You would need to create 5 different surfaces at least to have the hotspots located in the different spatial coordinates.

I will end up with a lot of pre-allocated surfaces.. or is there an easier way to do that, like a master surface that spawns the entire environment?

[andrewimm]

I haven't stress-tested, but I don't think there's a significant variable cost to adding more surfaces. If you have 5, I wouldn't worry about perf. I've done some hotspot demos that had 12 running on the screen with no measurable issues.

If you're going to end up with a lot, you may want to build a Native Module that implements a resource pool for Surfaces. I can describe below vaguely how I'd implement it, but it'd probably be helpful to get a demo of this in the Samples dir.

It comes down to three pieces: a method to request a surface, a method that tells client.js to render a specific root to a surface, and a method to reposition a surface.

From React, those might look vaguely like this:

const {SurfaceModule} = NativeModules;

SurfaceModule.requestSurface().then(handle => {
  // handle is a unique string identifier for referencing a surface
  // Resize to 500x300px
  SurfaceModule.resize(handle, 500, 300);
  // Tell client.js to instantiate a new instance of <ReactRootName> on the new surface
  SurfaceModule.renderToSurface('ReactRootName', { /* initial props */ }, handle);

  // Later on, free up the surface again
  SurfaceModule.unmountComponentAtSurface(handle);
});

On the client.js side, you'd implement a pool of declared-but-unused surfaces. When a surface is requested, you either pull one out of the pool, or create a new one. There's still some kinks in the model I've outlined above, like whether a surface gets released when you unmount its contents, or whether it needs to be explicitly released – that's why these APIs take some time to evolve. Also on that note, I don't think we've actually exposed the functionality to unmount surface content from client.js, but I'll get on that today.

I'm looking to build at least some of these as experimental APIs in the next week. My team has had a lot of discussions about this space, since it's also relevant to our internal Oculus use cases.

[flyandi]

Absolutely that is what I would have gone for. Ok so any NativeModule automatically extends the renderToSurface methods?

[andrewimm]

I think there might still be a disconnect here. A native module is simply a way to trigger something in client.js from within React, they do not automatically contain anything. The method names are arbitrary, I have just chosen to copy the API because it creates a sense of familiarity.

More on native modules, including a sample: https://facebook.github.io/react-360/docs/native-modules.html

[devsatish]

@andrewimm Hello Andrew; this is regarding referencing Surface in index.js. In my example; I am creating surfaces and a native module - "SurfaceModule". All I am trying to do is a reposition a surface , by getting a reference to the surface by the name; it doesnt seem to work; What's the api to reference a surface by name ?

//client.js
import {ReactInstance, Surface} from 'react-360-web';
import {Module} from 'react-360-web';
function init(bundle, parent, options = {}) {
  const r360 = new ReactInstance(bundle, parent, {
    fullScreen: true,
    nativeModules: [
      new SurfaceModule(),
    ],
    ...options,
  });

  const s1 = new Surface(256, 256, Surface.SurfaceShape.Flat);
  s1.setAngle(0,1.5707963267949); //pi/4
  r360.renderToSurface(
    r360.createRoot('MultiPanel', { lbl:'Cam1',}),
    s1
  );
  r360.compositor.setBackground(r360.getAssetURL('360_world.jpg'));
}

class SurfaceModule extends Module {
  constructor() {
        console.log('constructor - SurfaceModule');
        super('surfaceModule');
  }
  triggerMe(surfaceName, horizontalAngle, verticalAngle) {
    s1.setAngle(horizontalAngle,verticalAngle);
  }
}
window.React360 = {init};

//index.js
import React from 'react';
import {AppRegistry,
  StyleSheet,
  Text,
  View,
  VrButton,
  Image,
  asset,
  uri,
  Environment,
} from 'react-360';
import {
  NativeModules,
} from 'react-360';
const surfaceModule = NativeModules.SurfaceModule;

export default class MultiPanel extends React.Component {

 constructor(props) {
    super();
  }

  doClick = () => {
   console.log(surfaceModule);
    surfaceModule.trigger('s1', 1.5708, 0.7854);
  }

  render() {
    return (
         <View style={{flexDirection: 'column', height: 100, padding: 20}}>
         <Text style={{textAlign: 'center', color:'pink'}}>{this.props.lbl}</Text>
          <VrButton onClick={this.doClick}>
              <Image style={{
                  height: 128,
                  width: 128,
                }}  source={{uri: './static_assets/hotspot.png'}}/>
        </VrButton>
      </View>

    );
  }
};
AppRegistry.registerComponent('MultiPanel', () => MultiPanel);

[balintlaczko]

Dear all, I would like to do this: when clicked on a VrButton in index.js -> setangle of the Surface on which the VrButton is rendered in client.js. I understand that I probably have to set up an event listener in the client and fire it in the index (right...?), but I am a beginner on this and got stuck with this - which feels to be an easy thing to achieve I guess if you know what you're doing, so it would be so nice if some of you could throw me an example how to do this! Would be eternally grateful!! :)