Possible to layer Panos?

[ghost]

Description

I'm trying to figure out whether it's possible to layer one Pano overtop of another. For example, if I have a base Equirectangular image of a kitchen, and I wanted to select a different color countertop, I could layer a second Pano of an Equirectangular png cutout of just the countertop, instead of having to stitch together an entirely different panoramic image for each possible scenario.

From what I've tried up to this point, it looks like the answer is no, but if someone could say so definitively or give an idea of how this might be accomplished another way, that would be greatly appreciated!

What I've tried so far:

Layering one Pano element over another (ReactVR seems to try to make it work, but result is glitchy)

export default class TestVR extends React.Component {
  render() {
    return (
      <View>
        <Pano source={asset('Kitchen.jpg')}>
          <Pano source={asset('Kitchen_Cabs.png')}/>
        </Pano>
      </View>
    );
  }
};

Layering as Image over Pano (won't ever line up properly bc png itself is an equirectangular image)

export default class TestVR extends React.Component {
  render() {
    return (
      <View>
        <Pano source={asset('Kitchen.jpg')} />
        <Image source={asset('Kitchen_Cabs.png')}
               style={{
                 width: xxx,
                 height: xxx,
                 layoutOrigin: [0.5, 0.5],
                 transform: [{translate: [0, 0, -xxx]}],
               }} />
      </View>
    );
  }
};

[mpochiro]

So I’m not sure if it would completely line up but there is an example that this guy used to put a picture of his resume “on” a computer screen, not pulled up on screen but it’s an image on the screen, why he would do that is beyond me but he did. Here is the source...

https://medium.com/@evansjwang/how-to-deploy-react-vr-to-heroku-6e8e9ba06339

Another possibility is to manipulate he image separately so you could just upload the manipulated image instead of trying to patch it.

[amitrajput1992]

@dludwick I've tried this a couple of months back. I was trying to keep 2 separate panos with transparent patches in front of each other. The issue is that the radius of the pano is hardcoded to a 1000m, which is why when you try the above code, the layer intersect with each other. I Implemented my own version of Pano as a custom component which accepts a radius param, then pass in say 1000m to the outer pano and 900m to the inner pano. This way you can layer up any number of panos.

[ghost]

@amitrajput1992 thank you! That sounds promising and makes a lot of sense. Could you point me to an example or give a hint on how to implement? I looked up the Pano component source code in the react-vr library, but can't seem to find where/how the radius is defined, other than the comment at the top which explicitly states the sphere will be 1000m.

[amitrajput1992]

I don't have a working example with me right now I know it works since I've implemented the same. If you see on line #84, where the SphereGeometry is being created, the first param passed is radius of the sphere. You can pass this param to the CustomView that you will have to write, as a prop from the native code and appropriately generate the geometry. See this link on how to create CustomViews https://facebook.github.io/react-vr/docs/native-modules.html#content

[ghost]

Perfect, thanks for the help! For anyone else who runs into this situation, line 84 refers to this

[ghost]

I think I'm pretty close, but can't test as my CustomView is complaining about rnctx being undefined. I've passed rnctx to custom modules before, but can't get it to work here...anyone have an idea?

[amitrajput1992]

rnctx is not passed to CustomViews. https://github.com/facebook/react-vr/blob/master/ReactVR/js/Modules/UIManager.js#L219

[ghost]

Right - but if the goal is to create what is basically a custom Pano component with the added ability to define a radius param, wouldn't I need to base my CustomView on the existing Pano view to keep the same functionality/behavior as a native Pano component? As shown in the link and the RCTPano class itself, rnctx is needed for defining _localResource, which is used for handling the source param input...

I guess my question at this point is, what are my options? Is there any way at all to give my CustomView access to the rnctx, or is my only option to remove the rnctx reliant code and hope for the best?

Thanks again for helping talk me through this.

[ghost]

Actually, I just went ahead and changed the source code to include rnctx, and everything works now! It's definitely a lot closer to a working solution, but still too glitchy to use in production.

[ghost]

I take that back - I wasn't fully updating the sphere geometry after setting the radius. Now I know for certain the sphere is being created with a radius that I set (< 1000), but the results are like picture #1 above, with black around the png Pano image and no blending at all when the camera moves. @amitrajput1992 do you remember changing anything other than the radius when you made this work in the past?

[amitrajput1992]

@dludwick Can you post the code for the customView and the related native code for this component. I'll have to see how it's being used

[ghost]

CustomView:

import * as ReactVR from 'react-vr-web';
import merge from 'react-vr-web/js/Utils/merge';
import StereoOffsetRepeats from 'react-vr-web/js/Utils/StereoOffsetRepeats';
import { HPanoBufferGeometry } from 'react-vr-web/js/Utils/HPano';
import { CubePanoBufferGeometry } from 'react-vr-web/js/Utils/CubePano';
import { RCTBindedResource } from 'react-vr-web/js/Utils/RCTBindedResource';
import * as OVRUI from 'ovrui';
import * as THREE from 'three';
import * as Yoga from 'react-vr-web/js/Utils/Yoga.bundle';

import Prefetch from './Prefetch';

const panoRayCast = (function() {
  // avoid create temp objects;
  const inverseMatrix = new THREE.Matrix4();
  const ray = new THREE.Ray();
  const sphere = new THREE.Sphere(new THREE.Vector3(0, 0, 0), 1000);
  const intersectionPoint = new THREE.Vector3();
  const intersectionPointWorld = new THREE.Vector3();
  return function(raycaster, intersects) {
    // transform the ray into the space of the sphere
    inverseMatrix.getInverse(this.matrixWorld);
    ray.copy(raycaster.ray).applyMatrix4(inverseMatrix);
    const intersect = ray.intersectSphere(sphere, intersectionPoint);
    if (intersect === null) {
      return;
    }

    // determine hit location in world space
    intersectionPointWorld.copy(intersectionPoint);
    intersectionPointWorld.applyMatrix4(this.matrixWorld);

    const distance = raycaster.ray.origin.distanceTo(intersectionPointWorld);
    if (distance < raycaster.near || distance > raycaster.far) {
      return;
    }

    intersects.push({
      distance: distance,
      point: intersectionPointWorld.clone(),
      object: this,
    });
  };
})();

let sphereGeometry = undefined;
let cubeGeometry = undefined;

// Only dispose certain textures, not those that is from video.
// A texture manager would be useful to help track the lifetime of textures.
const tryDisposeTexture = texture => {
  if (texture._needsDispose) {
    texture.dispose();
  } else if (Prefetch.isCachedTexture(texture)) {
    Prefetch.removeTextureFromCache(texture);
  }
};

export default class RCTPanoLayer extends ReactVR.RCTBaseView {
  /**
   * constructor: allocates the required resources and sets defaults
   */
  constructor(guiSys, rnctx) {
    super();

    // Set radius to default of 1000m
    this._radius = 1000;
    console.log(this._radius);

    sphereGeometry = sphereGeometry || new THREE.SphereGeometry(this._radius, 50, 50);
    cubeGeometry = cubeGeometry || new CubePanoBufferGeometry(2000, 3, 2, 1.01);

    this._tintOpacity = 1.0;
    this._styleOpacity = 1.0;
    this._sphereGeometry = sphereGeometry;
    this._cubeGeometry = cubeGeometry;
    this._material = new OVRUI.StereoBasicTextureMaterial({
      color: 'white',
      side: THREE.DoubleSide,
    });

    this._globe = new THREE.Mesh(this._sphereGeometry, this._material);
    this._globe.onBeforeRender = function(renderer, scene, camera, geometry, material, group) {
      if (camera.viewID === 1 && material.stereoOffsetRepeats[1]) {
        material.uniforms.stereoOffsetRepeat.value = material.stereoOffsetRepeats[1];
      } else {
        material.uniforms.stereoOffsetRepeat.value = material.stereoOffsetRepeats[0];
      }
      if (material._rightEnvMap) {
        if (camera.viewID === 1) {
          material.envMap = material._rightEnvMap;
        } else {
          material.envMap = material._leftEnvMap;
        }
        material.needsUpdate = true;
      }
    };

    this._globe.raycast = panoRayCast.bind(this._globe);
    this._globe.rotation.y = -Math.PI / 2;

    this.view = new OVRUI.UIView(guiSys);
    // set zOffset to be the radius of the sphere. This helps prevent the pano's
    // transparency from affecting other views as a result of rendering order.
    this.view.zOffset = this._radius;
    this.view.add(this._globe);
    this._localResource = new RCTBindedResource(rnctx.RCTResourceManager);
    this.globeOnUpdate = this.globeOnUpdate.bind(this);

    // register a setter for the radius so the globe can change size
    Object.defineProperty(this.props, 'radius', {
      set: value => {
        this._radius = value;
        this._sphereGeometry = new THREE.SphereGeometry(this._radius, 50, 50);
      },
    });
    Object.defineProperty(this.props, 'source', {
      set: value => this.setSource(value),
    });
    // register a setter for the backgroundColor so the globe can be tinted
    Object.defineProperty(this.style, 'opacity', {
      set: value => {
        this._styleOpacity = value;
        this._material.opacity = this._styleOpacity * this._tintOpacity;
        this._material.transparent = this._material.opacity < 1;
      },
    });
    // register a setter for the backgroundColor so the globe can be tinted
    Object.defineProperty(this.style, 'tintColor', {
      set: value => {
        const opacity = parseInt(value.toString(16).slice(0, 2), 16) / 255;
        this._material.color.set(value);
        this._tintOpacity = opacity;
        this._material.opacity = this._styleOpacity * this._tintOpacity;
        this._material.transparent = this._material.opacity < 1;
      },
    });
  }

  globeOnUpdate(scene, camera) {
    const projScreenMatrix = new THREE.Matrix4();
    const modelViewMatrix = new THREE.Matrix4();
    modelViewMatrix.multiplyMatrices(camera.matrixWorldInverse, this._globe.matrixWorld);
    projScreenMatrix.multiplyMatrices(camera.projectionMatrix, modelViewMatrix);
    this._globe.geometry.update(this.maxDepth, projScreenMatrix);
    this._globe.material = this._globe.geometry.material;
  }

  setSource(value) {
    if (value && value.tile) {
      // use tile renderer
      this._globe.geometry.dispose();
      this.maxDepth = value.maxDepth || 2;
      this._globe.geometry = new HPanoBufferGeometry(this._radius, this.maxDepth, value.tile);
      this._globe.onUpdate = this.globeOnUpdate;
    } else {
      // use sphere renderer
      this._globe.geometry.dispose();
      if (value && value.layout === 'CUBEMAP_32') {
        this._globe.geometry = this._cubeGeometry;
        this._globe.scale.z = -1;
        this._material.useUV = 1;
      } else {
        console.log(this._radius);
        console.log(this._sphereGeometry);
        this._globe.geometry = this._sphereGeometry;
        this._globe.scale.z = 1;
        this._material.useUV = 0;
      }
      this._globe.onUpdate = null;
      // call onLoadStart in React
      this.UIManager._rnctx.callFunction('RCTEventEmitter', 'receiveEvent', [
        this.getTag(),
        'topLoadStart',
        [],
      ]);
      const loadRemoteTexture = (url, onLoad, viewID) => {
        // When a url is null or undefined, send undefined to onLoad callback
        const onError = () => onLoad(undefined, viewID);
        const onLoadDisposable = texture => {
          texture._needsDispose = true;
          onLoad(texture, viewID);
        };
        // No progress indication for now.
        const onProgress = undefined;
        if (url == null) {
          onError();
        } else if (Prefetch.isCached(url)) {
          // First Check if the texture hasn't already been prefetched
          const cachedTexture = Prefetch.getFromCache(url);
          onLoad(cachedTexture, viewID);
        } else if (Array.isArray(url)) {
          const loader = new THREE.CubeTextureLoader();
          loader.setCrossOrigin('Access-Control-Allow-Origin');
          loader.load(url, onLoadDisposable, onProgress, onError);
        } else {
          const loader = new THREE.TextureLoader();
          loader.setCrossOrigin('Access-Control-Allow-Origin');
          loader.load(url, onLoadDisposable, onProgress, onError);
        }
      };
      const onLoadOrChange = (texture, viewID) => {
        // ignore a old request result
        if (value !== this._currentSource) {
          return;
        }
        this._globe.scale.x = -1;
        if (this._material.map) {
          tryDisposeTexture(this._material.map);
        }
        if (
          this._material.envMap &&
          this._material.envMap !== this._material._leftEnvMap &&
          this._material.envMap !== this._material._rightEnvMap
        ) {
          tryDisposeTexture(this._material.envMap);
        }
        if (texture === undefined) {
          this._material.map = undefined;
          this._material.envMap = undefined;
        } else if (texture.type === 'MonoTextureInfo') {
          this._material.map = texture.texture;
          this._material.envMap = undefined;
        } else {
          if (!value.enableMipmaps) {
            texture.generateMipmaps = false;
            texture.minFilter = THREE.LinearFilter;
          }
          texture.wrapS = THREE.ClampToEdgeWrapping;
          texture.wrapT = THREE.ClampToEdgeWrapping;
          const cubeTexture = texture.isCubeTexture ? texture : null;
          const flatTexture = texture.isCubeTexture ? null : texture;
          if (texture.isCubeTexture) {
            this._globe.scale.x = 1;
          }
          this._material.map = flatTexture;
          this._material.envMap = cubeTexture;
          if (viewID === 1) {
            this._nextRightEnvMap = cubeTexture;
          } else {
            this._nextLeftEnvMap = cubeTexture;
          }
        }
        const stereoFormat = value && value.stereo ? value.stereo : '2D';
        this._material.stereoOffsetRepeats = StereoOffsetRepeats[stereoFormat];
        if (!this._material.stereoOffsetRepeats) {
          console.warn(`Pano: stereo format '${stereoFormat}' not supported.`);
          // fallback to 2D
          this._material.stereoOffsetRepeats = StereoOffsetRepeats['2D'];
        }
        this._material.needsUpdate = true;

        this._numTexturesToLoad--;
        if (this._numTexturesToLoad === 0) {
          if (this._material._leftEnvMap) {
            tryDisposeTexture(this._material._leftEnvMap);
          }
          if (this._material._rightEnvMap) {
            tryDisposeTexture(this._material._rightEnvMap);
          }
          this._material._leftEnvMap = this._nextLeftEnvMap;
          this._material._rightEnvMap = this._nextRightEnvMap;
          // call onLoad in React
          if (texture !== undefined) {
            this.UIManager._rnctx.callFunction('RCTEventEmitter', 'receiveEvent', [
              this.getTag(),
              'topLoad',
              [],
            ]);
          }
          // call onLoadEvent in React
          this.UIManager._rnctx.callFunction('RCTEventEmitter', 'receiveEvent', [
            this.getTag(),
            'topLoadEnd',
            [],
          ]);
        }
      };

      this._currentSource = value;
      this._numTexturesToLoad = 1;
      this._nextLeftEnvMap = undefined;
      this._nextRightEnvMap = undefined;
      if (Array.isArray(value)) {
        if ((value.length !== 6 && value.length !== 12) || !value[0].uri) {
          console.warn(
            'Pano expected cubemap source in format [{uri: http..}, {uri: http..}, ... ]' +
              'with length of 6 (or 12 for stereo)'
          );
          return;
        }
        const urls = value.map(function(x) {
          return x.uri;
        });
        this._localResource.unregister();
        if (urls.length === 12) {
          this._numTexturesToLoad = 2;
          loadRemoteTexture(urls.slice(0, 6), onLoadOrChange, 0);
          loadRemoteTexture(urls.slice(6, 12), onLoadOrChange, 1);
        } else {
          loadRemoteTexture(urls, onLoadOrChange, 0);
        }
      } else {
        const url = value ? value.uri : null;
        if (this._localResource.isValidUrl(url)) {
          this._localResource.load(url, texture => onLoadOrChange(texture, 0));
        } else {
          this._localResource.unregister();
          loadRemoteTexture(url, onLoadOrChange, 0);
        }
      }
    }
  }

  presentLayout() {
    super.presentLayout();
    this._globe.visible = this.YGNode.getDisplay() !== Yoga.DISPLAY_NONE;
  }

  /**
   * Dispose of any associated resources
   */
  dispose() {
    if (this._material.map) {
      tryDisposeTexture(this._material.map);
    }
    if (
      this._material.envMap &&
      this._material.envMap !== this._material._leftEnvMap &&
      this._material.envMap !== this._material._rightEnvMap
    ) {
      tryDisposeTexture(this._material.envMap);
    }
    if (this._material._leftEnvMap) {
      tryDisposeTexture(this._material._leftEnvMap);
    }
    if (this._material._rightEnvMap) {
      tryDisposeTexture(this._material._rightEnvMap);
    }
    if (this._localResource) {
      this._localResource.dispose();
    }
    super.dispose();
  }

  /**
   * Describes the properties representable by this view type and merges
   * with super type
   */
  static describe() {
    return merge(super.describe(), {
      // declare the native props sent from react to runtime
      NativeProps: {
        source: 'string',
        radius: 'number',
      },
    });
  }
}

[ghost]

Native Component:

const PanoLayer = createReactClass({
  mixins: [NativeMethodsMixin],

  propTypes: {
    ...View.propTypes,
    style: StyleSheetPropType(LayoutAndTransformTintPropTypes),
    radius: PropTypes.number,

    /**
     * source image in the form of
     * `{uri: 'http'}` for a panorama
     * or
     * `[{uri: 'http..'}, {uri: 'http..'}, {uri: 'http..'},
     *   {uri: 'http..'}, {uri: 'http..'}, {uri: 'http..'}]` for a cubemap
     * or
     * `[{uri: 'http..'}, {uri: 'http..'}, {uri: 'http..'},
     *   {uri: 'http..'}, {uri: 'http..'}, {uri: 'http..'},
     *   {uri: 'http..'}, {uri: 'http..'}, {uri: 'http..'},
     *   {uri: 'http..'}, {uri: 'http..'}, {uri: 'http..'}]` for a stereo
     * cubemap where the first 6 images are the left eye cubemap and the
     * following 6 are the right eye cubemap.
     *
     * stereo(optional): the stereo format of a panorama: '2D' | 'TOP_BOTTOM_3D' |
     * 'BOTTOM_TOP_3D' | 'LEFT_RIGHT_3D' | 'RIGHT_LEFT_3D'
     *
     * If stereo is not a supported stereo format, it'll by default use '2D'
     */
    source: PropTypes.oneOfType([
      PropTypes.shape({
        uri: PropTypes.string,
        stereo: PropTypes.string,
      }),
      PropTypes.arrayOf(
        PropTypes.shape({
          uri: PropTypes.string,
        })
      ),
      PropTypes.shape({
        tile: PropTypes.string,
        maxDepth: PropTypes.number,
      }),
      // Opaque type returned by require('./image.jpg')
      PropTypes.number,
    ]),

    /**
     * Option onLoad callback called on success
     **/
    onLoad: PropTypes.func,

    /**
     * Option onLoadEnd callback called on success or failure
     **/
    onLoadEnd: PropTypes.func,
  },

  viewConfig: {
    uiViewClassName: 'PanoLayer',
    validAttributes: {
      ...ReactNativeViewAttributes.RCTView,
      radius: true,
    },
  },

  _onLoad: function() {
    this.props.onLoad && this.props.onLoad();
  },

  _onLoadEnd: function() {
    this.props.onLoadEnd && this.props.onLoadEnd();
  },

  getDefaultProps: function() {
    return {};
  },

  render: function() {
    const props = {...this.props} || {};
    props.style = props.style || {};
    if (!props.style.position) {
      props.style.position = 'absolute';
    }
    // default panos to being a render group
    if (!props.style.renderGroup) {
      props.style.renderGroup = true;
    }

    // Default pano radius to 1000m
    props.radius = props.radius || 1000;

    const source = resolveAssetSource(this.props.source);
    if (!source) {
      // If source is not defined, set uri to undefined and RCTPanoLayer will
      // handle the undefined uri
      props.source = {uri: undefined};
    } else {
      props.source = source;
    }

    return (
      <RKPanoLayer
        {...props}
        onLoad={this._onLoad}
        onLoadEnd={this._onLoadEnd}
        testID={this.props.testID}
        onStartShouldSetResponder={() => true}
        onResponderTerminationRequest={() => false}>
        {this.props.children}
      </RKPanoLayer>
    );
  },
});

[ghost]

Relevant index.vr.js:

export default class ClientVR extends React.Component {
  constructor() {
    super();
this.state = {
      scenePano: 'panos/Foster_Int_FamilyRoom_AmericanClassic.jpg',
    };
  }
 render() {
    return (
      <View>
        <Pano source={ asset(this.state.scenePano) }>
          <PanoLayer radius={900} source={ asset('layers/Foster_Int_FamilyRoom_americanClassic_Fireplace.png') } />
        </Pano>
        {this.state.renderVrTextbox && <MenuVr text={ vrMenuContent } />}
      </View>
    );
  }

[ghost]

@amitrajput1992 not sure if this is helpful, but the closer I make the inner sphere radius to 1000 (995-999), the more the images blend and that interference pattern shows up...which makes sense, as they start to intersect. But any radius 990 and under, the interference goes away and it's just black. As if the transparent portion of the pano img isn't being treated as transparent.

[amitrajput1992]

@dludwick 2 things that you need to do to get this working:

1. Set the zOffset of the geometry to the radius of sphere.
2. set transparent property on the material

[ghost]

@amitrajput1992 YES! That was it!! I can't thank you enough for all your time and help. People like you are what makes the dev community so great.

[Deboracgs]

I'm trying the same, but the tag Image doesn't appear on the screen

import React from 'react';
import {
  AppRegistry,
  StyleSheet,
  Text,
  View,
  Image,
  asset
} from 'react-360';

export default class proj16_pano extends React.Component {
  render() {
    return (
      <View style={{ flex: 1, width: 2048, height: 1024}}>
        {/* <View style={styles.greetingBox}>
          <Text style={styles.greeting}>
            Welcome to React 360
          </Text> */}
          <Image source={asset('floor-02-r.png')} style={{ flex: 1, width: 2048, height: 1024}} />
        {/* </View> */}
      </View>
    );
  }
};

// This file contains the boilerplate to execute your React app.
// If you want to modify your application's content, start in "index.js"

import {ReactInstance} from 'react-360-web';

function init(bundle, parent, options = {}) {
  const r360 = new ReactInstance(bundle, parent, {
    // Add custom options here
    fullScreen: true,
    ...options,
  });

  // Render your app content to the default cylinder surface
  r360.renderToSurface(
    r360.createRoot('proj16_pano', { /* initial props */ }),
    r360.getDefaultSurface()
  );

  // Load the initial environment
  r360.compositor.setBackground(r360.getAssetURL('enviroment.png'));
}

window.React360 = {init};

Can you help me @dludwick ?