Integrate with other WebGL engines

pjcozzi commented 11 years ago

There's been some interest in integrating Cesium with other WebGL engines.

On the Cesium side, this should be pretty straightforward. The changes will require Cesium to only own the context during a call to Scene.render, not forever. This is already mostly there since objects only access the context in update functions called from within Scene.render, but we also keep some state between frames that we'll need to change:

Render State #638
Clear states (if I don't remove them sooner)

Also, when we do the post processing framework, we need to be mindful of when we are rendering to texture vs. the main framebuffer.

m-schuetz commented 9 years ago

Hi,

I've done a first test with cesium + three.js here: http://potree.org/demo/experimental/2015.08.19_cesium_test/examples/cesium_test.html

The basic idea is this:

Copy "cesium.camera.viewMatrix" and "cesium.camera.inverseViewMatrix" to "three.camera.matrixWorldInverse" and "three.camera.matrixWorld"
The potree point cloud is given in EPSG:21781(swiss). This is first transformed to WGS84(degrees) and from that to Cesium.Cartesian3.fromDegrees().
pointcloud.position is set to the Cartesian3 value of the bounding box minimum
pointcloud.lookAt(bottomLeftHigh) is called to align the point cloud to align the point cloud on the ground
potree.pointcloud.up.copy(latDir) is used to rotate the point cloud to align with longitude and latitude

Issues:

This example is specific to a point cloud in the EPSG:21781 coordinate system
three.js is rendered in its own canvas over the cesium canvas. Idealy, both rendering engines should use the same context and render into the same buffers, if possible.
Precision issues with the three.js camera, especially noticeable when the camera is moving close to the point cloud. Possibly because three.js uses floats? Does cesium use double precision values for transformations?
Only works if the aspect ratio is 1, therefore the fixed canvas size.

pjcozzi commented 9 years ago

Does cesium use double precision values for transformations?

Yes, see the link I sent on twitter: ~~http://blogs.agi.com/insight3d/index.php/2008/09/03/precisions-precisions/~~ EDIT: Link moved here: http://help.agi.com/AGIComponents/html/BlogPrecisionsPrecisions.htm

There will also be depth buffer compositing and perhaps precision issues with this approach.

m-schuetz commented 9 years ago

Thanks, I'll take a look at it!

m-schuetz commented 9 years ago

Found out how to render into the same canvas with the same context:

First, create a three.js renderer and pass the canvas as argument. The canvas/context was already initialized by cesium. Three.js will call getContext() for the same canvas and therefore, according to the spec, receive the same context as cesium.

three.renderer = new THREE.WebGLRenderer({alpha: true, canvas: cesium.viewer.canvas});

In the render loop, cesium has to be rendered first.

During the rendering with three.js, WebGLRenderer._currentProgram has to be set to null before using the three.js render function. Since _currentProgram is a private variable, this can only be done through WebGLRenderer.resetGLState(). resetGLState(), however, also calls state.reset which causes errors. This workaround therefore temporarily changes the state.reset function to an empty function

var resetFunc = three.renderer.state.reset;
three.renderer.state.reset = function(){};
three.renderer.resetGLState();
three.renderer.state.reset = resetFunc;

Now it's probably not that important for most cases but it would still be interesting to use the same depth buffer in threejs as well. Do you think that will be possible with the way cesium handles depth buffers? At https://cesiumjs.org/massiveworlds/ there are slides about rendering with multiple frustums in which case reusing the depth buffer is probably out of question.

pjcozzi commented 9 years ago

Thanks for the info.

Now it's probably not that important for most cases but it would still be interesting to use the same depth buffer in threejs as well.

I expect most apps will want this, e.g., depth testing points vs. Cesium's terrain. I expect this can be done by setting Three.js' camera near and far planes and explicitly rendering the Three.js scene for each frustum in Cesium. See executeCommands in Scene.js.

The jittering issue in Three.js we discussed on twitter would also need to be addressed.

brucefuqiming commented 8 years ago

We're doing a project to mix Cesium and Three, like potree. Cesium is rendered firstly,then Three's elements rendered on the Cesium's canvas.But the problem is Three's elements always block Cesium's. e.g.: there is a geometry created by Three in front of the Cesium‘s earth,it can block the Cesium‘s earth and other elements correctly, but when the scene is rotated to the backward, Three's object still block the Cesium,maybe depth buffer is the main reason,but we cannot find the solution to solve. Do you have any methods?We wanna make Cesium the part of the Three’s scene,like an ordinary geometry. Is it possible?

pjcozzi commented 8 years ago

@brucefuqiming this is because Cesium and Three have two different depth buffers and Three is being rendered second. To merge the scenes correctly, this will require core engine support in Cesium as described in this issue.

pjcozzi commented 7 years ago

For a prototype of integrating Cesium and three.js, see Section 6.2 here: https://www.cg.tuwien.ac.at/research/publications/2015/Adorjan-2015/Adorjan-2015-thesis.pdf

m-schuetz commented 7 years ago

One possibility could be to render color and depth buffers in one lib and then use the depth buffer(s) in the other lib to compose the renderings of both.

e.g.:

User renders his stuff in three.js and provides color buffer, depth buffer, and projection parameters to cesium.
Cesium does a screen pass (or one for each frustum) with the three.js color and depth map as input and discards all fragments that do not pass a custom depth test. The custom depth test would consist of transforming the three.js depth values to cesium depth values.

Vice-versa, cesium could provide a color buffer and all of its depth maps and someone can use this to do a custom compose step to merge the cesium renderings with his three.js or other renderings.