Embed widgets with jupyter-cadquery (threejs): wrong position on load

[Peque]

I am using jupyter-cadquery to visualize some 3D models made with CadQuery.

When visualizing the models on a Jupyter notebook, everything works as expected.

But when trying to embed the widget in an HTML document, it seems the camera, on load, is pointing to (0, 0, 0), not as expected. Once you interact with the widget, the camera will point to the expected coordinate.

Here is the code to reproduce the error and an animation of the mentioned problem (see instructions bellow on how to reproduce it with Binder):

from cadquery import Workplane
from ipywidgets import embed
from jupyter_cadquery.cad_view import CadqueryView
from jupyter_cadquery.cadquery import Assembly
from jupyter_cadquery.cadquery import Part

# Create a simple assembly
box1 = Workplane('XY').box(10, 10, 10).translate((0, 0, 5))
a1 = Assembly([Part(box1)], "example 1")

# Generate HTML
a1.collect_shapes()
view = CadqueryView()
for shape in a1.collect_shapes():
    view.add_shape(shape["name"], shape["shape"], shape["color"])
renderer = view.render()
embed.embed_minimal_html('export.html', views=renderer, title='Renderer')

renderer

Note how the view of the cube "jumps" suddenly on interaction.

Could it be an issue with ipywidgets? Since the view is okay when displayed in the notebook.

How to reproduce

You can reproduce it with Binder, without needing to create a local environment (admitedly, installing CadQuery/jupyter-cadquery is not the easiest/fastest thing to do):

https://mybinder.org/v2/gh/bernhard-42/jupyter-cadquery/master?urlpath=lab&filepath=examples%2Fcadquery.ipynb

Just execute the code above in a new empty notebook. See how the renderer shows the 3D model without any issues on the notebook:

After execution, an export.html document will also appear in the file list on the left. Open it and make sure to click on the "Trust HTML" button on top of the viewer and hit refresh. If you interact with the view, you can reproduce the issue.

[Peque]

Also asked in Stack Overflow, with a bounty, in case someone wants to reply there and get the bounty. :innocent:

[jasongrout]

Also there is a "wild guess" about what might be wrong at https://github.com/bernhard-42/jupyter-cadquery/issues/5#issuecomment-568199224

[Peque]

That wild guess may fix the camera target offset.

Note that, also, the perspective is lost (the view in the exported HTML is not an orthogonal view). So maybe something else is happening? (i.e.: camera settings, both the perspective and the target may be reset/lost)

[Peque]

Just to avoid having multiple conversations in different issues, I will add the relevant information from https://github.com/bernhard-42/jupyter-cadquery/issues/5.

@bernhard-42's "wild guess":

Maybe the embeddingjavascript code does not callupdate()` at the beginning but when you interact it will - just a wild guess (it will be hidden somewhere in https://unpkg.com/@jupyter-widgets/html-manager@^0.18.0/dist/embed-amd.js)

@jasongrout's pointer to the source code:

Relevant source is in https://github.com/jupyter-widgets/ipywidgets/tree/master/packages/html-manager/src

[bernhard-42]

I looked at the orthogonal issue and copied the example from https://github.com/jupyter-widgets/pythreejs/blob/master/examples/Combined%20Camera.ipynb and slightly changed it:

from pythreejs import *
from IPython.display import display

mesh1 = Mesh(SphereBufferGeometry(10, 8, 8), MeshLambertMaterial(color='red', opacity=0.5, transparent=True), position=[-20, 0, 0])
mesh2 = Mesh(BoxBufferGeometry(20, 16, 16), MeshLambertMaterial(color='green', opacity=0.5, transparent=True), position=[20, 0, 0])

view_width = 600
view_height = 400
camera = CombinedCamera(position=[0, 0, 60], width=view_width, height=view_height)

key_light = PointLight(position=[-100, 100, 100])
ambient_light = AmbientLight(intensity=0.4)
scene = Scene(children=[mesh1, mesh2, key_light, ambient_light, camera])
renderer = Renderer(scene=scene, camera=camera, controls=[OrbitControls(controlling=camera)],
                    width=view_width, height=view_height)
camera.mode = 'orthographic'
embed.embed_minimal_html('export.html', views=renderer, title='Renderer')
display(renderer)

It produces an orthographic view in Jupyter but a perspective view in the export.html. How can I export an orthographic view from a ˋCombinedCameraˋ?

[Peque]

@jasongrout It seems @bernhard-42 was able to find a reproducible case that does not require jupyter-cadquery. Just pythreejs. :blush:

[jasongrout]

Thanks! CC @vidartf as well, who maintains pythreejs these days.

[Peque]

@jasongrout Thanks to @bernhard-42's code I was able to create a similar scenario with only pythreejs to the one I provided above with jupyter-cadquery. It reproduces both the perspective problem and the initial camera look-at problem:

from ipywidgets import embed
from pythreejs import *
from IPython.display import display

base = Mesh(
    BoxBufferGeometry(20, 0.1, 20), 
    MeshLambertMaterial(color='green', opacity=0.5, transparent=True),
    position=(0, 0, 0),
)
mesh = Mesh(
    BoxBufferGeometry(10, 10, 10), 
    MeshLambertMaterial(color='green', opacity=0.5, transparent=False),
    position=(0, 5, 0),
)
target = (0, 5, 0)

view_width = 600
view_height = 400
camera = CombinedCamera(position=[60, 60, 60], width=view_width, height=view_height)

camera.mode = 'orthographic'

lights = [
    PointLight(position=[100, 0, 0], color="#ffffff"),
    PointLight(position=[0, 100, 0], color="#bbbbbb"),
    PointLight(position=[0, 0, 100], color="#888888"),
    AmbientLight(intensity=0.2),
]
orbit = OrbitControls(controlling=camera, target=target)
camera.lookAt(target)

scene = Scene(children=[base, mesh, camera] + lights)
renderer = Renderer(scene=scene, camera=camera, controls=[orbit],
                    width=view_width, height=view_height)

camera.zoom = 4
embed.embed_minimal_html('export.html', views=renderer, title='Renderer')
display(renderer)

The result looks good in the notebook:

But when opening the export.html file:

[Peque]

Interestingly/surprisingly, if I move the lookAt() call to just bellow the camera.mode setting:

camera = CombinedCamera(...)

camera.mode = 'orthographic'
camera.lookAt(target)

lights = ...

Then I can reproduce the camera look-at problem in the notebook too. :confused: :thinking: :question:

[jasongrout]

Then I can reproduce the camera look-at problem in the notebook too. 😕 🤔 ❓

This perhaps points to something wrong in pythreejs?

[Peque]

@jasongrout Should not always ipywidgets reproduce the same results when exporting?

I mean, the fact that changing the place where I call lookAt() results in the Jupyter view also reproducing the "jump" may be an issue with pythreejs, or may be just how three.js is expected to work. But I would expect the exported view to reproduce the same view as in the notebook. For both cases when the view does and does not "jump" in the notebook.

Also, the perspective issue, I can not reproduce it within the notebook.

[vidartf]

@Peque this has to do with a bug in how pythreejs manages the syncing <kernel/embed state> - JS widget model - threejs objects. I'll try to have a quick look.

[vidartf]

Note to self: Likely the CombinedCamera needs the following logic as well:

https://github.com/jupyter-widgets/pythreejs/blob/5b6dcaf7ec0a17a6e5f2820201730cf1056cebe5/js/src/cameras/PerspectiveCamera.js#L6-L10

[Peque]

@vidartf If you have an StackOverflow account and want to reply with "It is a bug":

https://stackoverflow.com/questions/59586889

At least the bounty will not be lost...

I put another 200 bounty in a previous question but that one was unfortunately lost (got no replies). :sweat_smile:

[bernhard-42]

@vidartf cc: @Peque

I have built a clearer example:

The green axes are at the origin of the coordinate system and the red axes at the center of the box which is also the target of OrbitControls:

from pythreejs import *
from IPython.display import display

class Axes:
    def __init__(self, target, color, length=1, width=3):        
        self.axes = [LineSegments2(
            LineSegmentsGeometry(positions=[[target, self._shift(target, vector)]]),
            LineMaterial(linewidth=width, color=color)
        ) for vector in ([length, 0, 0], [0, length, 0], [0, 0, length])]

    def _shift(self, v, offset):
        return [x + o for x, o in zip(v, offset)]

view_width = 600
view_height = 400

material = MeshLambertMaterial(color="#ff00ff", transparent=True, opacity=0.5)

lights = [
    PointLight(position=[100, 0, 0], color="#ffffff"),
    PointLight(position=[0, 100, 0], color="#bbbbbb"),
    PointLight(position=[0, 0, 100], color="#888888"),
    PointLight(position=[-100, 0, 0], color="#bbbbbb"),
    PointLight(position=[0, -100, 0], color="#888888"),
    PointLight(position=[0, 0, -100], color="#444444"),
    AmbientLight(intensity=0.2),
]

origin = (0, 0, 0)
center = (10, 30, -20)
box = Mesh(BoxBufferGeometry(10, 10, 10), material, position=center)

camera_position = [center[i] + 50 for i in range(3)]
camera = CombinedCamera(position= camera_position, 
                        width=view_width, height=view_height)
camera.mode = "perspective"

orbit = OrbitControls(controlling=camera, target=center, target0=center)
orbit.exec_three_obj_method('update')

axes_0 = Axes(origin, length=12, color="green")
axes_t = Axes(center, length=12, color="red")

scene = Scene(children=[box] + axes_0.axes + axes_t.axes + lights)
renderer = Renderer(scene=scene, camera=camera, controls=[orbit],
                    width=view_width, height=view_height)

camera.zoom = 1

display(renderer)

Now, if I export it, I get:

To me it looks like if the embedded renderer in the browser would per default center on the origin of the coordinate system, even when the target of the OrbitControls is somewhere else. As soon as one starts to drag the object, the update function of OrbitControls kicks in and re-centers the view to the target of OrbitControls.

Maybe that helps in identifying the underlying issue.

[Peque]

@vidartf Were you able to find some time to have a look at this? :innocent:

[vidartf]

I hope to do a cycle on pythreejs this week or next to ensure it works with jupyterlab 2.0. I'll go through the open issues/PRs at the same time, and do a release after lab 2.0 is out.

[bernhard-42]

Quite some time passed since we discussed it. I would still be interested in a solution for this ;-)

[bernhard-42]

@vidartf I just debugged it and actually, it's the call to OrbitControls.update that's missing. When you invoke the event MouseMove, this update will be executed and the object moved to the correct location.

Now that I found that, I see it is in line with the behaviour in JupyterLab. As seen above (https://github.com/jupyter-widgets/pythreejs/issues/308#issuecomment-575915840) the pythreejs code explicitly calls orbit.exec_three_obj_method('update'). If you omit that, then the same jumping behaviour appears in JupyterLab.

Now, can we call OrbitControls.update for all OrbitControlModels in the exported HTML file (e.g. by adding some javascript code to the template at the onload event?)

Or is there another way to enforce OrbitControls to execute update after loading of the HTML file with an embedded pythreejs renderer?

[bernhard-42]

@vidartf @jasongrout

I think I found it: The OrbitController in fact needs to call update when initialized with target != (0, 0, 0)

The OrbitController already has a pythreejs specific function update_controlled https://github.com/jupyter-widgets/pythreejs/blob/150ff1c10c868b17fefa63d19153b5ee1fe87f66/js/src/controls/OrbitControls.js#L25 which gets called when the end event of the OrbitController is dispatched.

However:

1. This event will not be dispatched at rendering end, only at three mouse events as far as I saw
2. Saving changes alone in update_controlled is not sufficient. The controller also needs to be updated.

A working code in JupyterLab is (see the 2 lines marked wit // <== NEW)

var OrbitControlsModel = OrbitControlsAutogen.OrbitControlsModel.extend({

    constructThreeObject: function() {
        var controlling = this.get('controlling');
        var obj = new OrbitControls(controlling.obj);
        obj.dispose();  // Disconnect events, we need to (dis-)connect on freeze/thaw
        obj.enableKeys = false; // turn off keyboard navigation

        return obj;
    },

    setupListeners: function() {
        OrbitControlsAutogen.OrbitControlsModel.prototype.setupListeners.call(this);
        var that = this;
        this.obj.addEventListener('end', function() {
            that.update_controlled();
        });

        this.update_controlled();  // <== NEW
    },

    update_controlled: function() {
        // Since OrbitControls changes the position of the object, we
        // update the position when we've stopped moving the object.
        // It's probably prohibitive to update it in real-time
        var controlling = this.get('controlling');
        var pos = controlling.obj.position;
        var qat = controlling.obj.quaternion;
        controlling.set(
            {
                position: pos.toArray(),
                quaternion: qat.toArray(),
                zoom: controlling.obj.zoom,
            },
            'pushFromThree'
        );
        controlling.save_changes();

        // Also update the target
        this.set({
            target: this.obj.target.toArray(),
        }, 'pushFromThree');
        this.save_changes();

        this.obj.update();  // <== NEW
    },

});

With this change everything works fine in JupyterLab. And given that the embedded widget uses the same Javascript code, I would expect that this would also work for embeddings.

Calling this.update_controlled in setupListeners doesn't sound right, but was a quick solution. A maybe better option would be to dispatch the end event for every widget at the end of the initialize method of ThreeModel. However, I do not understand enough of pythreejs to judge on how to implement it.

[bernhard-42]

I have now published orbitcontrol-patch (https://github.com/bernhard-42/orbitcontrol-patch) where I have simply added this.obj.update to setupListeners. It works in Jupyter and embedded HTML pages.

It seems to work for me, however, it would be nice if this would be supported by pythreejs at some point of time

[vidartf]

I agree with your analysis that obj.update() needs to be called at the appropriate times. It would probably be more generic to add it when any of the properties are set from the kernel via the syncToThreeObj method:

syncToThreeObj: function(force) {
  // call super method 
  OrbitControlsAutogen.OrbitControlsModel.prototype.syncToThreeObj.apply(this, arguments);
  this.obj.update();
}