2D screen point to 3D world point

[nyxaria]

I am trying to achieve converting a click on the screen to coordinates in the world on the plane x.y=0 (ie z=0, composed of the x and y axis).

I have found code to convert 3D -> 2D (https://github.com/enthought/mayavi/blob/master/examples/mayavi/advanced_visualization/mlab_3D_to_2D.py), however I am unsure how to go about reversing this as I have not covered those topics at university yet. I have attached a screenshot of my setup so far. It is basically a surface at z=0 on which you will be able to drag a 2D actor. For this, I need to find the point on the surface which was clicked and then handle collision detection using bounds of the 2D actor on the surface.

So far, I am able to get the point clicked in the scene window via:

            def press_callback(vtk_obj, event):
                x, y = vtk_obj.GetEventPosition()
                ...

            self.scene.interactor.add_observer('LeftButtonPressEvent', press_callback)

This gives the point from the left-bottom point of the scene UI pane/canvas.

Would it be possible to transform this point to a point on the surface (gray)?

I have tried using a picker but I can't get the correct bounds to be picked when clicking, when I click on the red shape it is not picked, only when I click within the bounds of the black square I drew on in photoshop does the picker pick the shape. I have checked the picker.pick_position, and picker.actor.bounds, and they match up. So it seems to me like the pick_position is not correctly transformed to the world.

However, as you can see by the two white points plotted, the bounds of the red actor/shape are outside the rectangle which is being picked up by the picker. This rectangle of detection bounds scales when you move in and out.

This is how I am calling picker.pick:

            def press_callback(vtk_obj, event):
                x, y = vtk_obj.GetEventPosition()  # screen point in scene UI pane

                result = picker.pick(x, y, 0, self.scene.scene.renderer)
                print(picker.pick_position)
                if result == 0:  # successful pick
                    return
                pickPosition = picker.pick_position
                print([o for o in picker.actors])
                print("pick", pickPosition, [picker.actor], picker.actor.bounds, picker.actor.position)
                if picker.actor == sphere_actor:
                    print("Bingo")

Should I be supplying the (x, y) of the click relative to the left-bottom corner of the scene UI?

Any ideas how to progress? I am trying to achieve dragging functionality of the red 2d shape along the surface at z=0. I am just blocked on getting the Screen Point -> World Point @ surface(z=0) transformation as I can do the boundary detection myself, no need for the picker. Many thanks!

[nyxaria]

If the camera is moved, the region (represented by the black rectangle drawn in post) which if you click in changes as well. To illustrate:

I have also added a dummy actor (the bigger white sphere) and its detection region by the picker is also shifted in an identical manner.

Just FYI: The bottom left corner of the gray square is the origin of the world (0,0,0)

[nyxaria]

It seems like AreaPick would be perfect for this, however it doesn't seem to be ported to the VTK Python API:

vtkAreaPicker.h
81-  /**
82-   * Perform pick operation in volume behind the given screen coordinates.
83-   * Props intersecting the selection frustum will be accessible via GetProp3D.
84-   * GetPlanes returns a vtkImplicitFunction suitable for vtkExtractGeometry.
85-   */
86:  virtual int AreaPick(double x0, double y0, double x1, double y1, vtkRenderer* renderer = nullptr);
87-
88-  /**
89-   * Perform pick operation in volume behind the given screen coordinate.
90-   * This makes a thin frustum around the selected pixel.
91-   * Note: this ignores Z in order to pick everying in a volume from z=0 to z=1.
92-   */
93-  int Pick(double x0, double y0, double vtkNotUsed(z0), vtkRenderer* renderer = nullptr) override
94-  {
95:    return this->AreaPick(x0, y0, x0 + 1.0, y0 + 1.0, renderer);
96-  }
97-