Interactivity: Picking

[felixguendling]

I would like to display additional information and change the mouse pointer type when the user hovers over rendered objects and create a new window when an entity is clicked. It would be nice, to be able to efficiently detect the rendered object that's under the user's mouse pointer.

The default way to do this seems to be picking. Examples:

• http://voxelent.com/html/beginners-guide/chapter_8/ch8_Picking.html
• https://github.com/FarhadG/webgl-picking
• http://coffeesmudge.blogspot.de/2013/08/implementing-picking-in-webgl.html

In JS this seems to be hassle-free. But I can't see how this is possible with elm-webgl. This is really sad because the modern web is defined by interactivity which includes not just rendering but also reacting on user input.

Google-ing this topic reveals that there has been a discussion about this topic which ended nowhere: https://groups.google.com/forum/#!topic/elm-discuss/PvVxDq0K8M8

In this thread there are further references to discussions where users ask for this feature:

• https://groups.google.com/forum/#!topic/elm-discuss/7e9e6ZBGkLk/discussion
• http://stackoverflow.com/questions/24345168/how-can-i-use-a-renderbuffer-as-a-texture-in-elms-webgl-library

So there is definitely some demand.

Are there any plans to support this?

If I would try to implement this (I have no idea how hard this will be or whether it's possible at all with the current architecture), does it have any chance to be merged?

Edit: The most important missing functionality seems to be WebGLRenderingContext.readPixels()(Docs)

Edit1: Currently, I'm mostly interested in the 2D case (map overlay) where depth information is not relevant.

[w0rm]

@felixguendling this looks like a very nice to have feature. How do you imagine the api from the elm side?

[felixguendling]

I think it could be accomplished by adding a parameter (List (WebGL.Attribute msg)) to the toHtmlWith function. The attribute would signal a message, i.e. when the mouse gets moved. The first two arguments to the message are the mouse coordinates (relative to the canvas, not to the window). The third argument is the color from the off-screen rendering.

type Attribute msg
    = OnMouseMove (Integer -> Integer -> Vec3 -> msg)
    | OnMouseDown (Integer -> Integer -> Vec3 -> msg)
    | OnMouseUp (Integer -> Integer -> Vec3 -> msg)

toHtmlWith : List FunctionCall -> List (Html.Attribute msg) -> List Renderable -> List (Attribute msg) -> Html msg

Additionally, there needs to be a uniform boolean variable telling the fragment shader whether to render the picking map (not visible to the user) or the real image (visible to the user). This variable would be provided by the system. The Elm code should not be concerned with the creation of the off-screen render buffer and other native functionality. Thus, the fragment shader would look like this:

void main(void) {
    if (uOffscreen) {
        gl_FragColor = uniquePickingColor;
        return;
    }
   // render content for user
}

When the OnMouseMove message is handled by the update method, it just needs to match the provided Vec3 color argument with the uniquePickingColor that was used in the off-screen rendering to know which object is currently under the user's mouse.

By default, this functionality (i.e. the uOffscreen variable) should not be activated. Perhaps it could be enabled by yet another parameter to toHtmlWith or implicitly if the provided list of "picking attributes" isn't empty. Or it could be enabled through a function call (Enable Picking).

I'm not a WebGL expert and I have only little experience bridging native (JS) code into Elm. So I'm currently not aware of the complexity imposed by the described API but I think this would be a feasible approach from an Elm client code perspective.

I currently have the 2D case (map overlay) in mind. Perhaps it would be interesting to have a mode providing depth information (in addition to the x/y coordinates and the color).

Edit: I think the signalling API should more like Html.Events with Signal.Address ... -> Attribute but I hope it's clear how I meant it.

The client code would look like this:

type Msg
    = OnMove Integer Integer Vec3

update : Msg -> Model -> ( Model, Cmd Msg )
update msg model =
    case msg of
        OnMove x y color ->
            ( { model | selected = getSelected color }, Cmd.none )

WebGL.toHtmlWith functionCalls attributes [ WebGL.OnMouseMove OnMove ]

[felixguendling]

Perhaps it would be a cleaner solution to just have two separate fragment shaders instead of one that uses the uniform variable to distinguish both modes. This way, the user can't forget to declare the uOffscreen variable in his shader code.

Feel free to comment or propose a totally different approach. :-)

[felixguendling]

I'm trying to implement a small proof of concept. But it seems not to be possible to enqueue messages from a custom virtual-dom component like the elm-webgl canvas. I don't have access to the enqueue function. elm-html uses the eventNode for this. But this functionality does not seem to be accessible.

Do you know some examples showing how to inject messages into the Elm loop? elm-d3 seems to use elm.notify but I couldn't find this function in Elm 0.17.

[felixguendling]

I've been experimenting with this concept. First result: https://github.com/felixguendling/elm-webgl

Feel free to be creative with the demo. :-)

Missing:

• Correct calculation for mouse coordinates (currently: only working because canvas is at (0, 0))
• Clean way to communicate pixel values (currently: dirty hack using DOM events and Elm Ports)
• Offscreen rendering (currently: the real rendered pixel value)

If you know how to get messages from native code into the Elm loop (i.e. access the enqueue function), please tell me.