Unity | HoloLens 2 - MouseEventsWebView.MouseEvent does nothing?

[Mt-Perazim]

Question: I have WebView2 set up in my Unity project. Now I would like to transfer my clicks from a canvas to the WebView. However, nothing happens after passing a WebViewMouseEventData. Did I forget something or do I have to update something again myself?

I have read the following article and at the end it roughly shows how to do it.

And even if I send wrong coordinates, I would have to send randomly usable coordinates at some point so that something on the web page gets clicked. But that doesn't happen.

Does there need to be more done than just handing over the MouseEvent?

My Setup:

        public void OnPointerClicked(MixedRealityPointerEventData eventData)
        {
            var result = eventData.Pointer.Result;
            if (result.CurrentPointerTarget.layer != 30)
            {
                Debug.Log("OnPointerClicked>layer != 30");
                return;
            }

            var clickPosition = result.Details.PointLocalSpace;
            var x = (int)Math.Ceiling(clickPosition.x);
            var y = (int)Math.Ceiling(clickPosition.y);
            Debug.Log("OnPointerClicked > X: " + x + " Y: " + y);

            var mouseEventsWebView = _webView as IWithMouseEvents;
            WebViewMouseEventData mouseEvent = new WebViewMouseEventData
            {
                X = x,
                Y = y,
                Device = WebViewMouseEventData.DeviceType.Pointer,
                Type = WebViewMouseEventData.EventType.MouseDown,
                Button = WebViewMouseEventData.MouseButton.ButtonLeft,
                TertiaryAxisDeviceType = WebViewMouseEventData.TertiaryAxisDevice.PointingDevice
            };

            mouseEventsWebView.MouseEvent(mouseEvent);
        }

[champnic]

@michaelfarnsworth Any ideas here?

[michaelfarnsworth]

@Mt-Perazim thanks for bring this up. Input is a topic that could use some more attention in our documentation and samples, as you've discovered. It's a bit of a tricky topic, for a variety of reasons. Regardless, here's an approach that will hopefully get you a little further along with the sample code.

At a high level, you need to map any pointer event from the space of the associated Unity object, e.g. the canvas that's hosting the WebView control, into the WebView's space. Essentially, you normalize the point, then map it onto the WebView using it's texture dimension. Note that that size 1280x720, is not the same that Unity is rendering. Hence the need for the translation.

First, add a public MeshCollider variable to the WebViewBrowser script. Then add a MeshCollider component to the WebView component within the UnityEditor. Then hookup the new MeshCollider to the new Collider you exposed on the WebViewBrowser script.

Like this:

Update WebViewBrowser to derive from IMixedRealityPointerHandler, then add the following code to the class:

  public void OnPointerClicked(MixedRealityPointerEventData eventData)
  {
      var hitCoord = NormalizeWorldPoint(eventData.Pointer.Result.Details.Point);

      hitCoord.x *= _webView.Width;
      hitCoord.y *= _webView.Height;

      var mouseEventsWebView = _webView as IWithMouseEvents;
      WebViewMouseEventData mouseEvent = new WebViewMouseEventData
      {
          X = (int)hitCoord.x,
          Y = (int)hitCoord.y,
          Device = WebViewMouseEventData.DeviceType.Pointer,
          Type = WebViewMouseEventData.EventType.MouseDown,
          Button = WebViewMouseEventData.MouseButton.ButtonLeft,
          TertiaryAxisDeviceType = WebViewMouseEventData.TertiaryAxisDevice.PointingDevice
      };

      mouseEventsWebView.MouseEvent(mouseEvent);

      // To register as a click, the WebView needs to be a mouse-up event.
      mouseEvent.Type = WebViewMouseEventData.EventType.MouseUp;
      mouseEventsWebView.MouseEvent(mouseEvent);
  }

  private Vector2 NormalizeWorldPoint(Vector3 worldPoint)
  {
      // Convert the world point to our control's local space.
      Vector3 localPoint = transform.InverseTransformPoint(worldPoint);

      var boundsSize = collider.sharedMesh.bounds.size;
      var boundsExtents = collider.sharedMesh.bounds.max;

      // Adjust the point to be based on a 0,0 origin.
      var uvTouchPoint = new Vector2((localPoint.x + boundsExtents.x), -1.0f * (localPoint.y - boundsExtents.y));

      // Normalize the point so it can be mapped to the WebView's texture.
      var normalizedPoint = new Vector2(uvTouchPoint.x / boundsSize.x, uvTouchPoint.y / boundsSize.y);

      return normalizedPoint;
  }

[Mt-Perazim]

@michaelfarnsworth Thank you very much! I would not have found out that way. I will try this out tomorrow.

[michaelfarnsworth]

@Mt-Perazim one limitation I forgot to mention with this solution: this will only work with far interactions. Near (i.e. poke interactions) won't work with this.

[Mt-Perazim]

@michaelfarnsworth I'm just getting around to looking at it and trying it out today. I have a few questions about it. I'm using instead of a Mesh Collider a Box Collider and I'm not creating a uvTouchPoint and normaliedPoint. But do I really need that if I receive the right coordinates? I mean, if I click and get a coordinate within the resolution 1280 x 720 then my way should work or not?

In my setup I have a canvas and a box collider and both are set to the resolution of 1280 x 720. If I click in the middle so I get the valid coordinates of 640 x 360. Shouldnt that be enough to send x:640 and y:360 to the webview?

In your implementation you only get the collider.sharedMesh.bounds.size and collider.sharedMesh.bounds.max because you need the "resolution" of your mesh collider or do I understand this wrong?

---- Edit ---- I tried your implementation and it works :)

[Mt-Perazim]

@michaelfarnsworth I debugged the coordinates you are getting and working with and I see what you meant. Those are different coordinates than I thought.

[michaelfarnsworth]

@Mt-Perazim you're correct. If your canvas has the same resolution and orientation as the WebView control, then you shouldn't need to transform the pointer coordinates.

[OdincoGaming]

@Mt-Perazim thanks for bring this up. Input is a topic that could use some more attention in our documentation and samples, as you've discovered. It's a bit of a tricky topic, for a variety of reasons. Regardless, here's an approach that will hopefully get you a little further along with the sample code.

At a high level, you need to map any pointer event from the space of the associated Unity object, e.g. the canvas that's hosting the WebView control, into the WebView's space. Essentially, you normalize the point, then map it onto the WebView using it's texture dimension. Note that that size 1280x720, is not the same that Unity is rendering. Hence the need for the translation.

First, add a public MeshCollider variable to the WebViewBrowser script. Then add a MeshCollider component to the WebView component within the UnityEditor. Then hookup the new MeshCollider to the new Collider you exposed on the WebViewBrowser script.

Like this:

Update WebViewBrowser to derive from IMixedRealityPointerHandler, then add the following code to the class:

  public void OnPointerClicked(MixedRealityPointerEventData eventData)
  {
      var hitCoord = NormalizeWorldPoint(eventData.Pointer.Result.Details.Point);

      hitCoord.x *= _webView.Width;
      hitCoord.y *= _webView.Height;

      var mouseEventsWebView = _webView as IWithMouseEvents;
      WebViewMouseEventData mouseEvent = new WebViewMouseEventData
      {
          X = (int)hitCoord.x,
          Y = (int)hitCoord.y,
          Device = WebViewMouseEventData.DeviceType.Pointer,
          Type = WebViewMouseEventData.EventType.MouseDown,
          Button = WebViewMouseEventData.MouseButton.ButtonLeft,
          TertiaryAxisDeviceType = WebViewMouseEventData.TertiaryAxisDevice.PointingDevice
      };

      mouseEventsWebView.MouseEvent(mouseEvent);

      // To register as a click, the WebView needs to be a mouse-up event.
      mouseEvent.Type = WebViewMouseEventData.EventType.MouseUp;
      mouseEventsWebView.MouseEvent(mouseEvent);
  }

  private Vector2 NormalizeWorldPoint(Vector3 worldPoint)
  {
      // Convert the world point to our control's local space.
      Vector3 localPoint = transform.InverseTransformPoint(worldPoint);

      var boundsSize = collider.sharedMesh.bounds.size;
      var boundsExtents = collider.sharedMesh.bounds.max;

      // Adjust the point to be based on a 0,0 origin.
      var uvTouchPoint = new Vector2((localPoint.x + boundsExtents.x), -1.0f * (localPoint.y - boundsExtents.y));

      // Normalize the point so it can be mapped to the WebView's texture.
      var normalizedPoint = new Vector2(uvTouchPoint.x / boundsSize.x, uvTouchPoint.y / boundsSize.y);

      return normalizedPoint;
  }

what if im not using openxr and wanted to implement this with unitys standard pointerEventData? a direct conversion isnt working, since regular pointereventdata doesnt contain the same kind of point information.