Windows (& popups) should spawn directly in front of user

[georgewsinger]

Problem: Right now all new windows spawn at (0,0,0). We'd like instead to spawn windows a few feet in front of the user's eye gaze.

First attempt: As a first attempt, the ARVRCamera has scene graph address "/root/Root/ARVROrigin/ARVRCamera":

Yet trying to grab the ARVRCamera via

arvrCamera <- getARVRCameraFromPath gsvs

where getARVRCameraFromPath :: GodotSimulaViewSprite -> IO GodotARVRCamera
      getARVRCameraFromPath self = do
        let nodePathStr = "/root/Root/ARVROrigin/ARVRCamera"
        nodePath <- (toLowLevel (pack nodePathStr))
        gssNode  <- G.get_node ((safeCast self) :: GodotNode) nodePath
        arvrCamera      <- (fromNativeScript (safeCast gssNode)) :: IO GodotARVRCamera
        return arvrCamera

compiles just fine, but yields run-time error

ERROR: _get_node: Can't use get_node() with absolute paths from outside the active scene tree.
   At: scene/main/node.cpp:1290.
ERROR: get_node: Node not found: /root/Root/ARVROrigin/ARVRCamera
   At: scene/main/node.cpp:1354.
handle_crash: Program crashed with signal 11
ERROR: notification: NativeScriptInstance detected crash on method: _on_WlrXdgShell_new_surface
   At: modules/gdnative/nativescript/nativescript.cpp:767.

My ultimate goal is to transform sprite's via

arvrCamera <- getARVRCameraFromPath gsvs -- getSingleton GodotARVRCamera "ARVRCamera"
hmdGlobalTransform <- G.get_global_transform (arvrCamera)
G.set_global_transform godotSprite3D hmdGlobalTransform

when they are first made. This should transform the sprite to the location of the user's face (and then if this works, I'll need to push them away from the user's face by a suitable distance + rotate them such that they are facing the user).

[georgewsinger]

The issue above was I was trying to use our gsvs :: GodotSimulaViewSprite in the call to G.get_node before it was actually added to the scene graph. Here I try making the call to G.get_node in our map handler after the gsvs has been added to the scene graph:

handle_map_surface :: GFunc GodotSimulaServer
handle_map_surface gss args = do
  putStrLn "handle_map_surface"
  case toList args of
    [gsvsVariant] -> do
       maybeGsvs <- variantToReg gsvsVariant :: IO (Maybe GodotSimulaViewSprite)
       case maybeGsvs of
         Nothing -> putStrLn "Failed to cast GodotSimulaViewSprite in handle_map_surface!"
         Just gsvs -> do G.add_child ((safeCast gss) :: GodotNode )
                                     ((safeCast gsvs) :: GodotObject)
                                     True
                         godotSprite3D <- readTVarIO (gsvs ^. gsvsSprite)
                         arvrCamera <- getARVRCameraFromPath gss
                         hmdGlobalTransform <- G.get_global_transform (arvrCamera) -- <-- Causes actual crash
                         G.set_global_transform godotSprite3D hmdGlobalTransform
                         -- ..
    _ -> putStrLn "Failed to get arguments in handle_map_surface"
  toLowLevel VariantNil
  where getARVRCameraFromPath :: GodotSimulaServer -> IO GodotARVRCamera
        getARVRCameraFromPath self = do
          let nodePathStr = "/root/Root/ARVROrigin/ARVRCamera"
          nodePath <- (toLowLevel (pack nodePathStr))
          gssNode  <- G.get_node ((safeCast self) :: GodotNode) nodePath
          arvrCamera      <- (fromNativeScript (safeCast gssNode)) :: IO GodotARVRCamera
          return arvrCamera

and this yields error:

handle_crash: Program crashed with signal 11
ERROR: notification: NativeScriptInstance detected crash on method: handle_map_surface
   At: modules/gdnative/nativescript/nativescript.cpp:767.
ERROR: notification: NativeScriptInstance detected crash on method: _handle_map
   At: modules/gdnative/nativescript/nativescript.cpp:767.
Dumping the backtrace. Please include this when reporting the bug on https://github.com/godotengine/godot/issues
[1] /lib/x86_64-linux-gnu/libc.so.6(+0x43f60) [0x7f9b5c110f60] (??:0)
[2] /home/george/.stack/programs/x86_64-linux/ghc-tinfo6-8.6.3/lib/ghc-8.6.3/rts/libHSrts-ghc8.6.3.so(stg_deRefStablePtrzh+0xa) [0x7f9b51d58e9a] (??:0)
-- END OF BACKTRACE --
make[1]: *** [Makefile:18: run] Aborted (core dumped)
make[1]: Leaving directory '/home/george/Simula/addons/godot-haskell-plugin'
/home/george/Simula

with gdb backtrace:

Thread 1 "godot" received signal SIGSEGV, Segmentation fault.
0x00007fffecb5ae9a in stg_deRefStablePtrzh () from /home/george/.stack/programs/x86_64-linux/ghc-tinfo6-8.6.3/lib/ghc-8.6.3/rts/libHSrts-ghc8.6.3.so
(gdb) bt
#0  0x00007fffecb5ae9a in stg_deRefStablePtrzh () at /home/george/.stack/programs/x86_64-linux/ghc-tinfo6-8.6.3/lib/ghc-8.6.3/rts/libHSrts-ghc8.6.3.so
#1  0x0000000000000000 in  ()

[georgewsinger]

The issue above was using fromNativeScript against an ARVRCamera. The function fromNativeScript only works to cast registered, custom types (like GodotSimulaViewSprite or GodotSimulaServer). Here's what does work, sort of:

handle_map_surface :: GFunc GodotSimulaServer
handle_map_surface gss args = do
    -- ..
    godotSprite3D <- readTVarIO (gsvs ^. gsvsSprite)
    arvrCamera <- getARVRCameraFromPath gss
    hmdGlobalTransform <- G.get_global_transform (arvrCamera)
    G.set_global_transform godotSprite3D hmdGlobalTransform
    -- ..
    where getARVRCameraFromPath :: GodotSimulaViewSprite -> IO GodotARVRCamera
          getARVRCameraFromPath self = do
            let nodePathStr = "/root/Root/ARVROrigin/ARVRCamera"
            nodePath <- (toLowLevel (pack nodePathStr))
            gssNode  <- G.get_node ((safeCast self) :: GodotNode) nodePath

            -- Causes the error above:
            -- arvrCamera <- (fromNativeScript (safeCast gssNode)) :: IO GodotARVRCamera

            -- Causes opaque typeclass error:
            -- arvrCamera <- (safeCast gssNode) :: IO GodotARVRCamera

            -- Works with caveats (see below):
            arvrCamera <- (coerce gssNode) :: IO GodotARVRCamera

            return arvrCamera

This succesfully spawns new windows at the location of the user's HMD. The problem is that it weirdly causes Vive Controllers to stop being able to recognize windows:

Notice that there are no rays connecting the controller to the surface. I'm unable to grab windows or interact with them in any way w/the controllers.

In a nutshell, messing with a sprite's global transform seems to have unintended side effects.

[georgewsinger]

Weird controller behavior solved. The weird controller behavior was being caused by me transforming just the Sprite3D and not the full GodotSimulaViewSprite.

Transform pseudo-solution. For some reason our godot-haskell API doesn't include access to the Transform functions, so solutions which involve constructing a single transform and then applying it all at once don't seem possible right now.

What does work sort of (using only Spatial functions, which we do have access to): (i) transform the sprite to the HMD's location; (ii) rotate it by 180 degrees along the y-axis; (iii) push it backward by 1 unit along the z-axis:

handle_map_surface :: GFunc GodotSimulaServer
handle_map_surface gss args = do
    -- ..
    arvrCamera <- getARVRCameraFromPath gss
    hmdGlobalTransform <- G.get_global_transform (arvrCamera)
    G.set_global_transform gsvs hmdGlobalTransform
    G.translate gsvs =<< toLowLevel (V3 0 0 (-1))
    G.rotate_y gsvs 3.14159 -- 180 degrees in radians
    -- ..

Here is what it looks when spawning terminator (keeping HMD the still):

This is far from perfect but definitely better than having sprite's spawn at (0,0,0). If anyone has any suggestions to make this better LMK. (The looking_at function would seem useful here, but we don't have access to this function AFAIK).

[georgewsinger]

• I commited the above code to our gdwlroots-xwayland branch in case Max finds it immediately useful.
• Per @kanetw, we actually can access Transform functions in godot-haskell via godot_transform_*. I'm working remotely today (so no access to VR), but I think something like the following should improve our current psuedo-solution:

handle_map_surface :: GFunc GodotSimulaServer
handle_map_surface gss args = do
    -- ..
    -- Get state
    arvrCamera <- getARVRCameraFromPath gss
    hmdT <- G.get_global_transform (arvrCamera)
    hmdGlobalCoordinates <- G.godot_transform_get_origin hmdGlobalTransform -- 
    rotationAxisX        <- fromLowLevel =<< V3 1 0 0
    rotationAxisY        <- fromLowLevel =<< V3 0 1 0
    rotationAxisY        <- fromLowLevel =<< V3 0 0 1

    -- Construct transform that warps to HMD, pushes back by 1 unit, looks at
    -- the HMD, and then flips around (to get window orientation correct).
    hmdPushedBack <- G.godot_transform_tranlated hmdGlobalTransform (fromLowLevel =<< V3 0 0 (-1))
    hmdPushedBackLookAtX <- G.godot_transform_looking_at hmdPushedBack hmdGlobalCoorindates rotationAxisX
    hmdPushedBackLookAtXY <- G.godot_transform_looking_at hmdPushedBackLookAtX hmdGlobalCoorindates rotationAxisY
    -- hmdPushedBackLookAtXYZ <- G.godot_transform_looking_at hmdPushedBackLookAtXY hmdGlobalCoorindates rotationAxisZ -- Don't think we need Z-axis looking_at?
    hmdPushedBackLookAtXYFlippedY <- G.godot_transform_rotated hmdPushedBackLookAtXY rotationAxisY 3.1459 

    -- Apply the transform all at once to the mapped Sprite3D
    G.set_global_transform gsvs hmdPushedBackLookAtXYFlippedY
    -- ..

I'll test it when I'm back to my VR station.

[georgewsinger]

Local transforms. The above doesn't work, and I think it's because the translations/rotations are global when I need some of them to be local. I discovered Spatial.translate_object_local and Spatial.rotate_object_local functions, which allows me to use local transforms when needed. The following almost works:

setInFrontOfHMD :: GodotSimulaViewSprite -> IO ()
setInFrontOfHMD gsvs = do
  rotationAxisY <- toLowLevel (V3 0 1 0) :: IO GodotVector3
  pushBackVector <- toLowLevel (V3 0 0 (-1)) :: IO GodotVector3 -- For some reason we also have to shift the vector 0.5 units to the right
  hmdGlobalTransform <- getTransform gss
  G.set_global_transform gsvs hmdGlobalTransform
  G.translate_object_local gsvs pushBackVector
  G.rotate_object_local gsvs rotationAxisY 3.14159 -- 180 degrees in radians

except exhibits a weird behavior whereby first sprites spawn too far to the left:

and then after the first launch spawn too far to the right:

(The leftward/rightward skew is much more intense from the PoV of the headset than it is in these pancake mode pictures).

[georgewsinger]

This leftward/rightward skew is problematic for popups in particular (@NerveCoordinator).

[georgewsinger]

:heavy_check_mark: Commenting out moveToUnoccupied in updateSimulaViewSprite seems to make new surfaces (including popups) reliably spawn directly in front of the user's HMD.

updateSimulaViewSprite :: GodotSimulaViewSprite -> IO ()
updateSimulaViewSprite gsvs = do
  drawParentWlrSurfaceTextureOntoSprite gsvs
  setExtents gsvs

  -- whenM (spriteShouldMove gsvs) $ do
  --   atomically $ writeTVar (_gsvsShouldMove gsvs) False
  --   moveToUnoccupied gsvs

  -- ..

Will let @NerveCoordinator test before closing this issue.