Residue selection using the laser pointer of VR controllers

[Nargaruga]

To the best of my knowledge, it is not currently possible to select a residue by pointing the VR controller's laser directly at the target (akin to clicking with the mouse on the molecule while in desktop mode). Has this feature ever been considered?

I would be interested in looking into ways to implement it, but I would like to know beforehand if there have already been attempts (or if I'm missing an already-existing feature!).

[JarrettSJohnson]

Hi. I haven't tried PyMOL with the OpenVR implementation in a while, but this block seems to suggest that selection should already be possible: https://github.com/schrodinger/pymol-open-source/blob/03d7a7fcf0bd95cd93d710a1268dbace2ed77765/layer3/Executive.cpp#L11231

[Nargaruga]

Hi, thank you for your response. You are right, the piece of code you highlighted picks the position of the pointed atom, but it only appears to be using it to limit the length of the laser pointer, unless I'm missing something.

For the record, I'm using PyMOL 3.0.0 on Ubuntu 22.04 with an Oculus Quest 3 device using ALVR. I was not able to select any atom or residue in VR without using the sequence viewer, but I don't exclude the possibility that it may be a controller configuration issue.

Perhaps (and I may be completely off-track here, of course) actual atom selection could happen by providing the pymol::CObject that is populated by SceneDoXYPick to https://github.com/schrodinger/pymol-open-source/blob/a0d739f8e0af1011ceea6b75949fbf5559e6c5b2/layer1/SceneMouse.cpp#L230 or a similar function whenever the user inputs the equivalent of a click through the controller?

[JarrettSJohnson]

Ah, your interpretation was right. I didn't look into ScenePickAtomInWorld; I assume it did something else.

Unless anyone else speaks up, I would just assume that this was just not implemented. 😄

[Nargaruga]

I was able to select atoms by introducing this function

Function Code

```cpp void OpenVRSelectPickedAtom(PyMOLGlobals * G) { CScene *Scene = G->Scene; if(Scene->LastPicked.context.object == NULL) { return; } COpenVR *I = G->OpenVR; OpenVRActionList* Actions = I->Actions; Actions->Update(I->Input); OpenVRAction* userActions[] = {Actions->Action1, Actions->Action2, Actions->Action3}; OpenVRController& RightHand = I->Hands[HRight]; for (int i = 0, n = sizeof(userActions) / sizeof(*userActions); i < n; ++i) { OpenVRAction* action = userActions[i]; if (action->WasReleased()) { EHand handIndex = EHand(action->DeviceIndex() == RightHand.m_deviceIndex); EUserActionSet userActionSet = I->UserActionSet[handIndex]; EUserAction userAction = s_userActionMapping[userActionSet][i]; switch (userAction) { case UserAction_Mouse_LClick: SceneClickObject(G, Scene->LastPicked.context.object, Scene->LastPicked, cButModeSeleToggle, ""); break; } } } } ```

and calling it after ScenePickAtomInWorld, though it's just an improvised solution.

One issue I am encountering is that the selection seems to happen a few centimeters under the laser. As can be seen in the following screenshot, the index of the atom picked by ScenePickAtomInWorld (printed in the terminal and circled in red in the viewport) does not correspond to the index of the atom pointed by the laser.:

Screenshot

[JarrettSJohnson]

Retrieved a VR headset for me to test this. I haven't opened PyMOL VR in many years, so I'm still learning the ropes again.

I think the issue is due to this line: https://github.com/schrodinger/pymol-open-source/blob/d82b626b60ef4382a49bbbf3b0327bad8d59e9e0/layer1/ScenePicking.cpp#L15 This value allows tolerance to identify atoms from the picking buffer after picking rendering. I'm assuming for VR, this is greatly exaggerated. Setting this value to 1 seems to tighten the tolerance a bit, but still isn't perfect (might be good enough?)