[MAYA-112806] Unexpected selection

[csyshing]

Describe the bug

Hi,

We found an unexpected selection issue that happens for some of our production shots, when selecting something on viewport, it's actually selecting another prim at opposite direction, please have a look for the GIF:

In the example scene, there are three prims:

/root/r_0014/r_01, /root/r_0008/r_01 : instanced prim referencing the same prototype
/root/r_0014/r_02                    : non-instanced prim

(Notice that the instanced prims in the example are referencing instancer in the same file but in our production shot the instancer is an external file).

The actual mesh was simply a scaled cube; there used to be USD geometries in the same position as 'sphere' and 'cube' but replacing with native Maya geometry to better isolate the issue.

As you can see from the GIF, select the area around the cube (do not need to accurately select the cube) on viewport is selecting the /root/r_0014/r_01 (or /root/r_0014/r_02) on the opposite direction.

All three prims should be individual and their xformOps do not seem to have obvious mistake, but the problem seems to be related to bounding box calculation or hit test intersection, not sure exactly how that happen.

Steps to reproduce Steps to reproduce the behavior:

1. Launch Maya and open the attached USD file
2. Create the sphere and cube to reproduce the selection:

   
   // MEL:
   CreatePolygonSphere;
   move -1633.949 243.508 -439.816 pSphere1;

CreatePolygonCube; move -1626.266 245.549 -443.803 pCube1;

3. Frame to the sphere, and rotate the camera angle to let the cube roughly in the middle of the viewport (see the GIF)
4. Select the cube and area round it, the /root/r_0014/r_01 (or /root/r_0014/r_02, depends on camera angle or selection point) prim would be selected.

**Expected behavior**
Select prim on viewport correctly.

**Attachments**
[incorrect_selection.zip](https://github.com/Autodesk/maya-usd/files/6838294/incorrect_selection.zip)

**Specs (if applicable):**
 - OS & version: CentOS 7.4
 - Compiler & version: gcc 6.3.1
 - Maya version: Maya 2020 / Maya 2022
 - Maya USD: Official release v0.9.0 / v0.10.0
 - Pixar USD Official release USD-20.11 / USD-21.05

(any of the version combination can reproduce the problem)

[williamkrick]

I reproduced this one today and logged MAYA-112806.

[fabal]

Hi @williamkrick , has there been any progress made on that one?

[williamkrick]

@fabal @csyshing I believe we've fixed this in the most recent Maya Preview Release.

[santosd]

Closing this out as it has been fixed in the latest Maya PR. @fabal let us know if you hit any issues and we can reopen this issue.

[csyshing]

Hi @santosd , may I ask which Maya PR you were referring to, so that I can double check? Wanted to see if we have the expected Maya PR installed. Thanks!

[santosd]

Hello @csyshing, I tested in PR131 and the issue is resolved in that version so anything above that PR should have the fix in it. You should also be able to test in Maya 2023 which recently released. Let me know if you come across any issues.