Raycast does not return any hits

[fieldsJacksonG]

In the sample OpenXR C++ project, I have a remote model loaded at 0,0,0 with collisions enabled in the model conversion, verified in the .info.json file: "generateCollisionMesh": true,

I am attempting to raycast in the PollActions() function after xrLocateSpace for the hand cube with this code:

RR::RayCast RayCast;

XrVector3f handPos = handLocation.pose.position;
RayCast.StartPos = RR::Double3{ handPos.x, handPos.y, handPos.z };
RayCast.EndPos = RR::Double3{ handPos.x, handPos.y, handPos.z - 10.0f};
RayCast.HitCollection = RR::HitCollectionPolicy::ClosestHit;

m_api->RayCastQueryAsync(RayCast, [this](RR::Status Status, RR::ApiHandle<RR::RayCastQueryResult> Result)
{
    std::vector<RR::RayCastHit> hits;
    Result->GetHits(hits);

    if (hits.size() > 0)
    {
        RR::Double3 hitPos = hits[0].HitPosition;
    }
});

However, hits is always an empty vector despite my starting position in a valid positive-z offset from the remote model. In the sample, negative-z is forward in the scene, but I have also tried with handPos.z + 10 in case the NegativeZ value in RemoteRenderingInitialization::Forward is converting this field. I have also tried with hard-coded values that should definitely intersect with the remote model.

I have also tried scaling the remote model up significantly to guarantee that my raycasts should hit.

Do I need to call any special ARR API to opt-into raycasts, or should this work as-is? Is there any way to debug the raycast on the server-side?

[jumeder]

Hi @fieldsJacksonG, sorry to hear you are having trouble with ARR. Can you check what 'Status' is getting passed into the lambda? It is possible an error occurs during the raycast.

To answer your questions, the only prerequisite for raycasting should be the presence of collision meshes in the respective scene, so I think it should work in your setup. I don't believe we have a debug mode for raycasts. If you want to verify if your ray should hit the scene, I recommend drawing it locally as a line and verifying this visually.

[jumeder]

Hi again @fieldsJacksonG, if you want to rule out the scene being a problem, you can load the builtin://Engine model, which can definitely be used with ray casts.

[fieldsJacksonG]

Unfortunately builtin://Engine is also failing to return any hits Status is "Ok"

[jankrassnigg]

In that case I would assume your raycast either has a wrong direction or starting position. It is possible that the hand position is returned in a different space, maybe relative to some reference point. It is probably best to somehow visualize the ray, either locally or by placing some dummy entities along it.

[fieldsJacksonG]

Ray goes from the right avocado to the left avocado, cleanly through the big center avocado, but no hits are registered:

All avocados are in the same space as the ray, with hardcoded values below: (Big avocado is at 0,0,0 with a scale of 10,10,10)

RR::RayCast RayCast;

XrVector3f handPos = handLocation.pose.position;
RayCast.StartPos = RR::Double3{ 0, 0.25f, 0.75f };
RayCast.EndPos = RR::Double3 {0, 0.25f, -0.75f};
RayCast.HitCollection = RR::HitCollectionPolicy::ClosestHit;

RR::Float3 StartScale = { 1, 1, 1 };
RR::Float3 EndScale = { 1.5f, 1.5f, 1.5f };
StartModelLoading(RayCast.StartPos, StartScale);
StartModelLoading(RayCast.EndPos, EndScale);

m_api->RayCastQueryAsync(RayCast, [this](RR::Status Status, RR::ApiHandle<RR::RayCastQueryResult> Result)
{
    if (Status != RR::Status::OK)
    {
        return;
    }

    std::vector<RR::RayCastHit> hits;
    Result->GetHits(hits);

    // hits.size() is always 0
    if (hits.size() > 0)
    {
        RR::Double3 hitPos = hits[0].HitPosition;
    }
});

void StartModelLoading(RR::Double3 pos, RR::Float3 scale) {
    m_modelLoadingProgress = 0.f;

    RR::LoadModelFromSasOptions params;
    params.ModelUri = m_modelURI.c_str();
    params.Parent = nullptr;

    // start the async model loading
    m_api->LoadModelFromSasAsync(
        params,
        // completed callback
        [this, pos, scale](RR::Status status, RR::ApiHandle<RR::LoadModelResult> result) {
            m_modelLoadResult = RR::StatusToResult(status);
            m_modelLoadFinished = true;

            if (m_modelLoadResult == RR::Result::Success) {
                result->GetRoot()->SetPosition(pos);
                result->GetRoot()->SetScale(scale);
            }
        },
        // progress update callback
        [this](float progress) {
            // progress callback
            m_modelLoadingProgress = progress;
        });
}

This is the world bounding box from the entity's QueryWorldBoundsAsync:

MaxBounds   {X=0.21280911564826965 Y=0.62848055362701416 Z=0.13809001445770264 }
MinBounds   {X=-0.21280911564826965 Y=-0.00047740340232849121 Z=-0.13809001445770264 }

The above ray should go through these bounds

[jankrassnigg]

OK, the code looks reasonable. It's possible the raycasts have problems with scaled objects, at least other engines have problems with that. Could you try it with all objects scaled to 1 and perfectly straight on a line? Also, just to be certain, you repeat the raycast every frame, right? Because right now the code looks like you only do it once when you start loading, and at that time it can't hit anything, of course.

[ChristopherManthei]

The problem is most likely that you didn't set up the right coordinate system for your client space so that ARR is able to convert between the client coordinate system and the server coordinate system. This is configured via the RemoteRenderingInitialization struct. This is for example the coordinate system used in our C++ sample app:

        RR::RemoteRenderingInitialization clientInit;
        clientInit.ConnectionType = RR::ConnectionType::General;
        clientInit.GraphicsApi = RR::GraphicsApiType::WmrD3D11;
        clientInit.ToolId = "<sample name goes here>"; // <put your sample name here>
        clientInit.UnitsPerMeter = 1.0f;
        clientInit.Forward = RR::Axis::NegativeZ;
        clientInit.Right = RR::Axis::X;
        clientInit.Up = RR::Axis::Y;

Most 3d engines have different coordinate spaces, e.g. Unity uses:

        clientInit.UnitsPerMeter = 1.0f;
        clientInit.Forward = RR::Axis::Z;
        clientInit.Right = RR::Axis::X;
        clientInit.Up = RR::Axis::Y;

While Unreal uses:

        clientInit.UnitsPerMeter = 100.0f;
        clientInit.Forward = RR::Axis::X;
        clientInit.Right = RR::Axis::Y;
        clientInit.Up = RR::Axis::Z;

[fieldsJacksonG]

I had switched from Unreal to the sample C++ OpenXR project for faster iteration time while debugging this, so a ray from +z to -z should hit (possibly on the opposite side if the start and end are flipped from the sample's clientInit.Forward = RR::Axis::NegativeZ)

Still no collisions when the remote model is not scaled. (Raycasts are attempted on every hand select click action)

Here's a repro: https://github.com/fieldsJacksonG/azure-remote-rendering/commit/7bca0e980d14f15bd9748cc32b8dd2a9dd3a884f

[jakrams]

Hey @fieldsJacksonG thanks for the repo case. After quite a bit of searching, we've identified the problem. The structure RR::RayCast has a member "CollisionMask", which the documentation says is "reserved for future use". We experimented with it a long time ago, and forgot to deactivate its use. In the C#/Unity projection the default value is set such that it works, but in the C++ projection it doesn't get the necessary default value, which is why here it can have an unintended effect.

The fix for you is, to set the value to 0xFFFFFFFF. We'll also push an update to the server code in one of the next releases, to actually not use the value, to be in line with the documented behavior.

Sorry for the inconvenience.

[fieldsJacksonG]

Thanks! That collision mask works!