RayCast from screen coordinates?

[Premaider2]

So for a university projekt I need to get the 3D Coordinates of car-wheels. My project-partner works on detecting the cars wheels as circles with openCV. From him I will get 2D screen coordinates. I did a bit of research and found Linetracing/Raytracing (https://developers.google.com/ar/reference/unreal/arcore/blueprint/ARCore_Line_Trace). But it seems not to be able to trace real world geometry.

Then I found Hitresult (https://developers.google.com/ar/reference/java/arcore/reference/com/google/ar/core/HitResult) that works with real-world geometry. But hitresult can only be called by tapping the screen (called by a detected plane), can it? I would need it to be called automatically.

I need a solution that just traces for real world geometry from a given 2d screen coordinate. It should detect the cars wheels as geometry and put my models on the right place. How can I make that happen? Please help. Thanks in advance

[Premaider2]

I made it work with ArFrame.HitTest(..), but this only includes scene geometry. What can I use for real world geometry?

[boehm-e]

Hi, you can do it like this:

        List<HitResult> hits = frame.hitTest(x, y);
        float minDist = 1000;
        HitResult hit = null;
        for (int i=0; i<hits.size(); i++) {
            if (hits.get(i).getDistance() < minDist) {
                minDist = hits.get(i).getDistance();
                hit = hits.get(i);
            }
        }

check this link: https://github.com/boehm-e/ofxARCore2/blob/master/ofxARCoreLib/src/main/java/cc/ofxarcorelib/ofxARCoreLib.java#L299

[Premaider2]

Hi, you can do it like this:

        List<HitResult> hits = frame.hitTest(x, y);
        float minDist = 1000;
        HitResult hit = null;
        for (int i=0; i<hits.size(); i++) {
            if (hits.get(i).getDistance() < minDist) {
                minDist = hits.get(i).getDistance();
                hit = hits.get(i);
            }
        }

check this link: https://github.com/boehm-e/ofxARCore2/blob/master/ofxARCoreLib/src/main/java/cc/ofxarcorelib/ofxARCoreLib.java#L299

But u still use frame.hitTest which in my expierence only works with detected planes etc.

I will still try

[boehm-e]

According to the documentation: Defines an intersection between a ray and estimated real-world geometry.

As ARCore only detect plane surfaces, it might not be possible to hitTest shapes like spheres. Some peoples has done 3D Scanning of random shapes (app link), so you might want to look at it.

[boehm-e]

So you might be using Point Cloud : https://developers.google.com/ar/reference/java/arcore/reference/com/google/ar/core/PointCloud

[Premaider2]

According to the documentation: Defines an intersection between a ray and estimated real-world geometry.

As ARCore only detect plane surfaces, it might not be possible to hitTest shapes like spheres. Some peoples has done 3D Scanning of random shapes (app link), so you might want to look at it.

Unfortunately it is only open source under OpenConstructor not on ARCore.

I now do not have any clue how to find the correct positioning. I would have to find real-world raytracing or detect my cars wheels as custom planes and I have no Idea on how to do that.

[boehm-e]

What about Point Cloud?

An other solution is to use Augmented Images. You take a picture of your wheels, you define the real size of it in your AugmentedImages Databse. And then when you point your camera at it, you can easily retreive its relative position, and then the distance from the phone.

[Premaider2]

What about Point Cloud?

An other solution is to use Augmented Images. You take a picture of your wheels, you define the real size of it in your AugmentedImages Databse. And then when you point your camera at it, you can easily retreive its relative position, and then the distance from the phone.

AugmentedImages is not an option unfortunately. We want it to be used on every possible car wheel.

What do you mean with point cloud? Please explain. Thanks Edit: You mean get a HitResult with a point on the cars wheels?

[boehm-e]

ARCore uses a point cloud to track the position: It tracks 3D points in space frame by frame. You could retrieve these 3d points, compute their position in 2d Screen, and check the distance between each screen point and the center of the circle you detected

See this : https://github.com/google-ar/arcore-android-sdk/issues/495#issuecomment-407234138

[Premaider2]

ARCore uses a point cloud to track the position: It tracks 3D points in space frame by frame. You could retrieve these 3d points, compute their position in 2d Screen, and check the distance between each screen point and the center of the circle you detected

See this : #495 (comment)

Sounds like an idea that could work but there would be way to many 3d points to compute their position at the 2D screen. How to I only use the 3D-points that actually matter to me? And how would I calculate their screen position? --> Edit: Found it. Camera.worldToScreenPoint By checking the distance between each screen point and the circle-center-screen-point I could calculate which 3D point I should use right? You are really getting me further, thanks :)

[boehm-e]

There will not be too many 3d points to compute (because they are not all stored by ARCore and accumulated at every frame: the points that are no longer used to determine the position will be removed) So you only have the points in the camera field of view (not sure of that, but you definitely not getting all points from the beginning of the tracking).

1) : you get all pointCloud : PointCloud : getPoints

2) for each point, you calculate its location on the screen :

• if you are using Java SDK
• if you are using SceneForm : way easier but I dont think you'r using it

3) You compare each points (from point cloud) location on the screen with the center of your circle (found with opencv) this formula

Hope it helps

[Premaider2]

I use sceneform. I tried the following:

`PointCloud pointCloud = globalArFragment.getArSceneView().getArFrame().acquirePointCloud(); Camera camera = globalArFragment.getArSceneView().getScene().getCamera();

    FloatBuffer pointCloudData = pointCloud.getPoints();
    float[] pointCloudArray = new float[pointCloudData.limit()];

    Vector3[] pointArray = new Vector3[pointCloudArray.length];

    if (pointCloudArray.length > 0) {
        for (int i = 0; i < pointCloudArray.length; i++) {
           pointArray[i] = new Vector3(pointCloudArray[i*4],pointCloudArray[(i*4)+1],pointCloudArray[(i*4)+2]);
        }

    }`

But I only get back Vectors with coordinates x=0,y=0,z=0. Basically empty points. What do I do wrong?