Implementing AR Mode

[georgewsinger]

Goal: Set the background Environment to a webcam Texture to create the illusion the user is wearing an AR headset. It should be toggleable by a keyboard shortcut.

Attempting to set the background to BG_CAMERA_FEED

The Environment type supports BG_CAMERA_FEED:

However, setting it in SimulaServer.hs

G.set_background environment 6 -- BG_CAMERA_FEED

Just causes the background sky to turn black.

Godot doesn't support webcams for Linux

If we inspect the number of detected cameras from godot via CameraServer

      cameraServer <- unsafeInstance GodotCameraServer "CameraServer"
      numCameras <- G.get_feed_count cameraServer
      putStrLn $ "CameraServer get_feed_count: " ++ (show numCameras)

...we just get "0", since it turns out that Godot doesn't even support webcams for Windows, let alone Linux. You can tell this also by file inspection:

george@UbuntuBox:~/Simula/submodules/godot/modules/camera$ ls
camera_ios.h  camera_ios.mm  camera_osx.h  camera_osx.mm  camera_win.cpp  camera_win.h  config.py  __pycache__  register_types.cpp  register_types.h  SCsub

E.g. there's no camera_x11.* here. So maybe we should make one?

godot-webcam doesn't work out of the box

There's godot-webcam, which I got to compile but which doesn't actually get webcam textures to show:

Glimmer of hope: running this demo does get my webcam's green light to turn on.

The CameraFeed machinery seems to already be in Godot, even though camera detection/texture grabbing isn't working

I think our preferred approach should be working with pre-existing CameraFeed machinery within Godot, since the steps are already in place for it to render webcam textures (even though webcams aren't working). E.g. taking a look at RasterizerSceneGLES3::render_scene() from SimulaVR/godot:

void RasterizerSceneGLES3::render_scene(const Transform &p_cam_transform, const CameraMatrix &p_cam_projection, bool p_cam_ortogonal, InstanceBase **p_cull_result, int p_cull_count, RID *p_light_cull_result, int p_light_cull_count, RID *p_reflection_probe_cull_result, int p_reflection_probe_cull_count, RID p_environment, RID p_shadow_atlas, RID p_reflection_atlas, RID p_reflection_probe, int p_reflection_probe_pass) {
//..
    Ref<CameraFeed> feed;
//..
} else if (env->bg_mode == VS::ENV_BG_CAMERA_FEED) {
        feed = CameraServer::get_singleton()->get_feed_by_id(env->camera_feed_id);
        storage->frame.clear_request = false;
    } else {
//..
case VS::ENV_BG_CAMERA_FEED:
                if (feed.is_valid() && (feed->get_base_width() > 0) && (feed->get_base_height() > 0)) {
                    // copy our camera feed to our background

                    glDisable(GL_BLEND);
                    glDepthMask(GL_FALSE);
                    glDisable(GL_DEPTH_TEST);
                    glDisable(GL_CULL_FACE);

                    storage->shaders.copy.set_conditional(CopyShaderGLES3::USE_DISPLAY_TRANSFORM, true);
                    storage->shaders.copy.set_conditional(CopyShaderGLES3::DISABLE_ALPHA, true);
                    storage->shaders.copy.set_conditional(CopyShaderGLES3::SRGB_TO_LINEAR, true);

                    if (feed->get_datatype() == CameraFeed::FEED_RGB) {
                        RID camera_RGBA = feed->get_texture(CameraServer::FEED_RGBA_IMAGE);

                        VS::get_singleton()->texture_bind(camera_RGBA, 0);
                    } else if (feed->get_datatype() == CameraFeed::FEED_YCBCR) {
                        RID camera_YCbCr = feed->get_texture(CameraServer::FEED_YCBCR_IMAGE);

                        VS::get_singleton()->texture_bind(camera_YCbCr, 0);

                        storage->shaders.copy.set_conditional(CopyShaderGLES3::YCBCR_TO_SRGB, true);

                    } else if (feed->get_datatype() == CameraFeed::FEED_YCBCR_SEP) {
                        RID camera_Y = feed->get_texture(CameraServer::FEED_Y_IMAGE);
                        RID camera_CbCr = feed->get_texture(CameraServer::FEED_CBCR_IMAGE);

                        VS::get_singleton()->texture_bind(camera_Y, 0);
                        VS::get_singleton()->texture_bind(camera_CbCr, 1);

                        storage->shaders.copy.set_conditional(CopyShaderGLES3::SEP_CBCR_TEXTURE, true);
                        storage->shaders.copy.set_conditional(CopyShaderGLES3::YCBCR_TO_SRGB, true);
                    };

                    storage->shaders.copy.bind();
                    storage->shaders.copy.set_uniform(CopyShaderGLES3::DISPLAY_TRANSFORM, feed->get_transform());

                    _copy_screen(true, true);

                    //turn off everything used
                    storage->shaders.copy.set_conditional(CopyShaderGLES3::USE_DISPLAY_TRANSFORM, false);
                    storage->shaders.copy.set_conditional(CopyShaderGLES3::DISABLE_ALPHA, false);
                    storage->shaders.copy.set_conditional(CopyShaderGLES3::SRGB_TO_LINEAR, false);
                    storage->shaders.copy.set_conditional(CopyShaderGLES3::SEP_CBCR_TEXTURE, false);
                    storage->shaders.copy.set_conditional(CopyShaderGLES3::YCBCR_TO_SRGB, false);

                    //restore
                    glEnable(GL_BLEND);
                    glDepthMask(GL_TRUE);
                    glEnable(GL_DEPTH_TEST);
                    glEnable(GL_CULL_FACE);
                } else {
                    // don't have a feed, just show greenscreen :)
                    clear_color = Color(0.0, 1.0, 0.0, 1.0);
                }
                break;
            default: {
//...
//..
}

If I'm right here, this means we should implement our own .submodules/godot/modules/camera/camera_x11.cpp (probably by borrowing some snippets from godot-webcam). @kanetw thoughts?

[KaneTW]

Should work. camera_win.cpp is basically a template, and you can look at _osx.mm, _ios.mm for details.

Alternately you could fix godot-webcam.

[georgewsinger]

Spliced some code into a compiling camera_x11.cpp:

• https://github.com/SimulaVR/godot/commit/83460bd0c08778ea6213eaf96997d18577141208
• https://github.com/SimulaVR/godot/commit/da6a6e7ec0eced301170665a950e4b94518a687f
• https://github.com/SimulaVR/godot/commit/76dbaca3ceb366411ed36d7df5f24b9aec113f29

On startup we now get:

libv4l2 found.
Registered camera UVC Camera (046d:0825) (/dev/video0) with id 1 position 0 at index 0
/dev/video1 is no video capture device.

yet counter-intuitively also get

CameraServer get_feed_count: 0

and the same black screen. This will be completed tomorrow.

[georgewsinger]

Attempting to ./simula_rr_record crashes rr: https://www.wolframcloud.com/obj/39c531e3-75aa-4354-ada1-69bba1f14a41

Makes debugging this much harder.

[georgewsinger]

@KaneTW Using this OS::get_singleton()->delay_usec(10) hack gets activate_feed() working without gdb step throughs:

bool CameraFeedX11::activate_feed() {
    // activate streaming if not already
    if (!device->streaming) {
        OS::get_singleton()->delay_usec(10);
        if (!device->check_device()) {
            return false;
        }
        OS::get_singleton()->delay_usec(10);
        if (!device->request_buffers()) {
            device->close();
            return false;
        }
        OS::get_singleton()->delay_usec(10);
        if (!device->start_streaming(this)) {
            device->cleanup_buffers();
            device->close();
            return false;
        }
    };

    return true;
};

Here's a demo of me using webcam mode with actual VR headset movement: https://www.youtube.com/watch?v=2z2EdsQj3Os&ab_channel=GeorgeSinger. Notice that webcam images are left-right inverted.

[KaneTW]

It's a hack though. There should be a proper solution.

O_NONBLOCK is passed to funcs->open. While maybe not a bad idea, this is definitely what's causing the issue. You need to wait for the IO to finish before using it.