Making wayrs-client generic over different wayland transports

[madushan1000]

Currently wayrs-client crate only supports unix socket as the transport for wayland protocol.

Right now, I'm trying to modify the crate to be generic over the transport so I can use it with virtio-gpu cross-domain protocol(used to allow programs running on crosvm/qemu guests to open windows in host wayland compositor).

Are you interested in merging the changes here? It will break the public crate api, because things like Connection struct will have to be generic over the transport.

[madushan1000]

I couldn't find good docs on any of this unfortunately. there are some info about the virglrenderer here.

From what I understand, when a egl client on guest wants to allocate a dma-buf, it'll request the buffer via virgl mesa driver. mesa will request a buffer from virtio-gpu kernel driver. kernel driver will request a buffer from the hypervisor(crosvm or qemu). hypervisor will request a buffer form virglrenderer running on the host, and it will finally request a real buffer on the host gpu using host mesa driver. When the client wants to render something onto this buffer, it'll do opengl calls, which will be serialized by virgl mesa driver and sent to the host virglrenderer, which will de-serialize the opengl calls and render stuff on the the allocated buffer. Because the buffer allocation request goes trough two mesa drivers, host and gust has two different "views" of the buffer.

[madushan1000]

I got it to work. I'll update the draft pr with the changes later. Screencast from 2024-03-10 13-31-11.webm

[MaxVerevkin]

Nice!

[madushan1000]

I've hit another snag :/ so far I've been using an ubuntu VM to test the virtio suff, now I have to start developing it on android which is where I intended to use it. It looks like mesa on android doesn't have EGL_KHR_platform_gbm extension. Is it required that I use wayrs-egl(or whatever egl) to use dam-buf with wayland? I don't understand this aspect of wayland very well.

[MaxVerevkin]

It looks like mesa on android doesn't have EGL_KHR_platform_gbm extension.

That's bad but it is not critical. AFAIK EGL_EXT_platform_device can be used instead, but it is more annoying. Does it have it? And what about the other extensions?

Is it required that I use wayrs-egl(or whatever egl) to use dam-buf with wayland?

No, wayrs-egl just glues GBM, EGL and linux-dmabuf-feedback-v1 protocol together:

• You need GBM to get a buffer on linux. There may be other allocators. BTW, android has it, right?
• EGL is a cross platform API to create an OpenGL context, to render to a previously allocated buffer. Again, there may be others.
• You need the dmabuf feedback protocol to create a wl_buffer. This is required to tell the compositor that you have a GPU buffer.

[madushan1000]

My compositor would just pass the buffers rendered by android to the host compositor and hopefully don't have to do any allocations. Android will do all the buffer allocations. Maybe I can get away with just using linux-dmabuf-feedback-v1 protocol and not using GBM and EGL at all.

About GBM on android, they have a system service to allocate graphics buffers called android.hardware.graphics.allocator and there is also a graphics mapper service. I'm not very clear on how they're supposed to work. Need to read more.

Looks like bellow are the available extensions on android. EGL_KHR_platform_gbm, EGL_EXT_image_dma_buf_import_modifiers, and GL_OES_EGL_image are missing.

EGL extensions string:
    EGL_ANDROID_front_buffer_auto_refresh
    EGL_ANDROID_get_native_client_buffer EGL_ANDROID_presentation_time
    EGL_EXT_surface_CTA861_3_metadata EGL_EXT_surface_SMPTE2086_metadata
    EGL_KHR_get_all_proc_addresses EGL_KHR_swap_buffers_with_damage
    EGL_ANDROID_get_frame_timestamps EGL_ANDROID_image_native_buffer
    EGL_ANDROID_native_fence_sync EGL_ANDROID_recordable
    EGL_EXT_buffer_age EGL_EXT_pixel_format_float EGL_KHR_config_attribs
    EGL_KHR_create_context EGL_KHR_create_context_no_error
    EGL_KHR_fence_sync EGL_KHR_gl_colorspace
    EGL_KHR_gl_renderbuffer_image EGL_KHR_gl_texture_2D_image
    EGL_KHR_gl_texture_3D_image EGL_KHR_gl_texture_cubemap_image
    EGL_KHR_image EGL_KHR_image_base EGL_KHR_no_config_context
    EGL_KHR_reusable_sync EGL_KHR_surfaceless_context EGL_KHR_wait_sync

Default display:
EGL API version: 1.4
EGL vendor string: Android
EGL version string: 1.4 Android META-EGL
EGL client APIs: OpenGL_ES
EGL extensions string:
    EGL_ANDROID_front_buffer_auto_refresh
    EGL_ANDROID_get_native_client_buffer EGL_ANDROID_presentation_time
    EGL_EXT_surface_CTA861_3_metadata EGL_EXT_surface_SMPTE2086_metadata
    EGL_KHR_get_all_proc_addresses EGL_KHR_swap_buffers_with_damage
    EGL_ANDROID_get_frame_timestamps EGL_ANDROID_image_native_buffer
    EGL_ANDROID_native_fence_sync EGL_ANDROID_recordable
    EGL_EXT_buffer_age EGL_EXT_pixel_format_float EGL_KHR_config_attribs
    EGL_KHR_create_context EGL_KHR_create_context_no_error
    EGL_KHR_fence_sync EGL_KHR_gl_colorspace
    EGL_KHR_gl_renderbuffer_image EGL_KHR_gl_texture_2D_image
    EGL_KHR_gl_texture_3D_image EGL_KHR_gl_texture_cubemap_image
    EGL_KHR_image EGL_KHR_image_base EGL_KHR_no_config_context
    EGL_KHR_reusable_sync EGL_KHR_surfaceless_context EGL_KHR_wait_sync

[MaxVerevkin]

It doesn't support any EGL_*_platform_*...

EGL_KHR_platform_gbm, EGL_EXT_image_dma_buf_import_modifiers, and GL_OES_EGL_image are missing.

That's bad.

Regarding GBM, is this related?

How do you get buffers from the android compositor? As dmabufs with known params? If so, and if you don't want to render anything additional (postprocessing?), sure, you can skip gbm/egl thing and just forward the buffers.

[madushan1000]

Regarding GBM, is this related?

yes, android supports both minigbm and mesa libgbm. But clients access it via the system graphics allocator service.

Composer clients sends buffers using this interface, it uses a NativeHandle I think that's a file descriptor with some additional metadata(need to read some android platform code to figure out what).

I'll try the pass trough proxy approach first. I might have to do some post processing later(mutter doesn't support the graphics buffer format android uses properly AFAIK, so colors are off). I'll figure it out later.

[madushan1000]

I tried a couple of ways to get async working, but I couldn't. The gpu fd doesn't seem to support epoll well. for read, it works fine. But for writes, it just waits forever. I guess because you can just write to a gpu driver fd whenever you want. so waiting for write availability is sort of meaningless.

Cross domain wayland channel is a GEM object, you can convert it to a fd by using an ioctl. So I tried using two fds for read(gpu fd), and write(cross domain channel fd). But this caused 100% cpu usage.

I think there is a "proper" way to do async stuff in gpu workloads using gpu fences. But I don't understand the documentation on this at all. So I think I'll just keep using the sync wayland client for now.

[MaxVerevkin]

If writing never blocks, then there is no problem at all. async_flush just needs to be modified to first try self.flush(IoMode::NonBlocking) and only in case of WouldBlock await the fd.

I'm pretty sure gpu fences have nothing to do with this.

[MaxVerevkin]

Actually this seems like a nice optimization even for unixstream, so I did that in 8b078fe.

[madushan1000]

Interesting, I will give it another try.

So the trasnport channels virtio-gpu uses to send/receive messages from host wayland are GEM objects. Looks like I can use gpu fences to synchronize access to these in an async way just like with actual graphics buffers. But I probably won't need this.

[madushan1000]

I tried with your latest commits to main, and async works okay now. It still uses three time the cpu as unix sockets implementation (~12% vs %4 cpu). But I think that's probably some other overhead. Nice!

[madushan1000]

I created a separate crate for this transport https://github.com/madushan1000/wayrs-client-transport-virtgpu, the code is not pretty and pipes(for copy pasting will not work). I'll clean things up in the future and add missing functionality.