search

swaywm / wlroots

A modular Wayland compositor library
https://gitlab.freedesktop.org/wlroots/wlroots/
MIT License
2.15k stars 342 forks source link

Invalid surface format for requested mode with vmwgfx #1791

Open agrajag9 opened 5 years ago

agrajag9 commented 5 years ago

I'm attempting to run sway in a VMware guest, but it refuses to start from a TTY. It works fine(ish) when launched from inside a weston instance. When launched from a TTY it fills my journal and stderr with the following:

[drm:vmw_kms_new_framebuffer [vmwgfx]] *ERROR* Invalid surface format for requested mode.
[drm:vmw_kms_fb_create [vmwgfx]] *ERROR* failed to create vmw_framebuffer: -22

Hypervisor is VMware Workstation 12 on Windows 10

Config, drm_info, and debug log available here: https://gist.github.com/agrajag9/22fccb2fd5fb4e8d028932fd9c78b7fd

For what it's worth, there's a lot of this in the logs:

2019-08-13 19:12:15 - [wlroots-0.6.0/backend/drm/drm.c:853] Virtual-1: Retrying pageflip
2019-08-13 19:12:15 - [wlroots-0.6.0/backend/drm/drm.c:376] Starting renderer on output 'Virtual-1'
2019-08-13 19:12:15 - [wlroots-0.6.0/backend/drm/util.c:202] Unable to add DRM framebuffer: Invalid argument
2019-08-13 19:12:15 - [wlroots-0.6.0/backend/drm/atomic.c:54] Virtual-1: Atomic commit failed (modeset): Invalid argument

wlroots has migrated to gitlab.freedesktop.org. This issue has been moved to:

https://gitlab.freedesktop.org/wlroots/wlroots/-/issues/1791

ascent12 commented 5 years ago

I've quickly glanced at the kernel code for this, which I am definitely not an expert on. There seems to be something called a struct vmw_surface, which I assume represents the window that we get displayed on. If our framebuffer format doesn't match that exactly, then it fails.

We only support XRGB8888 for the primary plane at the moment. Maybe the surface format is ARGB8888 or one of the 16-bit formats (XRGB1555/RGB565).

Does vmware have an option to change the colour it's trying to use? It's not a piece of software I've used before.

agrajag9 commented 5 years ago

I don't personally know, but I'm happy to run some tests. I do see in drm_info that the even-numbered Planes support XRGB8888, but the odd-numbered Planes do not.

ascent12 commented 5 years ago

Those are primary vs cursor planes. It's pretty typical for them to support different things, with cursors only support ARGB8888.

agrajag9 commented 5 years ago

Is there an ETA on when you may support ARGB8888 in the primay plane? From the Wayland Protocol Specification (https://wayland.freedesktop.org/docs/html/apa.html), under wl_shm::format - pixel formats:

This describes the memory layout of an individual pixel.

All renderers should support argb8888 and xrgb8888 but any other formats are optional and may not be supported by the particular renderer in use.

emersion commented 5 years ago

We don't do ETAs.

From the Wayland Protocol Specification

This is for buffers coming from clients, so it's unrelated. We already support ARGB8888 client buffers of course.

emersion commented 5 years ago

We already have the logic to select XRGB/ARGB: https://github.com/swaywm/wlroots/blob/master/backend/drm/drm.c#L102

Something is probably going wrong there.

agrajag9 commented 5 years ago

Again, I'm happy to compile some debug patches in VMware to try and figure out what's happening.

tom-pryor commented 4 years ago

Found this issue via google after encountering the same problem. Managed to fix it by enabling the "accelerate 3d graphics" option in the VM settings.