search

perceptia / skylane

98 stars 2 forks source link

Wayland-egl and mesa support #1

Open elinorbgr opened 7 years ago

elinorbgr commented 7 years ago

Hi I'm the maintainer of wayland-client, a crate of bindings to the wayland C libraries.

I see you've started the road of re-implementing the wayland protocol in 100% rust. Did you find a way to solve the OpenGL question, to go in such a road?

If you are not aware of this issue (which is the reason I gave up re-implementing the protocol), mesa (and I believe nvidia) link to the wayland C libs to implement the wayland-egl.so, which is used by clients to create EGL surfaces to do OpenGL drawing. Here, Mesa expects to be given pointers to objects coming from these C libs (both client-side and server-side) to do its internal magic, and last time I exchanged with the people on #wayland on freenode, they where unanimous that there was to other way to have EGL support than to use the C libs.

If you have a plan to side-step this issue, or something regarding this issue changed that I am not aware of, I'd really like to hear about it!

proton-decay commented 7 years ago

I stumbled across this problem couple days ago. After look at mesa code I think client side is not big problem but server side probably requires changes in mesa.

For clients mesa uses wl_display only to check type of native platform and this step is skipped if the platform is provided via environment variable. Client also has to implement listener for wl_drm in standard way.

Server on the other hand uses extension to mesa which adds global wl_drm implemented in mesa. It cannot be simply passed by. I see here two possible solutions:

Today I asked question about this on Wayland mailing list. Maybe they have better ideas.

I need also to check how guys from Mir did hardware acceleration.

elinorbgr commented 7 years ago

Oh, so it's possible for a client app to use mesa by manually handling drm buffers? Well, that's good to know. Do you have pointers to the relevant APIs to use? I must admit I'm not very knowledgeable regarding the subtleties of mesa. I thought wayland-egl.so was the only public API available for creating EGL surfaces for wayland.

proton-decay commented 7 years ago

I'm not knowledgeable about mesa too. I just tell what I saw after brief (2-3h) look. EGL is library for abstracting creation of windows. For Wayland it only uses yet another protocol extension, which as far as I can tell can/should be re-implemented in Rust. Maybe I didn't notice some big problem, but I'm optimistic (so I currently will focus more in perceptia than skylane).

trissylegs commented 7 years ago

As I posted on Reddit:

It's possible: EGLImage allows you to set the framebuffer using some externally created thing, like another API (CL, VG, Another GL Context, etc). One of those external images are DRM Buffers. So instead of letting OpenGL manage the buffers you: Do the drm magic dance manually via wl_drm, create an Render to an EGLImage, use eglExportDRMImageMESA to get a GEM handle, create PRIME FD for the gem handle, send to server via wl_drm. (I think you can also create the buffer using gbm, import to EGL, and then send the prime fd to the server).

Vulkan is a bit tenative... there's not extension for import/exporting DRM buffers. There is a special Intel function, that's not technically part of an extension but Intel's driver exports it. That can create a VkImage from a PRIME Fd (See VkCube's kms code for a example) Edit: Also not that this won't give you Nvidia support, but probably work with Radeon/AMDGPU (I assume, need to test)

trissylegs commented 7 years ago

Of course all of this is Mesa Specific.

Drakulix commented 7 years ago

This sounds like you would need to rely on a lot of unspecified implementation details, am I right?

elinorbgr commented 7 years ago

Just to be clear, this OpenGL/MESA question is really the only reason I've stuck with bindings rather than a protocol reimpl.

If this issue can be avoided in a reasonable way, then I believe reimplementing the protocol is by far the best course. The wayland C libs are very not rust-friendly in their design, and I had to create a lot of unsafe boilerplate in wayland-rs to make the whole thing safe by rust standards (well, at least it should be if there are no more bugs in my code, but I won't make this bold claim).

So, if this issue can be solved in a reasonable way, both for client-side and server-side, I'd be willing to work with you @darkelement on an unified effort for a rusty wayland implementation, if you are too.

If it cannot be solved, however, EGL with Mesa and Nvidia support is still one of my goals with wayland-rs and smithay, so I'll stick with the C bindings.

trissylegs commented 7 years ago

OH BOY IS IT UNSPECIFIED.

rust-drm currently seems to have docs than I could ever find in one place anywhere (that library needs some cleanup, it's currently really painful cause of lifetimes with Master, also no prime fd support)

last time I saw someone email mesa, we got that wl_drm is a implementation detail of mesa... There's also the zlinux_dmabuf protocoal. Which is public (in wayland-protocols) and can be used with EGLImageKHR and libgbm, you should be able to use a Render node instead of using auth magic.

That does remind me that I've written some rust-ish gbm bindings. I'll have to fetch them from my other laptop though.

Drakulix commented 7 years ago

Really? I would love to see them, cause I am working on some as well right now (https://github.com/Drakulix/gbm.rs). They are feature complete just docs are currently missing (which will take roughly another week). Although our goals are not exactly the same (I am also working on smithay with @vberger) we can maybe share some of the work.

I also got some fleshed out libinput bindings if you are interested in that.

proton-decay commented 7 years ago

Back to the main topic. dmabuf seems to be the way to go, but it has to be implemented in GUI framework, not here. skylane will only provide protocol. I will try to make example to check if it could work.

proton-decay commented 7 years ago

Little update: 2/4 of the plan is done. perceptia is now able to display dmabuf and EGL applications. Now I will clean up the code and next weekend is scheduled for example client running on perceptia and Weston just as @HornetBlack described.

proton-decay commented 7 years ago

I have dmabuf example client working (displaying one frame, then I switched to EGL) but with EGL there are two problems:

So generally I was able to share images. Question is why I needed root privileges in GBM case and then fix my crooked drawings. I have enough OpenGL for now. Here's code, maybe someone will find problem:

https://github.com/perceptia/perceptia/blob/wip_client_exmaples/src/skylane_simple_framework/examples/simple_egl.rs It won't compile, as there is small change in DRM bindings needed.

proton-decay commented 7 years ago

Ok, no root needed. Sometimes it take to see ones code in web browser to find bugs.

trissylegs commented 7 years ago

@darkelement I think you were right about the whole 3 different implementations of drm/gbm.

For drm thers:

For gbm:

I suppose we should at least have:

Things that are needed by kinda-out-scope.

I also think it would good to have base crate (like dri-base or some good name). That contains: basic definitions: Gem handles, perhaps a PRIME/DMABUF type (so GBM, DRM, EGL, Wayland etc can safely share a strongly typed Fd instead of AsRawFd everwhere). And some traits: Like AsGem or AsPrime or FromPrime. Edit: Also, Drm, Gbm, Egl and Vulkan need to share a Fd for the device. (Which logind might trash)

Consideringe at least 1 of each thisn exists it's probably best we figure out which one is the best fit for all. And setting and contribute to 1 library (for well.. the usual reasons).

Perhaps we should have a discussion board or MetaRepo, where we could organise ideas as well.

trissylegs commented 7 years ago

Oh and @darkelement has the rust-kms github organisation. If they're willing to hold the shared repos there (otherwise we can just make another one)

Drakulix commented 7 years ago

@HornetBlack There is also https://github.com/Drakulix/input.rs by myself (on crates.io) (high focus on safe bindings, but close to the real thing and complete) and unpublished, but nearly done, including safe buffer locking and mapping https://github.com/Drakulix/gbm.rs.

I would adjust these crates for what ever trait implementation you need, the more people use it, the better it will be tested.

proton-decay commented 7 years ago

Indeed we need to organize. perceptia has one meta-repo. We can start discussion there and possibly later move elsewhere if you know better discussion board than github.

https://github.com/perceptia/perceptia_dev/issues/2

proton-decay commented 7 years ago

Of course I forgot to gl::UseProgram... Now it (showing first frame) works. So yes, there is support for EGL. Now I'll clean up the code.

And examples do not show up in weston. For perceptia it's enough to commit surface and set it as toplevel. Weston probably has more complicated logic. Have to check.

@vberger, does this satisfy you? If so, maybe you would like to finish the examples as an introduction to skylane?

elinorbgr commented 7 years ago

Hmm, sorry for the lake of activity during these last days, I've been quite busy with personal issues.

@darkelement Just to be sure I understand correctly the situation, please correct anything I got wrong. The OpenGL stack is definitely something I'm not familiar enough with.

There are currently 3 possible ways (I know of) of EGL communication between a client and a server

If I understand correctly, you've re-implemented to handling of wl_drm to match the behavior of mesa and be compatible with it?

Regarding this route, there are two uncertainties in my opinion:

As I said before, I'm quite attached to remaining compatible with both Mesa and Nvidia regarding this.

Taking a step back, there is the general question of the future of OpenGL with wayland. From what I understood, neither GBM nor EGLStreams are considered satisfactory by the community at large. And I've read on #wayland that the future would lie with a 3rd approach to be determined. Would this approach potentially be dmabuf? Or am I just mixing up things I only superficially understand?

Finally, I haven't managed to take the time to look precisely at skylane and how it works. Do you have somewhere a document explaining roughly how the lib works and should be used?

proton-decay commented 7 years ago

I based my examples on examples from weston. There is support for EGL and dmabuf. Last time I heard about EGLStreams there was a fight between nVidia and wayland developers so I didn't consider this option (and I don't know if I have correct hardware. I have only secondary nVidia card and don't know if EGLStreams can handle such case). I didn't hear about third option. I'm curious what Vulcan will bring in future.

My EGL example are compatible only with mesa, but I consider it stable. It's goal was to demonstrate if creating EGL window is possible without libwayland and this goal was achieved. Probably there are better ways than I took, but as I said, I'm not knowledgeable about EGL.

EGL is about drawing with GL (telling GPU what to draw instead of setting pixels on pixmap in RAM) and dmabuf is about sharing buffers between devices (you could use dmabuf to tell compositor how to get images from webcam and draw them on subsurface of your program).

Documentation https://docs.rs/skylane/0.1.2/skylane/index.html https://docs.rs/skylane_protocols/0.1.2/skylane_protocols/ https://docs.rs/skylane_scanner/0.1.2/skylane_scanner/ is sparse and should be extended, but you can get first image. Examples can be found somewhere in links above.

Drakulix commented 7 years ago

@darkelement Is this still your most recent working example?

As far as I understand it, it uses gbm to initialize it's window. What @vberger was talking about, if I got that correctly is initializing an EGLContext with a wayland surface object and have mesa handle the sharing of the framebuffer automatically though the wl_drm object. Also were is your wl_drm implementation in skylane on the other side?

Don't get me wrong, it is awesome that initialization via gbm is actually possible. But it won't be of much use, if currently existing frameworks, like GTK, Qt or even glutin don't actually do it that way. Or am I missing something here?

elinorbgr commented 7 years ago

Don't get me wrong, it is awesome that initialization via gbm is actually possible. But it won't be of much use, if currently existing frameworks, like GTK, Qt or even glutin don't actually do it that way. Or am I missing something here?

Yes, that's my main concern. Currently, the stable API for initializing EGL contextes is the wayland-egl.so library (whose API is very simple). The wayland project actually only defines the API of this lib, and the implementation is provided by the OpenGL implementation (Mesa of Nvidia).

Most wayland clients use this API, so it's important to ensure that server-side, it'd be possible to handle clients using it. We cannot assume we control both ends of the pipe.

proton-decay commented 7 years ago

@Drakulix, no, the latest is on master.

EGL is API abstracting creation of OpenGL context for various platforms like Windows, X11, Wayland or GBM. It asks driver to create buffer on GPU and then communicates with platform specific entity like X Server or Wayland compositor to tell it how to get the buffer. GBM is most generic because it allows creation of offscreen context. In example I create such offscreen buffer and then "manually" tell server to use it.

With this approach skylane clients run on perceptia, weston and whatever and Qt runs on perceptia. Everything runs everywhere. (No, GTK does not use EGL.)

If you wish I can (some day) prepare skylane-egl.so for mesa project.

In perceptia it is implemented here: https://github.com/perceptia/perceptia/blob/master/src/wayland_frontend/protocol/mesa_drm.rs and https://github.com/perceptia/perceptia/blob/master/src/renderer_gl/renderer_gl.rs#L237 (as you can see in comments it's not finished)

Drakulix commented 7 years ago

yes it actually does, when you tell gtk that you need opengl. I will look up the code and point you too it.

Great thing that skylane clients are working everywhere, but the other way around is actually more important for @vberger and me.

And I don't think a skylane-egl.so would solve that issue, as that dll is driver specific. (meaning that would at least not work with nvidia.)

proton-decay commented 7 years ago

but the other way around is actually more important

When I wrote "everything runs everywhere" I meant every skylane client can run on perceptia and every mesa-based compositor and perceptia can handle any mesa-based and skylane client.

And I don't think a skylane-egl.so would solve that issue, as that dll is driver specific.

And I think I was just laughing at you. Not funny... ok...

Drakulix commented 7 years ago

Ah sorry, I misread some parts of that. I will check those examples out soon and definitely try it out. Looks promising.

jplatte commented 7 years ago

I am slightly confused about the direction this discussion has taken. I don't know much about all these C libraries, but to my understanding the latest bits of discussion are around how sustainable the approach taken by skylane is, right?

Isn't the larger issue support for nVidia's implementation of OpenGL though? My current understanding of it is that it's still unclear whether it can be supported at all by skylane. Is that correct @darkelement?

proton-decay commented 7 years ago

Correction: Couple of times I said "skylane has support for mesa". Technically, by design of EGL, that's driver that should provide support for skylane as another platform. Of course, politically, skylane means nothing so it has to manage somehow without such support. I could add support in mesa, but as I proved it is much easier to do it the other way and I actually prefer it this way. Sorry for misleading.

EGLStreams look good at first sight, I don't expect big problems and really would like learn more, but personally I don't care about closed source drivers and I consider other features more important. Beautiful thing in open source is that anyone concerned can check if it is easy to "add support" for nVidia without support from nVidia.

fooishbar commented 7 years ago

Just to clarify a few things.

wl_drm/zwp_linux_dmabuf_v1 etc are a bit of an orthogonal problem. You can use dmabuf to route around the need for EGL, but as you've discovered, you need something to allocate buffers.

GBM can do that for you, but some GBM implementations need to access the DRM 'card' node, which is only accessible to root or those who've specifically authenticated themselves (see wl_drm's authentication dance inside Mesa for how to do this). The solution to this is 'the allocator', which of course doesn't exist yet. ;)

zwp_linux_dmabuf_v1 is currently marked unstable, but really we're now in the phase of waiting for complaints to come along before we mark it stable. Mesa uses it by default for EGL/Wayland now.

EGLStreams are currently only implemented by NVIDIA, and no-one else has shown any enthusiasm for them. I personally believe they are fundamentally not a good match to how Wayland works: you can look at the long threads on wayland-devel@ if you're really keen.

Something unsaid but also important is retaining a single libwayland-client. EGL demands a 'struct wl_display *' when it wants to communicate with Wayland, and will try to call C API on that wl_display. This must be the same display connection as you're using for Rust, with locking between the two implementations so multiple threads don't try to dump data on the socket at the same time, allocate the same object IDs, etc. At this point, the only real answer to anything which requires libwayland types is to rewrite libwayland in Rust, export a completely C-compatible API/ABI as well as idiomatic native Rust, and get everyone to use that. Which, IMO, isn't at all a bad idea.

abooij commented 7 years ago

Something unsaid but also important is retaining a single libwayland-client. EGL demands a 'struct wl_display *' when it wants to communicate with Wayland, and will try to call C API on that wl_display. This must be the same display connection as you're using for Rust, with locking between the two implementations so multiple threads don't try to dump data on the socket at the same time, allocate the same object IDs, etc. At this point, the only real answer to anything which requires libwayland types is to rewrite libwayland in Rust, export a completely C-compatible API/ABI as well as idiomatic native Rust, and get everyone to use that. Which, IMO, isn't at all a bad idea.

For completeness, I have successfully written a C-compatible client-side ABI for sudbury.