Experimental Implementation of Raylib Ontop

[VisenDev]

Experimental implementation of a raylib-ontop example. The RaylibBackend.init function is modified to take a parameter which specifies whether the OS window is owner/managed by the end user or by dvui.

If the OS window is not owned by dvui, then dvui does not create the window or use functions like BeginDrawing

See https://github.com/david-vanderson/dvui/issues/89

TODO:

• figure out how to more gracefully control the event system when using the ontop mode. For example, how does dvui tell the user application that a particular mouse click was registered by dvui and does not need to be addressed by the application? Is this important?
• figure out how to copy once-per-frame data like key-down events, such that the user application can also access such data when using dvui

[VisenDev]

So for some reason objects drawn by raylib are always rendering ontop of objects drawn by dvui. Anyone have any ideas as to why this is happening?

[david-vanderson]

When looking at the raylib code, there was stuff about batching drawing. Is all the raylib stuff actually drawn at the point that EndDrawing() is called?

[david-vanderson]

// By default EndDrawing() does this job: draws everything + SwapScreenBuffer() + manage frame timing + PollInputEvents()

This comment in the raylib cheatsheet suggests that might be the case.

[VisenDev]

This comment in the raylib cheatsheet suggests that might be the case.

hmm... if so we may need to rework the dvui rendering function so that it integrates with raylibs batching system. I'm not sure if that will be possible or not

edit: perhaps we render all of our widgets to a rendertexture and then render that texture using ray.DrawTexture?

[david-vanderson]

In EndDrawing() the first thing it does is rlDrawRenderBatchActive(); If you call that after the raylib drawing and before the dvui stuff, does that change anything?

[VisenDev]

In EndDrawing() the first thing it does is rlDrawRenderBatchActive(); If you call that after the raylib drawing and before the dvui stuff, does that change anything?

I'm about to go to dinner. I'll test this out when I get back. Thanks!

[VisenDev]

The above suggest fixed the layering issue.

However, I have discovered a new issue with the raylib backend, memory usage seems to go up dramatically every time you switch themes. This seems to be caused by the creation of new textures. Whenever I swap themes, dvui uses up another ~250mb of ram. Once I do this enough times my application freezes and I have to kill the process

I think the freezing might be from my mac laptop being low on ram/vram - but the memory issues should probably be addressed.

I also tested with the sdl backend and the memory usage stays around a consistent ~30mb in total when swapping themes.

The way raylib loads and destroys textures may need to be reworked. I'm not completely familar with how texture loading backend functionality is used and managed throughout the rest of dvui

[david-vanderson]

I see the memory issues in main, and I also see a slow ramp in memory use even when not doing anything, which probably means we are not freeing something. Just looking at drawClippedTriangles, it looks suspicious that we call rlLoadVertexArray/Buffer/BufferElement and no matching unload.

Yep - that seems to have fixed it for me. I've committed that to main. raylib is now spamming the terminal with INFO messages, so maybe we are still not using these things the right way?

[david-vanderson]

Okay got raylib to stop spamming messages by only calling rlLoadVertexArray() once during init().

[david-vanderson]

Sorry - I forgot to address your TODOs.

For the first, all the dvui.addEvent...() functions return a bool - if true it means that dvui thinks that event is for a dvui floating window, and the application should not process it. I haven't gotten any feedback on this, which either means it works, or that nobody has tried to use it.

I don't quite understand the second. I expect that when using dvui in "ontop" mode, the underlying game/app would already have some kind of event handling system. Can you explain more or give an example here?

[VisenDev]

For the first, all the dvui.addEvent...() functions return a bool - if true it means that dvui thinks that event is for a dvui floating window, and the application should not process it. I haven't gotten any feedback on this, which either means it works, or that nobody has tried to use it.

Okay this is useful, I will think about how to make this data more exposed to the user.

I don't quite understand the second. I expect that when using dvui in "ontop" mode, the underlying game/app would already have some kind of event handling system. Can you explain more or give an example here?

Yes, what I meant was for functions like GetKeyPressed for raylib or SDL_PollEvent - dvui running ontop shouldn't extract all event data internally because that would prevent the application from also using that data for its event processing.

Knowing this, the sdl-ontop example makes more sense - the user polls the events and then sends them to dvui for extra processing. That approach seems to work for SDL, where all events are processed thru a single api.

However, for raylib, the only functions that give you a stream of events in this manner are GetKeyPressed and GetCharPressed.

Perhaps we should implement our own Event type that performs a similar function to SDL_Event but for Raylib. Our event processing function should split into two functions - one function that produces an array of all events, and another function that registers all of those events.

This would allow for end users to determine whether a specific event was used by dvui, as is done in the SDL example. This would help ease the integration of a floating window for example.

            if (try backend.addEvent(&win, event)) {
                // dvui handles this event as it's for a floating window
            } else {
                // dvui doesn't handle this event, send it to the underlying application
            }

(Also - whatever approach ends up being successful, maybe we should standardize it as part of the backend interface so all future backends will also handle ontop applications gracefully)

[david-vanderson]

Ooooo I think I get it - the raylib backend is doing all this work to construct this nice event stream, but maybe the app would like to use it outside of dvui?

I think you are on to something there, but let's keep it in the raylib backend until we have another similar backend. The raylib backend can have lots of functions that are extra on top of the ones needed by dvui.

Maybe the sdl backend way of checking each event is not right for raylib. For a game using dvui for extra floating windows, maybe we simplify the check to something like "at the beginning of this frame, was a dvui floating window focused?" If yes, the application ignores input for that frame. It would have to always send everything to dvui.

Hmmm - I guess it depends on if the app/game is using raylib's event queue or not. Well in the absence of perfect knowledge, we should strive first to fulfill your usecase.

[VisenDev]

I've pretty much done all I want to do with raylib-ontop until #94 and #95 are discussed.

Follow up work:

• If setting the window_owner during initialization is an acceptable way to differentiate ontop and standalone applications - this approach should be standardized across other backends. Variable naming is up for discussion. Perhaps we could use enum{ontop, standalone} instead of enum{dvui, user} for this purpose since it more closely reflects the examples.

EDIT: actually I like enum{ontop, standalone} a lot better, I am going to change this now

[VisenDev]

As to the best way to handle events, I'm really not sure whats gonna be the best way to do that.

Maybe the sdl backend way of checking each event is not right for raylib. For a game using dvui for extra floating windows, maybe we simplify the check to something like "at the beginning of this frame, was a dvui floating window focused?" If yes, the application ignores input for that frame. It would have to always send everything to dvui.

I like this idea though because it simplifies things - I'm gonna have to think about the best way to make this simple and consistent for ontop applications.

Actually I think a really good test bed would be making a theme editor as the raylib-ontop example. I would use the raygui color picker widget along with dvui widgets, which would let me test how I keep raylib and dvui from getting mixed up about event handling.

[david-vanderson]

Fantastic work - thank you!