Off screen rendering

[PooyaEimandar]

Hello Thank you for contributing to this amazing project. I find out that tao uses gtk and with gtk, we can use the following C codes in order to render the whole window into the pixels.

void webview_screenshot(struct webview *w) 
{
    GdkWindow* window;
    GdkPixbuf* buffer;
    gint x, y, width, height;

    window = gtk_widget_get_window(GTK_WIDGET(w->priv.webview));
    gdk_window_get_geometry (window, &x, &y, &width, &height);
    buffer = gdk_pixbuf_get_from_window (window, x, y, width, height);

    //test
    printf("w=%d h=%d \r\n", width, height);
    gdk_pixbuf_savev(buffer, "/path/1.jpg", "jpeg", NULL, NULL, NULL);
}

This approach can help us to enable off-screen rendering for wry (issue #391) feature for wry and then we can use wry in the field of 3D applications & games. (e.g html user interface in game ).

I could not find any rust function related to gdk_pixbuf_get_from_window am I miss something?

[wusyong]

Most gtk-rs crate documents have search alias that you can just type with c functions. Here's the one I got from gdk: https://docs.rs/gdk/latest/gdk/prelude/trait.WindowExtManual.html#tymethod.pixbuf

[blaind]

Did some experimentation on this, and opened the pull request for easier discussion (with code).

Seems like this would be doable at least for Linux, need to investigate other platforms.

https://github.com/blaind/wry/blob/headless/examples/headless.rs#L111 is an example to use together with the pull req, although the Event is currently a wrong one and only first two frames are being sent.

[wusyong]

@blaind Thanks for your attempt. When I was saying this need to do it yourself with tao window creation, I mean everyone can retrieve a raw handle (in this case gtk_window). You don't need to implement into the event loop. If you are only interested in off screen rendering on webview, let's move back to wry. I might have some results for you.

[blaind]

@blaind Thanks for your attempt. When I was saying this need to do it yourself with tao window creation, I mean everyone can retrieve a raw handle (in this case gtk_window). You don't need to implement into the event loop. If you are only interested in off screen rendering on webview, let's move back to wry. I might have some results for you.

Allright, thank you for the information. I did a new iteration by implementing a pub fn read_texture(&self, buffer: &mut [u8]) -> Result<(), ExternalError> into platform_impl/linux/window.rs, and calling that from inside event_loop with ControlFlow::WaitUntil.

It also seems that on Linux the gtk::OffscreenWindow could be used instead of gtk::ApplicationWindow for headless rendering. Got the headless mode initializing and providing pixbuffer, but not rendering. Will continue investigating. edit: got it working

[blaind]

I've been PoC'ing with a webview for Bevy 3D engine - repo can be found from here: https://github.com/blaind/bevy_webview/

The underlying logic is heavily based on wry, but without tao-window abstractions (see https://github.com/blaind/bevy_webview/blob/main/crates/headless_webview_engine/src/platform_impl/webkitgtk/mod.rs). Also extracted a "base" crate with traits (see https://github.com/blaind/bevy_webview/tree/main/crates/headless_webview).

For headless operation, there was quite a bit of extra to implement on top of wry-logic. I wonder if there any long-term chance for getting this functionality (with similar or different design?) to be merged into wry?

[wusyong]

I think we can provide another method in WindowBuilderExtUnix trait for this kind of attribute. Did you test it before that it's okay to replace current ApplicationWindow to OffscreenWindow. We use this windows simply because of adding menu. If that's working fine, then it's good to me. If not, then the worst case scenario would just need to document that menu won't work if this attribute is set.