Support user-managed paint calls

HexyWitch commented 2 weeks ago

Motivation

Right now yakui really wants to manage its own meshes, textures, and paint calls. It would be useful to be able to hook an external renderer into yakui for custom widgets.

Use cases

Custom text rendering (people get really into having their cool text effects, that don't necessarily belong inside yakui)
Rendering game elements inside UI widgets, particularly 3D models
Widgets with custom shader effects (without having to render to an intermediate texture)

API Proposal (rough)

Seeing yakui as primarily a layout engine and API for building UI, I think the right API for this is an extension API. Rather than packing more types of rendering and widget features into yakui itself.

In PaintDom addition to

    /// Add a mesh to be painted.
    pub fn add_mesh<V, I>(&mut self, mesh: PaintMesh<V, I>)

also support

    /// Adds a user-managed paint call to be painted
    /// This expects the user to handle the paint call in their own renderer
    pub fn add_user_call(&mut self, call_id: UserPaintCallId) {

PaintCall will then be restructured to include user calls:

#[derive(Debug)]
#[non_exhaustive]
#[allow(missing_docs)]
pub struct PaintCall {
    pub call: PaintCallType,
    pub clip: Option<Rect>,
}

/// A user-managed ID for an externally managed paint call
pub type UserPaintCallId = u64;

#[derive(Debug)]
#[allow(missing_docs)]
pub enum PaintCallType {
    Internal {
        vertices: Vec<Vertex>,
        indices: Vec<u16>,
        texture: Option<TextureId>,
        pipeline: Pipeline,
    },
    User(UserPaintCallId),
}

The existing render backends (yakui-vulkan, yakui-wgpu) ignore any User calls for their standard paint methods and just draw internal ones, making the API change non-breaking and opt-in.

An example of how yakui-vulkan would be extended to support user calls can look like:

    /// Paint the specified paint calls using the provided [`VulkanContext`]
    /// The provided closure will be called with the [`VulkanContext`] and the `vk::CommandBuffer`
    /// to allow the user to record additional draw calls.
    /// [...]
    pub unsafe fn paint_with_user_calls(
        &mut self, // [...]
        mut user_call_cb: impl FnMut(&VulkanContext, vk::CommandBuffer, UserPaintCallId),
    );

    /// Render the draw calls we've built up
    ///
    /// The provided closure will be called on any user-managed draw calls. The callback should
    /// return `true` if the pipeline should be re-bound after the user call.
    fn render(
        &self, // [...]
        mut user_call_cb: impl FnMut(&VulkanContext, vk::CommandBuffer, UserPaintCallId) -> bool,
    )

API Usage example

Short example of how the API would be used to render models with a separate renderer, using yakui-vulkan:

impl Widget for RenderModelWidget {
    fn paint(&self, ctx: yakui::widget::PaintContext<'_>) {
        let layout_node = ctx.layout.get(ctx.dom.current()).unwrap();
        let model_renderer = ctx.dom.get_global_or_init(ModelRenderer::default);
        let call_id = model_renderer.paint_ui(&self.model, layout_node.rect());
        ctx.paint.add_user_call(call_id);
    }
}

// Build user-defined draw calls, doing GPU transfers
fn build_draw_calls(paint: &yakui::paint::PaintDom, model_renderer: &mut ModelRenderer) {
    let calls = paint.layers().iter().flat_map(|layer| &layer.calls);
    for call in calls {
        if let yakui::paint::PaintCallType::User(call_id) = call {
            model_renderer.build_buffers(call_id);
        }
    }
}

// Drawing the UI with yakui-vulkan and our own renderer
fn draw_ui(
    yakui_vulkan: &mut YakuiVulkan,
    yakui_vulkan_context: &YakuiVulkanContext,
    paint: &yakui::paint::PaintDom,
    model_renderer: &mut ModelRenderer,
    command_buffer: vk::CommandBuffer,
    resolution: vk::Extent2D,
) {
    build_draw_calls(paint, model_renderer);

    // Set initial viewport, begin render pass
    // Draw the UI
    yakui_vulkan.paint_with_user_calls(
        paint,
        &yakui_vulkan_context,
        command_buffer,
        resolution,
        |ctx, buf, id| {
            // Binds our pipelines and pushes draw calls to the command buffer
            model_renderer.draw_ui(ctx, buf, id);
        },
    );

    yakui_vulkan.transfers_submitted();
}

HexyWitch commented 2 weeks ago

I will happily provide a PR for this if it's a desired change!

HexyWitch commented 1 week ago

This addresses #17 as well but in a more general way that allows other rendering hooks. But the API is a bit more complex than the one proposed in that issue.

msparkles commented 1 week ago

@HexyWitch We've been experimenting on this issue (perhaps not the approach described here, though) for wgpu over at https://github.com/automancy/yakui

Still not in a stage able to be extracted into a PR (we've been ironing out issues...) but worth looking at?

SecondHalfGames / yakui