gfx-rs / wgpu

A cross-platform, safe, pure-Rust graphics API.
https://wgpu.rs
Apache License 2.0
12.43k stars 908 forks source link

Bind groups with binding arrays with high max count in their bind group layout take an extremely long time to allocate descriptor sets for #3729

Open JMS55 opened 1 year ago

JMS55 commented 1 year ago

Description Creating a bind group every frame with 3 binding arrays, each with a maximum count of 50_000 bindings in the bind group layout takes an unreasonable amount of time each frame. Note that I have less than 50 actual bindings in the bind group - the cost is scaling purely based on the bind group layout binding count. Setting it even higher, like 1million, results in insane times like 450ms. image

Repro steps

  1. Create a bind group layout like
    BindGroupLayoutEntry {
    binding: 3,
    visibility: ShaderStages::COMPUTE,
    ty: BindingType::Buffer {
        ty: BufferBindingType::Storage { read_only: true },
        has_dynamic_offset: false,
        min_binding_size: None, // TODO
    },
    count: Some(unsafe { NonZeroU32::new_unchecked(50_000) }),
    },
    BindGroupLayoutEntry {
    binding: 5,
    visibility: ShaderStages::COMPUTE,
    ty: BindingType::Texture {
        sample_type: TextureSampleType::Float { filterable: true },
        view_dimension: TextureViewDimension::D2,
        multisampled: false,
    },
    count: Some(unsafe { NonZeroU32::new_unchecked(50_000) }),
    }
  2. Each frame, create a bind group based on this layout, with a binding array of a small number (<50) of buffers and textures.

Here's a C++ trace captured from Nsight, compressed via 7zip, and then into a zip because github won't let me upload 7z archives. solari2023_04_2512_15_53 (2).zip

You need to be on a GPU that supports ray tracing. You can also run the code directly via cargo run --example solari from here https://github.com/JMS55/bevy/tree/solari, and see the code here https://github.com/JMS55/bevy/tree/solari/crates/bevy_solari/src/scene.

Misc @danchia investigated and found that wgpu/gpu-descriptor was using the FREE_DESCRIPTOR_SET flag, which may be part of the cause https://github.com/zakarumych/gpu-descriptor/blob/413b91137138307afbede5bc918f77bae403493c/gpu-descriptor/src/allocator.rs#L278-L286. I'm not familiar enough with Vulkan to know.

Platform Windows 11, Ryzen 5 2600, AdapterInfo { name: "NVIDIA GeForce RTX 3080", vendor: 4318, device: 8710, device_type: DiscreteGpu, driver: "NVIDIA", driver_info: "528.49", backend: Vulkan }

JMS55 commented 1 year ago

Should probably be added to https://github.com/gfx-rs/wgpu/issues/3637