Vulkan perf regression: 0.13 to 0.14: 3-4 times higher CPU usage in empty project with DefaultPlugins due to wgpu 0.20

perry-blueberry commented 2 months ago

Bevy version

Original: 0.13
Current: 0.14

Relevant system information

Ubuntu 22.04 X11 AMD Ryzen 7 3700x 8-core processor cargo 1.79.0 (ffa9cf99a 2024-06-03)

bevy_render::renderer: AdapterInfo { name: "NVIDIA GeForce RTX 3070", vendor: 4318, device: 9348, device_type: DiscreteGpu, driver: "NVIDIA", driver_info: "550.90.07", backend: Vulkan } system{name="bevy_diagnostic::system_information_diagnostics_plugin::internal::log_system_info"}: bevy_diagnostic::system_information_diagnostics_plugin::internal: SystemInfo { os: "Linux 22.04 Ubuntu", kernel: "6.5.0-41-generic", cpu: "AMD Ryzen 7 3700X 8-Core Processor", core_count: "8", memory: "31.3 GiB" }

What's performing poorly?

I noticed my fans were spinning up on Bevy 0.14. I went into htop and saw that an app with only DefaultPlugins was using up to 7.5% CPU in Irix Mode Off (with Irix Mode On it was using ~130%). 0.13 on the other hand is only topping out at 2.2% with a similar setup. So 0.14 is using up ~3.5 times as many CPU resources.

Before and After Traces

Tried to zoom in as much as possible on these. I cannot tell much of a difference. The common parts are all very similar. CPU usage with Tracy was much more similar to each other (140% vs 200% with Irix Mode On). Top one is before.

Additional information

I originally ran with the nightly and the "fast compile" settings. The behavior was the same there.

alice-i-cecile commented 2 months ago

Can you bisect this to the winit 0.30 PR? I bet the windowing loop changes affected this.

perry-blueberry commented 2 months ago

Went up to the 0.30 bump directly. The CPU usage is low there as well. Will try a proper bisect from there tomorrow.

alice-i-cecile commented 2 months ago

Okay, very surprising. Thanks for investigating!

perry-blueberry commented 2 months ago

It's this one: https://github.com/bevyengine/bevy/commit/1ea1b76c75bbb62cb50b6be7eab758cd8cfd4906.

(On a sidenote: nice to see that every bisected version compiled and ran without problems)

perry-blueberry commented 2 months ago

Related to this: https://github.com/gfx-rs/wgpu/issues/5756 ?

Wumpf commented 2 months ago

That issue talks specifically about Vulkan perf issues (as it specifially calls out vkQueueSubmit as being slower). Is this the case here as well, i.e. does the perf regression show only with the Vulkan backend? Would be also great to further break down what part of wgpu got slower exactly

perry-blueberry commented 2 months ago

Ran the different versions on Windows as well with Task Manager. With Vulkan on Windows the difference was very noticeable, maybe a 5x difference this time. With DirectX12 it was hard to tell a difference.

(It looked like in WGPU 0.19 the Vulkan implementation was using a lot less resources than DX12 (maybe 1/3), but in 0.20 Vulkan was using a bit more (maybe 2/1))

Of course these measurements aren't very scientific.

jimblandy commented 2 months ago

We would definitely like to address this, but so far in this issue it's hard to discern how we can actually debug this. A reduced test case that we can run locally to see the problem ourselves would make progress a lot easier.

Wumpf commented 1 month ago

fyi: I just closed https://github.com/gfx-rs/wgpu/issues/5756 because for all we know so far it's debug assertions only which can be easily turned off for wgpu

alice-i-cecile commented 1 month ago

@Wumpf is there a way to turn off debug assertions for wgpu specifically, or is that a project-wide setting?

Wumpf commented 1 month ago

as per https://github.com/gfx-rs/wgpu/issues/5756#issuecomment-2233150074

[profile.dev.package.wgpu]
debug-assertions = false
[profile.dev.package.wgpu-hal]
debug-assertions = false
[profile.dev.package.wgpu-core]
debug-assertions = false
[profile.dev.package.wgpu-types]
debug-assertions = false

I'm still curious what caused this in wgpu and if there's something stupid going on there that should be fixed 🤔

Wumpf commented 1 month ago

debug assertion issue on wgpu got re-opened - apparently too many people getting lot-worse-than-to-be-expected perf It's still not 100% clear if debug assertion is the only cause of regression that folks are hitting, but it seems so at this point! We also still don't know which change on wgpu caused this / which assertions caused the slowdown

bevyengine / bevy