Pipeline stages confusion

[krOoze]

VK_PIPELINE_STAGE_ALL_GRAPHICS_BIT is specified thus:

VK_PIPELINE_STAGE_ALL_GRAPHICS_BIT specifies the execution of all graphics pipeline stages, and is equivalent to the logical OR of:

• VK_PIPELINE_STAGE_TOP_OF_PIPE_BIT
• VK_PIPELINE_STAGE_DRAW_INDIRECT_BIT
• VK_PIPELINE_STAGE_TASK_SHADER_BIT_NV
• VK_PIPELINE_STAGE_MESH_SHADER_BIT_NV
• VK_PIPELINE_STAGE_VERTEX_INPUT_BIT
• VK_PIPELINE_STAGE_VERTEX_SHADER_BIT
• VK_PIPELINE_STAGE_TESSELLATION_CONTROL_SHADER_BIT
• VK_PIPELINE_STAGE_TESSELLATION_EVALUATION_SHADER_BIT
• VK_PIPELINE_STAGE_GEOMETRY_SHADER_BIT
• VK_PIPELINE_STAGE_FRAGMENT_SHADER_BIT
• VK_PIPELINE_STAGE_EARLY_FRAGMENT_TESTS_BIT
• VK_PIPELINE_STAGE_LATE_FRAGMENT_TESTS_BIT
• VK_PIPELINE_STAGE_COLOR_ATTACHMENT_OUTPUT_BIT
• VK_PIPELINE_STAGE_BOTTOM_OF_PIPE_BIT
• VK_PIPELINE_STAGE_CONDITIONAL_RENDERING_BIT_EXT
• VK_PIPELINE_STAGE_TRANSFORM_FEEDBACK_BIT_EXT
• VK_PIPELINE_STAGE_SHADING_RATE_IMAGE_BIT_NV
• VK_PIPELINE_STAGE_FRAGMENT_DENSITY_PROCESS_BIT_EXT

It includes "graphics pipeline", fragment density, conditional rendering, and "graphics mesh shading pipeline".

Meanwhile it does not include "command processing pipeline", "ray tracing shader pipeline", "ray tracing acceleration structure operations", although those related commands can operate inside a render pass instance.

Meanwhile VK_NV_mesh_shader extensions says:

6. Should VK_SHADER_STAGE_ALL_GRAPHICS be updated to include the new stages? RESOLVED: No. [...]

The author was probably thrown off because VK_SHADER_STAGE_ALL_GRAPHICS definition does say "is equivalent to the logical OR" (but that is meant only figuratively, and in reality it is a single unique bit).

Meanwhile VK_PIPELINE_STAGE_ALL_COMMANDS_BIT says:

• VK_PIPELINE_STAGE_ALL_COMMANDS_BIT is equivalent to the logical OR of every other pipeline stage flag that is supported on the queue it is used with.

It is not clear what "supported on the queue it is used with" means, considering the stages are often used before the queue type is known (render pass creation time; the subpass dependencies).

Also it leads to silly situations, like VK_PIPELINE_STAGE_ALL_COMMANDS_BIT would be allowed in render passes, if the queue family happens to be Graphics-only, but not if the queue can do Compute too.

Meanwhile subpass self-dependencies introduce that srcStages must be logically earlier than dstStages, which does not make clear how VK_PIPELINE_STAGE_ALL_COMMANDS_BIT and VK_SHADER_STAGE_ALL_GRAPHICS_BIT behave there (and if they are even valid). Conceivably the ALL flags may mean "all valid flags in a given context" (given that definition of ALL_GRAPHICS_BIT include flags that, if taken literally, would make it invalid everywhere if extensions are off). But if valid it is not clear what e.g. srcStage = ALL_GRAPHICS_BIT, dstStage = ALL_GRAPHICS_BIT would mean in self-dependency.

Meanwhile what I said above about VK_SHADER_STAGE_ALL_GRAPHICS_BIT that it includes flags that would make its use invalid everywhere in an app with extensions disabled.

Meanwhile subpass implicit dependency is defined thus:

VkSubpassDependency implicitDependency = {
    .srcSubpass = VK_SUBPASS_EXTERNAL;
    .dstSubpass = firstSubpass; // First subpass attachment is used in
    .srcStageMask = VK_PIPELINE_STAGE_TOP_OF_PIPE_BIT;
    .dstStageMask = VK_PIPELINE_STAGE_ALL_COMMANDS_BIT;
[...]
};

which looks like that VK_PIPELINE_STAGE_ALL_COMMANDS_BIT is allowed inside a Render Pass??

Meanwhile some of the stages (e.g. VK_PIPELINE_STAGE_CONDITIONAL_RENDERING_BIT_EXT) are not specified with logical-order relation to VK_PIPELINE_STAGE_*_OF_PIPE_BIT.

[Tobski]

Hi @krOoze oof this is a lot of (admittedly inter-related) issues in one place. I can attempt to answer your questions as best I can, but you may want to follow up with individual extension authors (their github tags are included in the xml specifically so you can tag them on issues like this!). So here's my best attempt:

It includes "graphics pipeline", fragment density, conditional rendering, and "graphics mesh shading pipeline".

The "graphics stages" are kind of orthogonal to the "pipeline order" definitions. Those definitions simply define that if you wait for X then there's an execution dependency on "logically later/earlier stages". And the pipeline definitions are basically just trying to express where that order comes from (though admittedly the section is a little rough around the edges now with all the additions). There might be some documentation cleanup to do in this section.

With that contextual statement out of the way...

Meanwhile it does not include "command processing pipeline", "ray tracing shader pipeline", "ray tracing acceleration structure operations", although those related commands can operate inside a render pass instance.

GIven that there's an open issue on the RT stuff I'm not going to speculate, but there's a lot of uh, mess, between the pipeline bind points and pipeline definitions. This seems to be one of those things. I think it's best to ask the author of VK_NVX_device_generated_commands about whether allowing those commands in a render pass was deliberate in the same manner - perhaps raise a separate issue as you have for the RT extension?

It is not clear what "supported on the queue it is used with" means, considering the stages are often used before the queue type is known (render pass creation time; the subpass dependencies).

Admittedly the language is a tad vague, though the validity is effectively "deferred" until the render pass is used in this case, so it still stands - wording could certainly be a bit cleaner.

Also it leads to silly situations, like VK_PIPELINE_STAGE_ALL_COMMANDS_BIT would be allowed in render passes, if the queue family happens to be Graphics-only, but not if the queue can do Compute too.

Only if you read that literally and ignore the VU statements littered through the spec - statements like this are kind of designed to just suggest a direction without being solid validation. Again, language could be a bit tighter.

Meanwhile subpass self-dependencies introduce that srcStages must be logically earlier than dstStages, which does not make clear how VK_PIPELINE_STAGE_ALL_COMMANDS_BIT and VK_SHADER_STAGE_ALL_GRAPHICS_BIT behave there (and if they are even valid). Conceivably the ALL flags may mean "all valid flags in a given context" (given that definition of ALL_GRAPHICS_BIT include flags that, if taken literally, would make it invalid everywhere if extensions are off). But if valid it is not clear what e.g. srcStage = ALL_GRAPHICS_BIT, dstStage = ALL_GRAPHICS_BIT would mean in self-dependency.

Taken at its literal current meaning (it is the exactly the same as if you specified all those things separately) then no, these are not valid. This is backed up by the validation layers currently. Granted this is different from what we talked about on #1014 so I'll take that into advisement for that :)

Meanwhile what I said above about VK_SHADER_STAGE_ALL_GRAPHICS_BIT that it includes flags that would make its use invalid everywhere in an app with extensions disabled.

Eh this is a quirk (bug!) of the documentation - in the "all extensions" spec it's historically been assumed that all extensions are enabled. So that's more of a general issue, but we should probably fix that here with an addendum of "if the relevant extension is supported".

Meanwhile subpass implicit dependency is defined thus:

I hate the implicit dependencies so much. They have a number of retroactive issues which need fixing, this is one of them.

[krOoze]

Apologies, If I had more time, I would write a shorter Issue :D

And the pipeline definitions are basically just trying to express where that order comes

Yes, except the spec further works with the term of "graphics pipeline" for other purposes than just the stage order. I think I linked to this in a VU restricting stages to only "graphics pipeline stages", except this text does not really clearly disambiguate which stages actually belong to "graphics pipeline".

I think it's best to ask the author of VK_NVX_device_generated_commands about whether allowing those commands in a render pass was deliberate in the same manner

Well, I had no doubt about that one. For one it is marked as Render Pass: Inside. Secondly if it was not allowed inside render pass, that would make the whole extension invalid\pointless.

Admittedly the language is a tad vague, though the validity is effectively "deferred" until the render pass is used in this case, so it still stands - wording could certainly be a bit cleaner.

Perhaps, except where would be the deferred VU? Stage bits are checked at Render Pass creation time, and at point it must be known if ALL_GRAPHICS\ALL_COMMANDS bit is valid to be used there.

Only if you read that literally and ignore the VU statements littered through the spec

Yes, except not:

For any element of pDependencies, if the dstSubpass is not VK_SUBPASS_EXTERNAL, all stage flags included in the dstStageMask member of that dependency must be a pipeline stage supported by the pipeline identified by the pipelineBindPoint member of the destination subpass

Aaaaaand we are back to that initial ALL_COMMANDS definition, so we can judge whether it is a "stage supported by the pipeline identified by GRAPHICS bind point".

This is backed up by the validation layers currently.

Layers should be backed by spec, not other way around. I mean, I just pushed relevant PR there (https://github.com/KhronosGroup/Vulkan-ValidationLayers/pull/1135), which is probably BS. I hope someone will not come back here and interpret spec based on my arbitrary guesswork I used in the Validation Layer implementation. :sweat_smile: