Define the API in a structured format like vk.xml or dawn.json

[kainino0x]

There are many useful properties of the API that aren't expressible in C, but are still programmatically useful, especially for generating bindings to other languages. Vulkan has vk.xml, for example. It would make sense for us to have such a thing too.

Dawn already generates webgpu.h and webgpu_cpp.h from dawn.json. If we do define a file like this here, we should generate webgpu.h from it too.

See previous discussion on #169, where knowing "in" pointers from "out" pointers would be useful for C# bindings.

[dj2]

For a file like this, would it be exclusively webgpu, or would it also have the ability to contain platform specific information (similar to how dawn.json has the dawn and native flags)?

I could see it either way, it's outside the realm of the common header so it doesn't fit, but on the flip side, there maybe common extensions we work on and sharing the definition would make them more portable.

[kainino0x]

• KN: How important to do this now?
• CF: We (wgpu) don’t need it
• RM: Language bindings will find it very useful
• CF: Have heard vk.xml is very hard to parse, from the person who maintains the Vulkan bindings for Rust
• Write it in another language that bakes down to JSON that automated tools can use?
  • json5, yaml, javascript, typescript?

[kainino0x]

I guess the takeaway is: this would be really nice, we should really do it someday, but it's hard to ever call it urgent. So I don't know when we should work on it actively.

I actually quite like the idea of writing this as a typescript program that outputs JSON. It would provide a lightweight schema though wouldn't be a formal schema for the output JSON, and I don't know how important that will be to tooling (guessing not critical? but there are clever things like https://www.npmjs.com/package/typescript-json-schema).

For a file like this, would it be exclusively webgpu, or would it also have the ability to contain platform specific information (similar to how dawn.json has the dawn and native flags)?

IMO we should maintain a file that doesn't have that, but it should be powerful enough to also define implementation-specific extensions using the same format.

[dj2]

Ok, I'll drop the branch I started (https://github.com/dj2/webgpu-headers/blob/gen/gen_webgpu.rb) where I did it in Ruby as there doesn't seem to be much desire on that front.

The vk.xml file is hard to parse, but a chunk of that is due to the complexities of Vulkan and capabilities and extensions. XML itself is pretty easy to parse and procecess with XPath.

[eliemichel]

FTR, I wish we had something like this when I wrote my shallow C++ wrapper: https://github.com/eliemichel/WebGPU-Cpp

Instead I scrap the webgpu.h header (and the spec's web page for default values...), which is an approach that has the merit to work with both Dawn's custom webgpu.h and wgpu-native's combination of webgpu.h+wgpu.h (thanks for adding WGPU_NULL and other such extra annotations btw).

But I am starting to seriously consider the idea of parsing wgpu-native's headers into a dawn.json-like file, so that I can use Dawn's generator and thus use Dawn's C++ wrapper transparently on both implementations. In a way that'd make dawn.json a de facto structured format.

[dj2]

That's what I did to generate the webgpu.xml file that I was then generating the webgpu header from with gen_webgpu.rb. You can see the script I was using to parse the dawn.json file at https://gist.github.com/dj2/296828495dfd794d401656f2c6e238c8 if it's useful (although it's pretty rough as I didn't intend to commit it anywhere just to create the initial XML file).

[kainino0x]

Another reference that @rajveermalviya recommended taking ideas from: wayland.xml https://gitlab.freedesktop.org/wayland/wayland/-/tree/master/protocol

[kainino0x]

Meeting:

• RM: Have a prototype that uses YAML: https://github.com/rajveermalviya/webgpu-headers-gen
• RM: Tried to remove C-isms. Some things like strings have abstractions. Others like arrays are harder because they expand into two argos or two members
• RM: Struct types: base_in, base_out, extension_in, extension_out, standalone
• KN: Looks nice, we should try to get this in so we can start writing docs!
• AE: Might be nice to abstract the async stuff (callback,userdata pairs) as well so language bindings can structure things differently, but can do this later
• RM: Aside from the TODO doc comments, it currently generates (about?) the same webgpu.h we have now
• RM to open a pull request

[Kangz]

(Note that YAML is well known to look nice but have 1000 corner cases which make it awkward in practice)

Couldn't we have some form of IDL and corresponding parser that outputs various formats needed by the consumers of webgpu.h?

[rajveermalviya]

Previous meetings involved discussions on somehow adapting WebIDL, but the potential complexity of providing IDL for the third-party language bindings generators could be a problem. Unless we also generate a yaml/json/xml from our IDL-based-generator.

[dj2]

I'm not sure I understand the aversion to XML. Yes, it can be a bit more verbose, but is that really an issue here? It's know to handle what we're trying to do (given both vk.xml and wayland.xml exist and do this now). There are tools in every language to handle XML already.

It doesn't end up being too verbose from my prototype (https://github.com/dj2/webgpu-headers/blob/gen/webgpu.xml).

[kainino0x]

Dec 14 meeting:

• CW: Personally no preference for using Dawn's current generator - less work for us to keep it, but we can convert formats
• CW: But think if we generate upstream webgpu.h it should be useful for other things. Ideal if Dawn can use it (by having a second file with Dawn extensions). Once you do that, may as well have multiple files, for example one for native swapchain stuff, one for [some extension], etc.
• CW: Implementations also likely to need to add custom attributes to existing items. Maybe upstream, maybe separate file
• CW: Impls need to be able to add new enums (that don’t collide with core)
• CW: And how many languages we want this to be able to target. Dawn’s targets C (and C++, but not as well as it could). Example futures could be targetable to Rust or Java.
• AE: RM had a proposal for a format in YAML. Concerns?
• CW: Concerned with how YAML has so many ways to write the same thing, bare strings, etc. Kai suggested a YAML linter?? Maybe a schema can check our types.
• RM: Main good thing about YAML is it can be used with JSON schemata. XML could have a schema as well.
• CF: XML harder to write schemata for. Also rust APIs for XML parsing are very annoying.
• RM: Need to discuss whether generator generates single file or separate headers. Wayland for example generates a new header for each new extension, I think. C API some things can’t be extended, e.g. what we use chained structs for.
• CW: Enums and bitmasks can definitely extend. “Classes” can have methods added to them. Structures we extend with chained structs. We could require that all chain types have no conflicts, so that some language bindings could know there won’t be conflicts and generate a flatter API.
• RM: Extensions outside the core header could have different prefixes. Wayland and Vulkan similar.
• CW: Willing to say that if the upstream adds something that conflicts with something Dawn already has, then Dawn has to change to match it
• (discussion of conflict handling…)
• (discussion about whether starting with YAML will be flexible enough for what we need)
• YAML still sounds good, let’s do it and keep hacking on it to add the things we need (and a schema)

[kainino0x]

To add some context from a conversation I had with Corentin shortly before the meeting:

• We can "compile" the YAML to JSON so tools can continue to operate with JSON
• I guessed that a linter could protect us from oddities of YAML. There is indeed a linter with a very nice ruleset (https://yamllint.readthedocs.io/en/stable/rules.html)

[kainino0x]

I might go so far as to say - I'd like to author this in YAML, but downstream projects should consume the JSON instead, and if they add extensions as new files, they may choose to write the extension files as JSON instead of YAML.

[kainino0x]

We have most of this now so I think we can close this as fixed! With enhancements ongoing (like handling implementation-specific extensions)