common extensions for wgsl

[mighdoll]

Would you help us understand what a project like luma would need for an extended version of wgsl?

We've started an effort in the wgsl-tooling-wg to define a community extended version of wgsl called wesl. It'd be valuable for us to hear from you what a project like luma needs for flexibly making shaders. Ideally we can make something that supports visgl's needs and help get you more tool support.

Our goal is to 1) add needed features to the language (imports, conditional compilation, simple templates, etc), and 2) enable wgsl libraries (npm, crates.io). There's been some discussion of adding fancier flexibility features like generics, virtual functions, whether we need a hook for code generated wgsl elements, etc. We're prototyping the needed tools (runtime linking/transpiling, IDE language servers, etc.).

Early drafts sketches are here if you want a peek. Discussion's mostly on github and discord.

[ibgreen]

@mighdoll Appreciate the reach-out. Sounds like a great initiative.

We are currently porting our shader module system from GLSL to WGSL and we can collect a list of the issues we run into and share with your group. At the moment it is the additional constraints of the WGSL language that are causing us some trouble, things like:

• No easy ability for a shader module to just add a new attribute - it has to be spliced into the VertexInputs struct
• No easy ability for a shader module to just add a "varying" - it has to be spliced into to the FragmentInputs struct
• No ability to pass samplers as arguments
• Requirement to specify locations explicitly. Something like @location(auto) would be nice. (we already parse the locations from the generated shader source)

We can also see that things like samplers need to be quite specifically typed reducing reusability of code, I assume this is where your templating ideas come in.

Note: As you are probably aware, there is a shader module system for GLSL (glslify from stack.gl) that adds imports but it did not turn out to be very practical for us so we removed support early on. But we will monitor what you doing in this area. If the luma.gl shader modules could be published in this way that could be interesting for us.

As for tooling we do things run-time, so a JS implementation of the preprocessor would be valuable. We'd likely not use a solution that requires adding new tools to our toolchain.

[mighdoll]

@ibgreen great stuff here. We’re thinking about some of these issues but not all, so glad we’ve connected.

we can collect a list of the issues we run into and share with your group.

Thank you. Your issues will be very valuable, I can see.

• No easy ability for a shader module to just add a new attribute - it has to be spliced into the VertexInputs struct

• No easy ability for a shader module to just add a "varying" - it has to be spliced into to the FragmentInputs struct

• No ability to pass samplers as arguments

• Requirement to specify locations explicitly. Something like @location(auto) would be nice. (we already parse the locations from the generated shader source)

We can also see that things like samplers need to be quite specifically typed reducing reusability of code,

We’ll discuss with you how we can address these and each of the issues you raise over the coming weeks.

I assume this is where your templating ideas come in.

texture_external vs texture_2d was one of the cases that drove early thinking, but I’m glad you bring up the specific cases you see.

There’ll always be an escape hatch to insert arbitrary wgsl from TypeScript, though generally we’re hoping to find wgsl extensions that are more portable and feel more integrated.

Note: As you are probably aware, there is a shader module system for GLSL (glslify from stack.gl) that adds imports but it did not turn out to be very practical for us so we removed support early on.

Do you recall what issues you found problematic with glsify? It’d be nice to not recreate those.

But we will monitor what you doing in this area. If the luma.gl shader modules could be published in this way that could be interesting for us.

We’d like to first standardize a way to have wgsl in npm packages. As you think about it, if you discover that you need standard javascript packaging as well that would be noteworthy. (Some shaders require some complicated glue code, but we’re hoping to defer figuring out how to standardize packaging the glue code.)

As for tooling we do things run-time, so a JS implementation of the preprocessor would be valuable. We'd likely not use a solution that requires adding new tools to our toolchain.

Yep, we’ve heard from several projects that runtime transpilation/linking is important.

BTW, are you on the vite bandwagon as well? I ask not because we’d require a particular build tool. But we have discussed the possibility of building some optional niceties (like automatically searching for package files) into a vite/rollup plugin.

[ibgreen]

BTW, are you on the vite bandwagon as well? I ask not because we’d require a particular build tool. But we have discussed the possibility of building some optional niceties (like automatically searching for package files) into a vite/rollup plugin.

Yes at the moment we are on esbuild/vite, however this changes with time, JS tooling is notoriously fast moving. Also, as one example, we also make a point of being able to run code directly in Node.js, without bundlers.

One of the biggest reasons that we ditched glslify was exactly this, it requires a webpack plugin, but as we evolved our tooling we kept hitting cases where it didn't work and it wasn't even remotely worth investing time in porting that plugin to different environments, so we moved all our shaders from dedicated .glsl files (that javascript tools don't understand) to embedded strings in typescript (which works in all environments), and we haven't looked back.

It may work for other projects, but it won't work for us. That's OK though, we can rarely win them all.

[ibgreen]

texture_external vs texture_2d was one of the cases that drove early thinking, but I’m glad you bring up the specific cases you see.

FWIW, I suspect you will want to pay attention to what Steven Wittens is doing in use.gpu. He has long blog posts explaining his struggles with how WGSL is overly restrictive, and I understand that he developed a "WGSL callback system" to work around some of these things, that could perhaps inspire your roadmap.

[mighdoll]

FWIW, I suspect you will want to pay attention to what Steven Wittens is doing

He's a true pioneer, helpfully leaving breadcrumbs for those of us stumbling along after.

One of the biggest reasons that we ditched glslify was exactly this, it requires a webpack plugin,

Thanks. I can hear a siren singing about how plugins make for a smoother DX. Good to keep us from its watery embrace. Perhaps we'll just skip the plugin, but if we make one it needs to be an optional convenience, not a requirement.

The issue I was thinking of comes from the proposed import extension. We want to enable shader composition via something like js imports. Each 'module' of wgsl needs a name so that it can be addressed in an import statement. We'll provide a manual way to name modules. But the glob imports provided by most bundlers make for a convenient way to default all those names.

we moved all our shaders from dedicated .glsl files (that javascript tools don't understand) to embedded strings in typescript (which works in all environments), and we haven't looked back.

We will support embedded strings. That's all a runtime linker can see in any case.

For authoring wgsl, though, it'll be harder to get IDE language servers to be as helpful with embedded strings, so there may be some things to discuss when we get a little further along. Perhaps most of the wgsl could be in files and only the most dynamic generated bits in typescript.. Or perhaps we can find a scheme to help language servers pick through the .ts to find the wgsl modules and their names..

[ibgreen]

The issue I was thinking of comes from the proposed import extension. We want to enable shader composition via something like js imports. Each 'module' of wgsl needs a name so that it can be addressed in an import statement. We'll provide a manual way to name modules. But the glob imports provided by most bundlers make for a convenient way to default all those names.

I think the idea of trying to integrate with the JS import system leads to a lot of complications. Having a mechanism to register a bunch of shader modules with your run time and then have it resolve import statements inside the WGSL is where I would start.

// Imports in JS
import {shaderModule1, ....} from '@luma.gl/shaders'

const transpiler = new ExtendedWGSLTranspiler({modules: [shaderModule1, ...]});

// Imports inside WGSL
const extendedWGSL = /* wgsl */ `import {...} from shadermodule' @vertex main ....`;

const pureWGSL = compiler.transpileShader();

const gpuShader = gpuDevice.createShader(pureWGSL);

For authoring wgsl, though, it'll be harder to get IDE language servers to be as helpful with embedded strings

Yes that is a downside. FWIW, I find the following vscode plugins extremely useful: https://github.com/visgl/luma.gl/blob/master/docs/developer-guide/editing.md

Beautiful syntax highlighting inside all our embedded shader code.

[mighdoll]

The issue I was thinking of comes from the proposed import extension. We want to enable shader composition via something like js imports. Each 'module' of wgsl needs a name so that it can be addressed in an import statement. We'll provide a manual way to name modules. But the glob imports provided by most bundlers make for a convenient way to default all those names.

I think the idea of trying to integrate with the JS import system leads to a lot of complications. Having a mechanism to register a bunch of shader modules with your run time and then have it resolve import statements inside the WGSL is where I would start.

// Imports in JS
import {shaderModule1, ....} from '@luma.gl/shaders'

const transpiler = new ExtendedWGSLTranspiler({modules: [shaderModule1, ...]});

// Imports inside WGSL
const extendedWGSL = /* wgsl */ `import {...} from shadermodule' @vertex main ....`;

const pureWGSL = compiler.transpileShader();

const gpuShader = gpuDevice.createShader(pureWGSL);

Yep, in sync, that was more or less the api. Plus we need to name the modules somehow, either passing a map of named modules, or naming every module inline in the wgsl, or allowing both options.

The api is in flux, but had this to pass the map of modules:

  /** record of file names and wgsl text for modules */
  wgsl?: Record<string, string>;

Something like that (passing an array or a record of modules) will continue to work. After that comes temptation to save some of the manual effort of listing the modules, but I'm taking a way a helpful reminder to make sure the basics work first.

Beautiful syntax highlighting inside all our embedded shader code.

Right that'll work! I was thinking that some of the advanced features will be harder, like tab completing function names and function parameters from functions in other modules.

Yes that is a downside. FWIW, I find the following vscode plugins extremely useful: https://github.com/visgl/luma.gl/blob/master/docs/developer-guide/editing.md

Good links thanks.

[ibgreen]

Plus we need to name the modules somehow, either passing a map of named modules, or naming every module inline in the wgsl

Adding some kind of @module <name> directive in the module source code would be my initial thought.

• Once you add metadata outside of WGSL you start to involve another language (JS, Rust, etc) and your WGSL modules become less portable.
• Your transpiler would extract the necessary information for each module `transpiler.getModuleMetadata(wgsl: string): {name: string; ...}
• You could extend the @module syntax to allow the module to declare additional metadata about itself.

[mighdoll]

Indeed, an earlier prototype linker required that every file of wgsl include a module statement. But requiring module names to be manually entered in every file feels like unnecessary work for most projects that store wgsl in files.

The stakes in the design here are not large. We'll need to support authors who write wgsl in strings and authors that write wgsl in files in any case. We've presumed that authoring wgsl in files would be more popular.

Can you explain further about the virtues you've seen for shaders in strings? You mentioned using node.js w/o a bundler for example. But don't node programs routinely use non .js files as resources? I fear I may have missed an implication that would be obvious to a more experienced node.js programmer..

[ibgreen]

Can you explain further about the virtues you've seen for shaders in strings? You mentioned using node.js w/o a bundler for example. But don't node programs routinely use non .js files as resources? I fear I may have missed an implication that would be obvious to a more experienced node.js programmer..

The only mechanism for importing non-JS code (short of registering custom import hooks which tends to be very brittle) is https://v8.dev/features/import-assertions. It currently only supports JSON files.

I think it is fine if your project support both files and strings and you leave the choice to users.

• I would not be using separate files in my projects,
• and I would expect to run into overwhelming bundling issues if I tried to use published libraries that include shaders in separate files (unless they go through a build step before publishing that inlines the shaders into strings).

[ibgreen]

Indeed, an earlier prototype linker required that every file of wgsl include a module statement. But requiring module names to be manually entered in every file feels like unnecessary work for most projects that store wgsl in files.

If you are writing a module it seems that your design requires you to name it. So the user needs to do that work anyway, whether in wgsl or JS/rust/... Seems to me you are just moving the work outside of the shader code at the cost of less portability.

If there are cases where your design does not require a name (for instance for the main shader importing modules) then of course you don't require a module statement in that WGSL.

[mighdoll]

Thank you for all this. I come away with new appreciation for wgsl in strings, new motivation to not rely on bundlers, ideas for @location(auto), use cases to consider for flexibility in referencing samplers, extending VertexInput and FragmentInput and more.

Looking forward to more issues you discover and sharing prototype tools and designs with you as we have them.

[ibgreen]

Thanks. My final thoughts would be: Nothing wrong with developing a set of tools that allow users to work with WGSL in separate files (bundler plugins and the like), but I would at least initially keep this tooling as an independent offering from your module system. Make it so that they can be used together, but do not build in any special couplings or assumptions (at least not until you have built more understanding of and confidence in the implications)