wesl.toml => wesl.json

[mighdoll]

I propose we store wesl package metadata in json format, since we're likely to want to use some package metadata for linking.

• Parsing toml in a runtime linker is certainly possible, but every byte counts for web libraries.
• JavaScript bundlers can generally import json with no additional plugins.
• json is also probably easier to target for tools, e.g. for a tool that extracts versioning data from cargo/npm and writes it into wesl.json.

[stefnotch]

If we want it to be human editable, then toml is probably the better choice. JSON doesn't support comments.

[k2d222]

I was envisioning that the manifest file would be used just for offline compilation and creating libraries for cargo and npm, not for runtime linking. In the runtime case, one can just pass the metadata as a parameter to the linker.

And side question, do we want to standardize the manifest? It is just an implementation detail. What we care about is the behavior.

Here's the three possible use-cases (in js, but the same applies to rust)

• A user is writing plain old javascript. They don't import any npm packages, they only use their own wesl files. Then when they run the linker, they can just declare the file structure there and pass it to the linker.

  const modules = { foo: 'src/foo.wesl', 'foo::bar': 'src/foo/bar.wesl' };
  for mod in modules {
  const source = await fetch(modules[mod]);
  linker.add_file(source);
  }
  const wgsl = linker.link();

• A user is writing javascript in Vite. They are using wesl libraries via npm. Their shader files are preprocessed by an offline compiler, invoked from a vite-plugin-wesl plugin. The plugin looks at a manifest file, or maybe just the package.json to resolve dependencies. Maybe they also need a runtime pass afterwards, but a js file would have been generated by the vite plugin that contains all the source code and dependencies inlined and the user just has to provide the runtime flags to specialize the shader.

  // vite.config.js
  export default {
  // the plugin handles the imports that end in '.wesl'.
  // The plugin can be given configured here or perhaps in a `weslconfig.json`.
  plugins: [Wesl(options)] 
  };
  
  // shader.js
  import MyWeslShader from './shaders/main.wesl'; // importing runs the offline compilation and returns a class that can specialize the shader.
  let wgsl = MyWeslShader.link({ some_feature_flag: false });

• A user is writing a wesl package to be published to both cargo and npm. They use the wesl-bundler CLI that generates two packages from source files. The CLI can be fully configured with cmdline args or read a wesl.toml config file. The file structure looks like this:

  my-library/
  |- wesl.toml
  |- src/
  |  |- lib.wesl
  |  |- util.wesl
  |- out-cargo/    <= generated by wesl-bundler
  |  |- Cargo.toml
  |  |- src/...
  |- out-npm/      <= generated by wesl-bundler
  |  |- package.json
  |  |- src/...

[stefnotch]

I think we'd want to standardise the manifest and the library formats, if for no reason other than to make it easy for language servers and linkers (plural).

For example, if we have two separate WESL linkers in the JavaScript world, it would be nice if they could consume each other's packages without any extra configuration work. Similar idea applies to language servers being able to assume a standard manifest.

[mighdoll]

Excellent points and nice breakdown @k2d222.

I agree that standardizing the format of the manifest is goodness @stefnotch, and with both of you that json is annoying.

I was thinking of runtime linking and in #21 of the need to communicate package versioning metadata to a runtime linker. A tool will normally be expected to run to produce that metadata (i.e. the vite-plugin-wesl or wesl-bundler from @k2d222's comment).

In that comment, you suggest that the tool ought generate a javascript file rather than json for the web package (and presumably a rust file for the cargo package?). If we do that, that addresses the motivation for this issue in a nicer way. JavaScript is even easier for web projects to consume than json.

Note also we need to consider #22 (web projects that use npm but not vite). For #22, manually producing a JavaScript file (to list dependencies) is just as easy as a json one. (I like the idea of a vite plugin too, but we should take care not to require one.)

So now I guess there's three files we need to standardize. The wesl.toml (which I yesterday thought needed to list package dependencies, but I think now doesn't), the JavaScript file (containing dependencies at least) and the similar Rust file. And on the plus side, none of them are in json format.

Let me know if that sounds right, if so I'll close this issue and open up separate issues for speccing the files.

Also, I think the nice vite example should migrate to the spec site. I'll file an issue for that.

[ncthbrt]

Authoring will still need a manifest file per the latest discussion on discord

[ncthbrt]

Link for reference https://discord.com/channels/1275293995152703488/1275318205530898503/1285198630470549534