Split Import Spec: Source code paths vs module paths?

[ncthbrt]

The current import spec is trying to solve two problems at once.

The first is basically you have the concern of where is the source code in this project?

And the second, which is a more logical one, is what symbols are used, and in scope in the context of a particular module?

The first concern can be solved with things like a load statement (or some sort of manifest file listing patterns of source files to include in compilation). This information can be used to trivially bundle a library into a single file by wrapping loads within an explicit module. Thereafter, the physical location of a particular symbol becomes irrelevant.

The second concern is made more complicated by potential language features such as module aliasing/reexports, nested modules, includes, generic modules, etc.

I'd like to propose that we separate these two concerns into two different specifications source code loading and symbol usages. This I believe will allow us to build consensus faster and give us a strong foundation to build additional language features.

[mighdoll]

I agree that import syntax needs room to deal with future namespacing within files, and that there's a conceptual difference between files and wgsl/wesl elements.. (for example javascript import syntax makes this difference explicit with elements before the from and files after.) So I follow that far.

I'm not understanding why we should split things up in the spec though... the user task seems unitary - the programmer trying to choose the things from other files they want to use. I'd imagine a reader coming to the spec would want to read about it on one page, which would be harder if we split things up.

We still need to discuss files, namespacing, elements, etc. And I think you may be onto something about the importance of being clear in discussion about the difference between file paths and module/namespace paths. (When I implemented rust style imports in an earlier version of the wgsl-linker, I tripped over that distinction at first, and might trip again.)

[ncthbrt]

So I've been coming to the conclusion that the WESL config file should list the source files using a sort of "import map" as @stefnotch put it. For example, it might look something like this:

{
    "modules": {
        "utils": "./src/utils/**/*.wesl"
    }
}

This would allow us to omit the file paths from the import spec and instead only have to consider module/namespace paths in the usage/import spec.

This neatly solves the problem, while allowing bundlers to easily produce single files that can be used without additional tooling other than a transpiler.

Web only transpilers could also use this same mechanic to resolve modules on the server.

[ncthbrt]

The above would address issue #25

[mighdoll]

This neatly solves the problem,

What problem does this solve? (aside from #25?). I'd like to make a separate bug for the issue if it's not this one (about splitting up the import spec).

This would allow us to omit the file paths from the import spec and instead only have to consider module/namespace paths in the usage/import spec.

Are you thinking we should allow manual module paths as optional? Or reviving the idea of requiring manually entered module paths in every wesl file? For the former, optional manual module paths has come up but I don't think we've had a focused discussion on the motivations for it. Requiring manual module paths is possible too, we could revisit though we've so far chosen to take advantage of the convenience of file names and file paths, but perhaps there's new motivation for requiring manually entered paths that you discovered in the mod explorations in #20? Anyway, let's make a separate bug to discuss the motivating problem unless #25 already covers it (vs. this spec splitting issue).

[ncthbrt]

Are you thinking we should allow manual module paths as optional? Or reviving the idea of requiring manually entered module paths in every wesl file?

Neither. The import map/manifest would cover this requirement. Another advantage of it is that it would work with ordinary wgsl files

[ncthbrt]

What problem does this solve? (aside from #25?). I'd like to make a separate bug for the issue if it's not this one (about splitting up the import spec).

While #25 hints at part of the issue, the crux of the problem is how to unify the concept of importing and using a collection of symbols which exist as a file vs importing and using a collection of symbols which exist within a file (inline modules/namespaces). Having these two concepts be separate means that implementations have to support two kinds of symbol collections and similarly for users, having to conceptualise and understand the difference between the two adds to the learning curve.

There are also a number of additional reasons why it'd be preferable to opt for logical module paths rather than file paths in the import spec.

An important one is the bundler: using file paths means that the bundler needs to introduce some concept of a virtual file system which is quite a lot of additional complexity. In contrast, using logical module paths makes implementing a bundler very simple. You just place all files into nested inline modules within a single file and you're done. The latter is something I've already implemented.

Another reason why module paths are preferable is that it makes mangling quite easy. You just need to use the fully qualified module name as the input. Using file paths, particularly relative ones, makes this a lot more murky.

Finally, logical module paths grant more flexibility around how one structures code than file paths because modules are relocatable.

[stefnotch]

I did implement file paths in a bundler (and I believe mighdoll did too?) and didn't notice any additional complexity.

The flexibility argument is understandable, the main thing I would miss would be the ability to re-export a module.

[ncthbrt]

@stefnotch what did your output look like? Was it wesl or wgsl?

By bundler I'm referring to a program that takes in a set of wesl files and turns them into a single wesl file while preserving all wesl features and semantics. The difference between a bundled and unbundled file would be transparent to the user

[stefnotch]

@ncthbrt My output was WGSL.

For WESL outputs that can be distributed, one has to think about each package manager. e.g. With npm, it might be valid to do zero transforms, because one can ship arbitrary stuff through npm. e.g. For Cargo, one might want to generate a .rs file along the lines of

use math_utils::code;

fn config() -> &'static WeslConfig {
  ...expose config and expose the math_utils::code...
}

fn code() -> &'static HashMap<&str, Entries> {
    static CODE: OnceLock<HashMap<..,..>> = OnceLock::new();
    CODE.get_or_init(|| {
        let mut m = HashMap::new();
        m.insert("foo.wesl", Entries::Folder(put nested hashmap.here));
        m.insert(bar.wesl", Entries::File("@vertex fn main...");
        m
    })
}

after all, we'd want something that can be plugged right into a Rust build system. If a build script is responsible for this, then it should be able to import and compile the code and its dependencies. Such a Rust file would contain a straightforward imitation of the tree structure of thr actual file system.

[ncthbrt]

Sorry. I think I'm not explaining myself well enough. The bundler is part of the compilation/linking process. It allows for incremental compilation