[Needed Crate] A pure rust SPIRV generator

[Lokathor]

Right now the modern graphics APIs are moving towards using the SPIRV binary format as the default way to upload a shader to the GPU.

In some cases, an alternative format is used and the spirv_cross crate (a C++ wrapper) will convert that SPIRV into the alternative format.

As with all C/C++ wrapping crates, the biggest problem with using the crate is just getting the C part compiled in the first place. For as portable as C the language is, C the build process is one of the least portable things you can find around. When using these things as a crate, people need a pure Rust solution simply because that's the only way to ensure that cargo build will "just work" regardless of platform or local setup. The pure Rust version could provide the exact same abilities as the C version, and it would still be better simply because it would interact better with the rest of cargo's build process.

Javelin is a new project by the gfx-rs team to replace the spirv_cross functionality with a pure Rust version. Amazing, and I support them.

What we need is another project that does the same thing for the first part of the pipeline.

• Shader Language --> SPIRV --> API Shader

Javelin replaces that second arrow in the pipeline with pure Rust. Right now that first arrow, converting some sort of textual shader language into SPIRV, doesn't have a pure Rust answer.

Now there's no single source language. SPIRV is just a format the same as LLVM IR is a format, and potentially anything can produce SPIRV if it wants. There are, however, two main shader languages that people are likely to be familiar with (or be able to find books about):

• GLSL, used by OpenGL
• HLSL, used by DirectX and Unity

The differences between these two languages are very superficial overall. There's automatic converters between the two for simple shader programs, and a small 30 minute tutorial is all that a person who knows one of them would need to learn the other.

So, what we need is: a crate that lets you convert a &str holding a shader program into the correct binary SPIRV data (Vec<u32>).

• There's multiple shader stages in a complete rendering/compute pipeline. All stages must be supported.
• The particular source language that's first supported by the crate is unimportant compared to having this crate at all, so this initial issue does not take a position as to which should be supported first.
• Ideally, both languages would be supported eventually. Perhaps as one unified crate, perhaps as two separate crates. However is determined to be best by the compiler writers.
• Ideally, once the crate has a solid lib form, it will also get a cli front end so that people can just install the cli program once and then they can run that as a stand alone thing. Many programs, perhaps even most programs, don't need to dynamically compile SPIRV at runtime, and a cli utility that can be run during a build script is sufficient for those programs.

[seivan]

One approach is to make static libs of glslang to popular platforms (iOS, macOS, Windows, Android, Linux) and offer Rust wrappers.

Same thing with spirv_cross, which today compiles out of the box for iOS, same can't be said for glslang.

It's more productive to use existing tools and tweak them than to wait until pure Rust alternatives show up.

Looking at Winit, it's still far behind SDL2 in terms of stability and features - so I am thinking the same issues will pop up when waiting for these crates. I also think the teams working on gfx-rs are already spread thin with the addition of wgpu and other tools. It might take a while - but I could be wrong.

[hecrj]

I would be interested in contributing to this project. It has a clear goal and it sounds fun!

It's more productive to use existing tools and tweak them than to wait until pure Rust alternatives show up.

With that mindset, a pure Rust alternative will never happen. Why wait? We can do it ourselves!

I think a pure Rust solution will not only bring a lot of value to the Rust ecosystem itself, but also to other ecosystems as well (like the Web).

[grovesNL]

Also interested in contributing. For what it's worth, I've been prototyping some ideas for a (W)HLSL parser a bit over the past few weeks (primary goal of compiling WHLSL to SPIR-V), but it's not ready for anything to be built on top of it yet.

[kvark]

@Lokathor Thank you for writing this down! We've had quite a bit of discussion on https://github.com/gfx-rs/gfx-rs.github.io/pull/30 , and I'm happy to see the steps you are making to solve this :clap: :medal_sports:

HLSL, used by DirectX and Unity

Also used by PS4, pretty much. HLSL overall is the dominant source language today, according to Google SPIR-V team data.

The differences between these two languages are very superficial overall.

I wouldn't call them superficial. The binding model is different. The semantics of operations is different. Even things like the control flow analysis is different. There are a lot of subtleties. The biggest user of GLSL -> HLSL is probably Angle, but I don't know anyone doing the opposite translation in production.

[Ralith]

It's worth noting that most Vulkan documentation assumes GLSL, too.

[Lokathor]

There are many things wrong with what @seivan said, but I'm just going to tackle one of them that's of the highest concern in case anyone gets the idea to try it:

Simply shipping static libs for various platforms is, at least on Win32, fundamentally opening the door for an unsound compilation. MSVC allows itself the freedom to potentially change the C ABI of static libs the same as we allow the call ABI of "rust-call" functions to potentially change over time. At this time it's next to impossible to ensure, in an automatically checked way, that the version of the C runtime that was used to build the static lib shipped to you is the same as the version of the C runtime that will go with the link.exe invoked during the cargo build call. If the user examines their MSVC installation they can look at files and determine if it's safe, but that's not suitable for a major crate to rely upon.

[AlexEne]

I asked some clarifications on what the status of shader text -> SPIRV -> Other representations is and how all things fit together. So here I will attempt to reproduce what I learned:

There is a project that both of the arrows above require: rspirv. This is not complete and it needs to be used as a basic building library for both generating as SPIRV as well as manipulating SPIRV to generate other things. @kvark Will make a list of things needed so there's a bit better visibility of what's lacking there.

Following that I think there are some next steps in this particular project discussed here, as I see there is a lot of interest shown on this subject:

1. Come up with some high-level design on how everything fits. (including what's needed from rspirv)
2. We do a review of that design and then some sort of git repo can be started.

Between 1 and 2 it looks like features have to be added to rspirv so that's going to be step 1.5 as it looks like it is a dependency of this and the Javelin.

[jaynus]

Can I ask what the actual goal behind this library is? Is this to remove the dependency on spirv_cross (e.g. the C/C++ build dependency?). From what I can gather on this post, that is the stated goal here: can I assume this is correct?

If the above is correct, I would like to preface everything by saying: I think if the only goal of this project is to remove a single C dependency, than this is poor choice to use the WG's time and energy, and the initiative itself is being approached naively.

Currently, as written, in my mind, this post seems to say we as a community wish to:

1. Replace shaderc (glslc?), written by Google, with our own project (This post)
2. Replace Spirv-Cross, by KhronosGroup, with our own project (Javelin)

I ask because this is a much more nuanced endeavour than it seems on the surface, and I believe even more complex than iterated here by @kvark and others. I think some consideration needs to go into existing frameworks, libraries and initiatives on this front as well.

I see rspirv was mentioned, which is a good start. There also exists LLVM-SPIRV - which is an official, from KhronosGroup LLVM backend for SPIRV. This would be a much better approach to Spirv generation than baking a custom one into rust. The Rust ecosystem seems to constantly fall into traps of reinventing the wheel or reimplementing projects in rust, which many times leads to poor implementations to the overall detriment of the community as a whole. I believe this is a case where everyone should take a step back and consider what the goal is here.

Although the concept of a Rust-compiled shader language is neat for an experiment, and in the future could be interesting, it currently would lock out the majority of current graphics engineers and shader designers from the rust gamedev core ecosystem. GLSL and HLSL are the current primary shader languages in the entire graphics industry. There is good reason for that - the problem space and execution space of shaders is drastically limited compared to general purpose computing and languages.

So, taking that into consideration, that means GLSL/HLSL are the best routes to go - but again, each has their own specifics as mentioned here and the choice is actually not so clear cut. Although I understand the desire for a rusty specific solution, I don't necessarily think the community has the resources, time and financial backing currently to keep up with LunarG, KhronosGroup and Google in re-implementing rust reimplemented replacements. I fear the fragmentation of these solutions would overall do more harm than good, as our rust solutions would constantly be lacking and playing catch-up with the "Big" Solutions.

Lastly, is the concept of mobile and web devices, and how much legacy wants to be supported. If that is the case, there are many - many quirks and iterations of previous versions of GLSL/HLSL that will need to be supported. This is, again, a huge overhead.

I'd just like to ask, if the goal is because of build issues, is our time best spent re-implementing these production-ready, enterprise solutions, or perhaps determining the deficiencies in either the Rust or shaderc build pipelines, and perhaps solving those?

[Lokathor]

Not all of this is contained in the initial post, some of this clarification comes from today's WG meeting:

• Removing the dependency on spirv_cross is a related project that's just starting up called Javelin. That's part of gfx-rs and they're all on board with doing that. They're friends of this wg but not actually this wg so we can at best advise them on what we think they should do. They still get to make their own decisions.
• To make Javelin work well, we'll need a currently WIP crate, rspirv to become a lot more "production ready".
• Once rspirv is a good and solid crate, that can programmatically manipulate SPIRV and perform optimizations and translations and such, Javelin can proceed with converting SPIRV to various backend formats.
• At the same time, we'll naturally be in a good position to translate various front end formats into SPIRV at a relatively low cost.
• Exactly which front ends will be developed isn't yet set but I think we (probably) all agree that GLSL or HLSL would be the most practical first goals. That's not to say that other front ends couldn't be made too, just that if we focus on GLSL or HLSL then we can help the most users the fastest.
• Personally, I don't think I'll end up committing any code to this compiler project idea, that's not really where my skills are at, so I don't want to take a position on GLSL vs HLSL. I think that should be left in the hands of whoever is willing to actually do the work. In Discord, @tinaun was very interested in writing a GLSL front end. Someone else might also be interested in an HLSL front end. If they're both working off of rspirv as a target lib I see no reason at all that both front ends can't be developed at the same time.

[AlexEne]

@jaynus Javelin will happen anyway due to unrelated reasons and backed by gfx-rs. They have certain needs that spirv-cross does not satisfy or is unwilling to satisfy so there are other reasons for that work there discussed on the gfx-rs group.

Now, Javelin happening means some other things around manipulating SPIRV data need to also happen or improve (in rspirv).

Given this there's still a gap left in making actual spirv code from something (GLSL/HLSL), and given all the work that needs to happen for Javelin, the remaining work to fill that gap might or might not be a lot. People have responded quite positively to this initiative, however, my position right now is that there is not enough data explained on what this work actually is.

That's the reason why why I asked for a sort of high-level design so we all understand what the space is and what's missing for this to happen. The proposal does match the charter, being a low-level library that enables engines to happen so at least that's a checkbox that got checked. The effort required to replace everything from scratch is quite big and people are aware of this from discussions on at least the discord room, and I am probably one very cautious member on booting up or assuming responsibility here for maintaining existing projects (see my comments on other issues :D)

However, I am yet to see enough data for us to lean towards a decision here, thus I asked for some high-level design + review on that. My bad if it sounded like it will happen anyway. The review's purpose is exactly what the name says, to review, then help us decide how far away this is from happening then either move to the next step or postpone, or something else.

[LukasKalbertodt]

Another aspect, which might serve as additional motivation: if we tackle this issue, we should build a number of crates instead of "just" a shaderc clone. People might want to translate languages other than GLSL/HLSL into SPIRV. For example, the idea of a Rust-like shader language (that is potentially compiled as procedural macro) has come up a number of times (compare glassful). These projects should share as much code as possible.

As consequence, I would like to ask for a nice toolbox for compiling something to SPIRV. Including:

• Backend: some IR --> SPIRV basically. Related projects: rasen and rspirv
• Parsers for GLSL and HLSL: well, just the parsing aspect really.
• Tools for analysis and intermediate processing: of course, some checks in the middle of the compiler are language-dependent, but many others are probably not.

With this, creating a shaderc clone is simply a matter of putting these libraries together.

[Lokathor]

There's also a project for RLSL but that's apparently "totally unusable" at the moment.

[kabergstrom]

@jaynus Javelin will happen anyway due to unrelated reasons and backed by gfx-rs. They have certain needs that spirv-cross does not satisfy or is unwilling to satisfy so there are other reasons for that work there discussed on the gfx-rs group.

@AlexEne Do you have a link to this discussion?

FWIW this seems like a very costly project to me, and at face value I don't really see how a rewrite of both shaderc and spirv-cross can be considered the highest priority for the ecosystem if the value to be gained is primarily a more streamlined build system. I believe they are both multi-man-year efforts before all the quirks are ironed out, especially considering various driver workarounds etc.

shaderc is only needed when compiling source text of shaders and, at least in Amethyst's case, is not required at runtime.

[kvark]

@kabergstrom this not the link you are asking about, but it might be helpful - https://gfx-rs.github.io/2019/07/13/javelin.html

the highest priority for the ecosystem

It's an interesting philosophic question. Can we just write bindings to Unreal, Godot, etc, and call it a day? It's called an "ecosystem" because we have a number of connected things written in Rust. SPIRV-Cross and shaderc are not a part of our ecosystem, they are temporarily necessary evils.

[kabergstrom]

@kabergstrom this not the link you are asking about, but it might be helpful - https://gfx-rs.github.io/2019/07/13/javelin.html

Thank you for the quick reply! Yes I had read this post but I guess I hadn't read the linked issues that closely.

It's an interesting philosophic question. Can we just write bindings to Unreal, Godot, etc, and call it a day? It's called an "ecosystem" because we have a number of connected things written in Rust. SPIRV-Cross and shaderc are not a part of our ecosystem, they are temporarily necessary evils.

Yes, we could write bindings to those if they were actually composable and under a FOSS license. But individual features in the engines are not re-usable by other engines, as so many parts of the engines are very strongly coupled.

For me, it's all about the reusable capabilities of FOSS within gamedev. I don't participate in Rust gamedev projects because I need all game engine code to be written in Rust. It's easier to trust Rust code due to its safety guarantees, its standardized documentation tools and Rust's traits make it easy to loosely couple crates with traits at no runtime cost. But I use lots of C code every day and it works great, as I am sure all of us do.

This means people can build their own engines for their games out of various independent crates, as is often necessary. That's where the value for gamedevs is. Apparently a majority of the most popular indie games use custom engines, for example.

So what's an ecosystem anyway? I think it's about people using other people's code, and organically contributing into that pool of code that people use. Rust's most important features for enabling an ecosystem are its capabilities for instilling trust and making it easier to re-use other people's code with confidence. But we can still use C/C++ codebases as long as there is trust in it, as the interop provably works well.

When it comes to Javelin's question of is it really worth it? Mozilla is free to pour its resources where it pleases, of course, and we are all better off for it, with graceful thanks. gfx-rs is a unique and amazing project. Developing Javelin will splinter the greater gamedev community on shader processing so I assume reasonable effort has gone into communicating this decision with SPIRV-Cross maintainers as well to see what can be remedied? It is backed by Khronos and has a lot of industry powerhouses behind it. VkPipelineCache support may be more important to focus on first if you are concerned about performance issues in games IMO.

As for a shaderc replacement intiative by the gamedev working group, I personally think there are other projects that could provide more value per time unit for those trying to make games or game engines in Rust. shaderc is not ideal, but works fine. I don't want to discourage anyone that really wants to level up their language translation skills of course - go right ahead and make it happen if you want to! I'll try to end this unnecessarily long post here, and I'll try to make it a productive discussion by following up with another post that covers the possibility space for other potential working group projects.

[icefoxen]

Just poking my nose in to say I'm surprised that nobody has mentioned cranelift; its aim is to be a general-purpose compiler backend, and from the sound of it could be useful in either the HLL -> SPIRV or SPIRV -> target language portion depending on whether you used SPIRV as an input langauge or output. Doesn't look like there's been any work on that specifically yet, but afaik it's the current-best project for interchangable compiler-y tools in pure Rust. Might be a useful piece of the puzzle.

[kvark]

@kabergstrom thank you for an extensive reply!

When it comes to Javelin's question of is it really worth it?

I strongly believe that it's critical to gfx-rs long-time strategy. Javelin by itself is not something the WG needs to focus (or even care much), it's just that we need the very same things out of rspirv that other projects do, and that becomes a matter of the WG.

Mozilla is free to pour its resources where it pleases, of course

This has very little to do with Mozilla. There is no "pouring of resources" happening.

Developing Javelin will splinter the greater gamedev community on shader processing.

Only if it turns out to be successful. If we didn't want to bring a better alternative to the status quo, we wouldn't have gfx-portability, wgpu-native, and... well... Rust.

reasonable effort has gone into communicating this decision with SPIRV-Cross maintainers as well to see what can be remedied

The point is to have a clean Rust rewrite. I don't know what to say to SPIRV-Cross maintainers. Some of the issues we experienced are getting addressed, like the removal of STL. It doesn't change the bigger picture though.

It is backed by Khronos and has a lot of industry powerhouses behind it.

It's backed by Valve and sponsored by Valve. It's possible that Google started reworking some parts of it for the WebGPU implementation needs. But if we ever want to be faster and more robust than Chrome/MoltenVK/younameit, we can't use all the same tech. Angle, for example, has been both a blessing and a curse in the browsers world. It's the convenience that buys you in, and then you end up linked and dependent for the rest of the product life, periodically suffering from upstream issues.

[AlexEne]

While the discussion here is interesting, we're slowly drifting into subjects further and further away from the initial proposal, like discussing what an ecosystem is or debating if Javelin (a project unrelated to this wg) is actually worth it. For the issue here, we haven't yet seen what this cost is and what the design on the specific proposed library here actually is.

For this reason, I'd kindly ask you all to focus on the subject discussed here and what I consider a reasonable next step, that is a design of what this SPIRV generator with it's dependencies, missing parts and problems that still need to be solved.

If we're such a high-level design for the SPIRV generator can't be produced, spending our time debating the cost of something that is still not well defined is not a proper use of anyone's time and I will vote us to park this issue until more data is produced that we can reason on.

[icefoxen]

To recount, mainly for the sake of my own memory, the problem we are considering is essentially this:

Shader Language --> SPIRV --> API Shader

This process can be divided two steps, shader language --> SPIRV and SPIRV --> API shader. gfx-hal performs this process with shaderc for the first step and spirv_cross for the second step, and their (currently experimental/in progress) Javelin project is intended to replace that second step. So the question is, what else would we need for that first part. For me, the tools I would want are roughly in order of priority:

• rspirv or something like it, a library that is good at parsing, emitting, and maybe manipulating the binary output format.
• A good way to run whatever test suite or validation exists for SPIRV, so we can easily check output -- WebAssembly has a giant and comprehensive test suite, for example, but it needs some tooling to hook into it effectively. Docs on how the system works and how to use it can make life much better.
• Reference parsers, or at least grammars, for GLSL and/or HLSL. Digging them out of the standards (if they even exist) doesn't sound like much fun, so we might as well make sure it only has to be done once.
• Ideally, a SPIRV backend for cranelift, though that is very much a "maybe it's useful". LLVM might be another way to go but that just trades one big bulky C++ dependency for another.

[LukasKalbertodt]

Just popped up in my feed: glsl-1.0 & glsl-quasiquote-1.0 released. It uses shaderc to do the compilation but it seems to be a full GLSL parser and seems to offer useful functionality related to this issue.

[hadronized]

glsl author here. Yes, glsl can parse GLSL450 formatted shaders (adding support for 460 is just the matter of a single ; for structs IIRC). I’ve been working on it for a while now (the project started as part of cheddar, a super-set of GLSL or some sort).

The current state of glsl is that I actively maintain it and I just, as @LukasKalbertodt stated, released the 1.0 version. For those wondering, glsl-quasiquote is proc-macro parsing Rust code at compile-time and replacing it with the generated AST (so that parsing isn’t done at runtime for constant shaders) and, no, it doesn’t support variable interpolation yet (I need help 😅 ).

glsl also gives you a visitor-like mechanism to perform deep mutation of your AST — I needed that in cheddar to, for instance, replace function calls and identifiers at once without getting mad with destructuring the whole AST.

* Reference parsers, or at least grammars, for GLSL and/or HLSL.  Digging them out of the standards (if they even exist) doesn't sound like much fun, so we might as well make sure it only has to be done once.

That is completely solved by my crate, yes. However, the rest is not: obviously glsl won’t (ever) parse HLSL. The SPIR-V solution (using shaderc) is currently just a workaround.

I hope it helps the discussion.

[icefoxen]

@phaazon Duuuuuuude. :clinking_glasses: Good timing.

[fu5ha]

@icefoxen

To recount, mainly for the sake of my own memory, the problem we are considering is essentially this: Shader Language --> SPIRV --> API Shader

Actually, for this specific issue, the problem we are considering is ONLY the Shader Language --> SPIRV piece, and the only reason that the SPIRV --> API Shader part should even come into the conversation is maybe to discuss tools that could be used in both parts (like rspirv).

[hadronized]

Just so you know, GLSL460 is now supported and is part of glsl-1.0.2. If you need anything else, ping me! I’ll be happy to help and dig any further.

[icefoxen]

Welp, I knew it was only a matter of time before I got distracted by shiny programming language things. So, apparently now I'm playing with a language to compile to SPIR-V: https://hg.sr.ht/~icefox/chrysanthemum

Suggestions welcome, I have no idea what I'm doing.

[dxenonb]

@icefoxen I love it! For what it's worth, I am also working on a functional shader front end: a visual material editor that will compile to (for now) GLSL. It will hopefully target SPIR-V as I learn more.

On the subject:

a design of what this SPIRV generator with it's dependencies, missing parts and problems that still need to be solved.

Is SPIRV actually a good IR for targeting other shading languages? Since it seems there's a fair number of us writing shader frontends, it may be beneficial to create an API that takes something resembling an AST and produces the desired output. It wouldn't need to do optimization (but could) and would just take a program description. A higher level abstraction like this could better capture any metadata needed to produce things other than GLSL/SPIRV.

The result would be an object that describes whether or not each shader stage you supplied was compatible with the target, or if you used any features that could not be emulated. It would contain the source you need for your target API.

// either specify a platform/api or shading language version, depending on this project's scope
let target = Target::... ; 
let r = generator(target, Stages {
    vertex: VertexShader { main: Block(vec![Assignment {/* and so on */}]) },
    pixel: // ...,
    ..Default::default()
});
let glsl_vertex_source = r.unwrap().glsl_source().vertex; // more or less

Just the rough idea, it would need to be thought about (e.g. input definitions). 😄

From the charter:

What are we doing: To improve the experience of people using rust for game development.

This idea doesn't directly help anyone doing game development (get your own SPIRV; provide your own control flow graph/ast thing; both have high bars to entry). But, it would help everyone writing shader frontend libraries in Rust produce usable products faster, which in turn helps people new to game engines or new to Rust get going.

This could be easier to complete than Javelin since it doesn't have to divine anything not included directly in the SPIR-V source. (Or this could just produce SPIR-V, and hand it to Javelin, which still fits the bill for "a pure rust spirv generator").

[icefoxen]

Is SPIRV actually a good IR for targeting other shading languages? Since it seems there's a fair number of us writing shader frontends, it may be beneficial to create an API that takes something resembling an AST and produces the desired output.

My gut response is "oh gods not ANOTHER layer in the shader translation pipeline!" 😋

My more measured response is that this is basically the job of, well, a compiler backend. I haven't gotten around to investigating Cranelift seriously though.

From my (still very incomplete) knowledge of SPIR-V so far, it seems fine as an IR. It's basically a structured list of extended basic blocks, and the instruction set is a SSA form. I don't think there's any real difficulties translating it into a (slow) program in a higher level language.

[mitchmindtree]

I'm so glad to have come across this issue and to see some serious discussion! Thanks for opening this @Lokathor.

Taking into account the existence of glsl, cranelift and rspirv, would an expanded view of

Shader Language --> SPIRV

that looks something like the following be feasible?:

            `glsl`               ???
GLSL &str ---------> GLSL AST --------- 
                                       \
             ???                 ???    v             ???+rspirv
HLSL &str ---------> HLSL AST ---------> cranelift IR ---------> SPIR-V
                                        ^ 
            `rustc`              ???   /
Rust &str ---------> Rust MIR ---------

Intuitively, it seems like having some sort of cranelift IR ---> SPIR-V solution might be a big win by acting as a pure-rust SPIR-V code generator? Then again, I'm not familiar enough with compiler design or cranelift itself to have any idea of how feasible cranelift is for this role. E.g. perhaps cranelift-IR only really makes sense for targeting CPU architectures? How much work would be required to target something designed to run on a GPU like SPIR-V? Does cranelift even provide any benefits or should we just be focused directly on targeting SPIR-V? I'm going to have to do some digging into the cranelift docs I think. In the meantime, @sunfishcode could you possibly comment on the feasibility of something like this?

cc @MaikKlein I would love to hear your thoughts or advice on this issue considering all your experience working on RLSL. Also I realise cranelift wasn't really around when you were first starting the project - do you think it would provide some benefit as an intermediary today?

cc @rukai as I remember you were also very interested in this!

[Lokathor]

Cranelift seems to be using some python stuff to build, and it takes 50s to build in release mode. Do end users need those python tools to use cranelift as a library? How much do we get for spending that extra build time?

Keeping in mind that rspirv is already planning to do optimization passes to turn bad SPIRV into better SPIRV, it seems easiest to simply parse your format, produce some SPIRV of questionable quality, and then hand it over to rspirv to handle the rest. As you say, if we're going to have to translate an AST into something else, why put in a middle step when we could go directly to SPIRV?

Maybe there's a win for RLSL, but GLSL and HLSL don't seem to need cranelift at all. They should each keep their workflow as simple as possible, with as few dependencies as possible.

[est31]

SPIRV is high level compared to x86. There is no need to do register allocation/spilling for example. The only reason to put cranelift into the middle would be for optimizations which don't seem to be implemented in cranelift yet.

[MaikKlein]

Keeping in mind that rspirv is already planning to do optimization passes to turn bad SPIRV into better SPIRV, it seems easiest to simply parse your format, produce some SPIRV of questionable quality, and then hand it over to rspirv to handle the rest. As you say, if we're going to have to translate an AST into something else, why put in a middle step when we could go directly to SPIRV?

That would be amazing, especially if it could recompute the structured control flow. I was already planning on doing something similar.

SPIR-V is pretty easy to generate, but the tooling wasn't there yet. I haven't worked on rlsl for a while so I am not sure how much the situation has improved. For example spirv-val only caught the more obvious misuses. Any misuse usually resulted in a segfault which isn't super fun to debug. That was the reason why I wrote https://github.com/MaikKlein/rspirv-cfg because I spent the majority of my time reading through SPIR-V code.

Additionally structured control flow wasn't specified fully in the spec, and I ran into a couple of odd edge cases.

The SPIR-V spec is fairly nice, and it should be relatively straight forward to generate SPIR-V from glsl.

@mitchmindtree

I already investigated cranelift when I started rlsl. The main reason why I didn't choose it was because it didn't do any optimizations. And back then IIRC optimization was a low priority, not sure what the current state is.

I pretty much need something similar for rlsl, and I was already planning to do a bidirectional llvm spir-v compiler at one point if the official one didn't pan out. Not really sure in what state the compiler is https://github.com/KhronosGroup/SPIRV-LLVM-Translator/issues/30

For rlsl I sort of planned on doing mir -> spirv -> llvmvir -> spirv. I'd consider cranelift if the focus would be on optimizations.

[kazimuth]

Following @jaynus and @seivan -- it seems like there's a gap in the ecosystem that would be fairly simple to fill and could help a lot of different projects: a crate that downloads a prebuilt binary from some url, verifies its hash, and installs it to the target directory for linking.

This would make it easy to include shaderc as a binary dependency, while these other projects gestate (for however many years it will realistically take them to get to production-readiness).

Ideally this crate would use a minimal-dependencies solution for downloading files, like alexcrichton's curl crate / raw winapi, to minimize download and compile times.

To really polish it off, it could include instructions like trust on how to set up CI builds that publish to github releases / bintray. Plus instructions for setting up a cargo feature that switches from downloading a prebuilt crate to compiling from scratch, to support people who don't want to download binaries for whatever reason.

I've contemplated writing a crate like this for my own purposes but haven't gotten around to it yet. Is anyone else interested in working on something like this?

[Lokathor]

On windows there's an unfortunate soundness hole with using static libs because the MSVC compiler allows static lib ABI breaks between versions (just like rustc does).

If there's an ABI mismatch when you link two things together it's UB and you hopefully get a linking error, but you can also end up putting together a binary with silent runtime UB lurking in the program.

I don't know if this is a thing that C compilers also do on other platforms, but it could be a concern on linux and mac and such as well.

[mitchmindtree]

Thanks for clarifying @Lokathor @est31 @MaikKlein!

In that case, from my understanding it seems like the remaining step to close the gap for a very basic, "working" GLSL -> SPIR-V might be some pass that takes the glsl crate's AST representation as an input and uses rspirvs module builder to output a Module that can be used to produce the SPIR-V assembly? I also realise that there's a lot of other important work to be done in rspirv for improving errors and the quality of the SPIRV output as you mentioned @Lokathor! Please forgive my naivety - I'm just trying to get a clearer idea of exactly what's left to close the gap and get some very basic, pure-rust GLSL working.

[Lokathor]

That's the missing step.

Then we put it together in an easy to use crate once the parts are ready.

[Ralith]

It's worth emphasis that good SPIR-V is not nearly so important as any (correct) SPIR-V at all. Demanding applications can use external optimizers and distribute optimized SPIR-V code directly if necessary, but just being able to produce something that works will make life much easier for most purposes, and optimization can be iterated on endlessly.

[MaikKlein]

It's worth emphasis that good SPIR-V is not nearly so important as any (correct) SPIR-V at all. Demanding applications can use external optimizers and distribute optimized SPIR-V code directly if necessary, but just being able to produce something that works will make life much easier for most purposes, and optimization can be iterated on endlessly.

Exactly, I do quite a few passes inside rlsl to make sure that I am producing "correct" spir-v. That is why I was considering to produce "Relaxed SPIR-V" (rspirv :p), which I would produce from rlsl and then have an external lib/tool make everything correct.

I'll just go an mention a few things here:

For example this detects the merge block bb7 correctly, but in SPIR-V every merge block as to be unique

Or you can't really have pointers in SPIR-V. It somewhat got nicer with the variable pointer extension but in rlsl I just optimize them away if I know they are static.

Also functions that return a pointer for example Deref::deref, my current workaround is to just inline those.

Obviously (as I have said above) restructuring the CFG so that you can actually recompute the control flow would be nice. Just look at this cfg, it is just not very fun to recompute structure control flow for this. I have mostly solved this in rlsl though, execpt I have a few violations with complex loops sometimes.

I forgot the name for it but expanding structs to scalars would be nice to have. SPIR-V doesn't allow pointers in structs, but this should be legal if you know the pointer is statically known.

 fn foo(&[u32]){..}

This doesn't work because a slice contains a ptr and a length, and it is illegal to have pointers in structs.

but

fn foo(array: &impl Array<u32>, len: usize){..}

works just fine.

where

impl Array<u32> for [u32; LEN] {..}

This is also the reason why iterators can't be easily expressed in SPIR-V by default.

I could go on but I don't want to make this too long.

Everyone has to solve a subset of these problems if they want to generate SPIR-V. I assume this is why we don't see many other SPIR-V backends. With a relaxed SPIR-V spec, even non compiler people could port their language of choice to SPIR-V in a relatively short amount of time.

And I would love to see more GPU languages besides GLSL and HLSL.

Sorry for the brain dump, hopefully not too offtopic.

[seivan]

Following @jaynus and @seivan -- it seems like there's a gap in the ecosystem that would be fairly simple to fill and could help a lot of different projects: a crate that downloads a prebuilt binary from some url, verifies its hash, and installs it to the target directory for linking.

Yup I am doing that as we speak. Although I agree with the issues @Lokathor brought up - I still think there's potential for this to work as an intermediate solution until a better approach comes up that tackles the same issues.

This would make it easy to include shaderc as a binary dependency, while these other projects gestate (for however many years it will realistically take them to get to production-readiness). Ideally this crate would use a minimal-dependencies solution for downloading files, like alexcrichton's curl crate / raw winapi, to minimize download and compile times.

Correct - that being said, as usual I applaud the people here to taking this on and I too would love to see pure rust tools that can replace those. Dealing with all thee build tools is the lesser known tenth circle of hell, known in ancient Latin++ as CMake.

To really polish it off, it could include instructions like trust on how to set up CI builds that publish to github releases / bintray. Plus instructions for setting up a cargo feature that switches from downloading a prebuilt crate to compiling from scratch, to support people who don't want to download binaries for whatever reason.

Well I'm not planning supporting compiling from scratch, I do allow defining your own bindgen config, formatter, wrapper.h & header(s) and/or static lib as ENV variables - wish feature flags could take arguments!

I've contemplated writing a crate like this for my own purposes but haven't gotten around to it yet. Is anyone else interested in working on something like this?

I'm currently doing it for SDL2 and planning for spriv_cross Will have to eventually take on glslang - which is the hardest of them since it doesn't compile to arm (iOS) out of the box but fram what I can tell, the MoltenVK people have managed to do just that, so it's at least possible.

Sorry for detracting. I'm all for the work being done here, I just can't wait :). Only concern is that you don't miss the biggest gap is converting <human_text> to something gfx can use as a shader - for me anything else is less interesting, but I'm sure you guys see all sorts of potential there which is awesome.

[hadronized]

I update the documentation of glsl. It now contains an example to see how to parse and use it.

I will make small patches releases from now on to add more documentation about transpiling, whenever needed.

[kazimuth]

@seivan cool!

Although I agree with the issues @Lokathor brought up - I still think there's potential for this to work as an intermediate solution until a better approach comes up that tackles the same issues.

I think if you distribute glslang as an executable the ABI becomes a non-issue, is that an option? and just write a rust wrapper that runs the executable

Well I'm not planning supporting compiling from scratch, I do allow defining your own bindgen config, formatter, wrapper.h & header(s) and/or binary as ENV variables - wish feature flags could take arguments!

hm, those options are good, but not including a build-from-source option will likely cause problems down the line -- projects like debian, some enterprise environments, and prickly security people often don't like downloading binaries. plus you don't want to irreparably break everybody's builds if your binary host goes down. i'd strongly recommend including a build-from-source feature flag.

oh wait, you support a custom binary through an ENV variable; that's fine i suppose. i'd still suggest including a build-from-source feature flag just for ease of use purposes. you have to write those build files anyway for making the binaries, it's not much more work to just distribute them as well.

good luck on the project!

[hadronized]

For those interested, I created an IRC channel dedicated to the glsl crate. It’s simply #glsl-rs@irc.freenode.org. If you have any question or recommendation that you would like to ask live instead of using this WG (mostly about usage, I guess, not archictecture, which can belong to the glsl issue tracker), feel free to.

Also, whoever decides to write the crate that will eat glsl strings and output to the IR you’re targetting, I can help with that crate too (i.e. not only mentoring on glsl but also provides patches).

[icefoxen]

Welp, my toy shader language now can produce an actual functioning shader:

I'm probably not going to make it actually useful ever, but if anyone ever wants to do something similar I highly recommend they give it a go. SPIR-V is a pretty nice compiler target, apart from entry point definitions at least; it seems a shame to waste it on junk like GLSL.

[kvark]

@jrmuizel has been working on a high-level representation ("hir") of GLSL, based on @phaazon 's parser. They hooked it up with a basic SPIR-V module builder, using rspirv data representation. It's not useful yet, but I think it's a great start, and we can quickly bring it to a level of being useful. See https://github.com/jrmuizel/glsl-to-spriv . Contributions welcome!

[newpavlov]

What is the status of shaders in WebGPU? Is it still possible we will get a binary SPIR-V-like format? (I really-really hope we'll do...) In that case it would be nice for pipeline to be flexible enough to also eventually include this hypothetical format.

[kvark]

Shaders in WebGPU is a very difficult topic that causes most of the conflict within the working group. No decision has been made yet. Perhaps, in a month from now, at face-to-face meeting we'll see better.

[seivan]

While I know shaderc-rs runs easily on macOS I finally managed to get it to run on iOS as well, now assuming it also runs on Android and Windows out of the box so one might be able to call it cross platform.

It might not be useful for Rust puritans, especially how painful it was to fix for iOS as it required a lot of changes to all its submodules like spirv_tools, shaderc, and glslang but at the end of the day you get both spirv_cross and glslang ready to work with today and that might be useful for some people reading this.

All I really care about is the ability to do <shader_lang> |> spirv |> <platform_shader_lang> with one neat package. In other words, if you write your code in glsl it will work out of the box with gfx-rs on "all" platforms at runtime. No more precompiling to spv or writing build scripts, which is very useful for runtime iteration.

That being said it also takes around a fun year to compile so I do hope we can replace it with https://github.com/jrmuizel/glsl-to-spirv one day.

[parasyte]

Coming back to this conversation, mostly just a quick update of the state of things as I am aware.

The Javelin project has been renamed to naga and it has a frontend for WGSL. Which is WebGPU's WIP shading language specification.

It's all very rough around the edges, still. Most of the WGSL spec is filled with placeholders, and I wasn't able to get naga to compile the simple static shaders in pixels. That said, I did find it promising that I could almost do so without any extra dependencies beyond what is included with wgpu.

[kvark]

@parasyte we are approaching a point where wgpu-rs examples are written in WGSL, see https://github.com/gfx-rs/wgpu-rs/pull/679, so I believe it's going to be usable soon.

[ozkriff]

@kvark @Lokathor naga seems to be getting more mature by the day, so I guess this issue can be closed?

[kvark]

Actually, got ahead of myself here a bit. The issue talks about spirv-cross, but the subject is largely orthogonal to it. If we are considering the need to go from HLSL to SPIR-V or GLSL to SPIR-V, then Naga isn't going to save us right now. Our GLSL frontend is being rewritten, maybe it will be ready soon? https://github.com/gfx-rs/naga/pull/776 At the same time, one can get good results with WGSL -> SPIR-V, but this wasn't included in the issue description.