Hi Rich,
You happen to be always a few steps in front of mine. Your work on svelte that I love inspired me in my career and I've tried to be a svelte evangelist whereever I work. Again you surprised me (in a way I'm not surprised you thought about it) with this breadcrumb you left when thinking about svelte-gl. I was creating a repo named svelte-gl to host some on my recent experiments on a new webgl proto I'm working on when I stumble upon this one.
intro
Many people have worked on 3D engines using declarative syntax and reactive features, these initiatives are great !
However I do believe that threejs and babylonjs are big chunky monsters, not built to be lightweight. I wouldn't use them for Times data visualization. The fact that you need at least 600/700KB (2MB for babylon) to show a rotating flat shaded cube makes me sick. The same example is raw webgl takes 10KB. As a frontend developer I can't sleep well knowing about this waste. I tried to lighten threejs on private experiments, I managed to reduce to 220KB by leaving only the core. It's doable but still huge and to make it work (run) it requires some decent refactor. The webgl adoption could be hindered by this problem, personally I don't feel comfortable creating 3d apps for the web (not games) because of this lack of optimization. Don't get me wrong, these two libs a literally bibles to me, I read them everyday, the 3D expertise they compile is amazing. Sadly their architecture design didn't age well. The book keeping, invalidation and reactivity is specific to each construct, there is not system behind it.
solution
Therefore, my point of view is that the community would benefit from a new webgl engine (not lib) built with modern JS standards, with lightweight, decoupled, extensible design. Some patterns allow to only bundle what you use, this part is straight forward. As a pure runtime lib, you could do better than threejs but the syntax wouldn't be the easiest to learn for noobies (dependency injection to name one). There are many lightweight libs already.
That's why I think that the compiler Idea offers more advantages. I agree that it's a challenge to make svelte outputs WebGL. too many parts are adapted to HTML specifics, and the server components, that could help with shader templating (pure strings) have no reactive support. Overall most of the concepts are contained within svelte but not optimally adapted. Bothering the community with support for a 3rd type would be a big risk.
I'm working on two front to make it more chewable considering my modest abilities and resources.
1/ compiler
This is a tough one, I'm browsing the whole svelte code to extract what ever inspiration I can get. By the way good job because the whole compiling thing is very complex. I'm leveraging the same parser, walker, but I'm literally at 1% completion.
My Idea is to do static analysis of the app to extract a tree of possibility with all the static stuff pre-compiled, including 3D content data but also GL api calls, and all the reactive stuff built to run fast, with near 0 overhead.
Some elements would have parameters only if some values are variable.
We can consider a webgl app to be :
a string of function calls.
some shaders that could be templated
state and contexts (like the svelte component context)
string of calls
Depending on the moment (initial, rendering, updating) some parts of the string need to be run. The order of elements is the key part. because some scenes require multiple draw (one for each different object / material). This graph needs to be ultra optimized for runtime performance, by being made using pre-compiled primitives that assemble for cheap (filtering,ordering,looping).
We can think of the vinyl turntable head as an analogy for the control. Because disks are linear but we can take the head and switch track.
detached elements
Some elements are detached because they are triggered by state update. When you move a 3d object, 2 things need to happened, modify the uniform containing the matrix, then render. The render is part of the string (it's the tail end) but the uniform change sometimes takes a different shape than the uniform creation which was part of the initial setup part of the string. In the case of a scene with a single object/material, even with instances or using batching (one draw call for multiple objects/materials), you update nothing between frames rendered, only the single bit that changed needs to be updated. Only perfect optimization can reach the full performance in this case, only invalid data needs to be re-uploaded, that's where svelte is strong. For data viz we would need such a feat to achieve good lightness.
pitfalls
Even if pre-compiled, some things can change drastically so many magic tricks need to exist to avoid limiting content creators.
The context(s) may need to be merged, swapped and assigned during the string execution to allow some specific features. Even the string of element could have some dynamic part, but only if necessary.
shaders
The shader requires some templating, here are the main elements that are needed :
imports (or includes)
values
conditions
loops
Looks familiar right ;-), From my limited knowledge, Svelte template syntax elements can be added to basic glsl parsers. I played with a very simple one that I forked to add the values but using the js string literal syntax :
https://github.com/rbenzazon/glslx-tokenizer
which relies on
https://github.com/rbenzazon/glslx-parser
it's amateurish, but I feel that it's achievable to have a language extension and tooling that goes with it. (lint/format/minimize/bundle,...) There are some alternative that I don't prefer like glsl in js kind. I feel that the talent of writing shader deserve respect and ergonomic so we should have plain glsl files with all the goodies added to it
state / context
I played with some higher order function with context in my svelte runtime POC, seems realistic. Some part of the context is shared along the string, some part is program specific (each program represent a unbatched/uninstanced different object/material). I want to avoid at all cost a virtual dom situation. These value containers should be almost inert. It's only because the chain of function calls is the most efficient that we need them, to be compared to the full state book keeping representation that threejs has.
Main language
I don't think that using XML like syntax has any benefits at all. I do feel the temptation to always go to XML for declarative but in this case I don't get it. If the game engines were written in XML I would maybe think about it, but really here it's not the way. I was putting xml everywhere in the past (ah the good old e4x days) but today I consider it off topic for 3D apps. JS is the main dish. We still need components, but not only one form, we need shader components => JS + shader templates, object components (composition, dependency injection but made simple thanks to the compiler), and many other kinds as well (app/scene,...)
2/runtime
I'm working on a POC using bare svelte 4 to run some simplistic webgl apps compiled by svelte. At this point it leverages a svelte store structure (sry svelte 5 don't want to run for me using vite or rollup $effect is null or something). The point is to map how reactive statements and stores can run a webgl app efficiently. This map would serve as guide to design the compiler and runtime. Webgl is a different animal than HTML, not a lot in common. It's procedural, not declarative, and the GPU state is write only, you can get back some render buffer but it's not the usual flow, and the state is not permanent (I still don't grasp some basic details), but the app should be partially declarative and reactive, this POC has to bridge the two worlds while being the most efficient way to call the gl api as well as low overhead state management. This lib could still be useful in a niche way.
Conclusion
I would love to work on prototyping/modeling and brainstorming with like minded people. I'm not even hoping to be part of a team but even some shared idea would be nice.
Hi Rich, You happen to be always a few steps in front of mine. Your work on svelte that I love inspired me in my career and I've tried to be a svelte evangelist whereever I work. Again you surprised me (in a way I'm not surprised you thought about it) with this breadcrumb you left when thinking about svelte-gl. I was creating a repo named svelte-gl to host some on my recent experiments on a new webgl proto I'm working on when I stumble upon this one.
intro
Many people have worked on 3D engines using declarative syntax and reactive features, these initiatives are great ! However I do believe that threejs and babylonjs are big chunky monsters, not built to be lightweight. I wouldn't use them for Times data visualization. The fact that you need at least 600/700KB (2MB for babylon) to show a rotating flat shaded cube makes me sick. The same example is raw webgl takes 10KB. As a frontend developer I can't sleep well knowing about this waste. I tried to lighten threejs on private experiments, I managed to reduce to 220KB by leaving only the core. It's doable but still huge and to make it work (run) it requires some decent refactor. The webgl adoption could be hindered by this problem, personally I don't feel comfortable creating 3d apps for the web (not games) because of this lack of optimization. Don't get me wrong, these two libs a literally bibles to me, I read them everyday, the 3D expertise they compile is amazing. Sadly their architecture design didn't age well. The book keeping, invalidation and reactivity is specific to each construct, there is not system behind it.
solution
Therefore, my point of view is that the community would benefit from a new webgl engine (not lib) built with modern JS standards, with lightweight, decoupled, extensible design. Some patterns allow to only bundle what you use, this part is straight forward. As a pure runtime lib, you could do better than threejs but the syntax wouldn't be the easiest to learn for noobies (dependency injection to name one). There are many lightweight libs already.
That's why I think that the compiler Idea offers more advantages. I agree that it's a challenge to make svelte outputs WebGL. too many parts are adapted to HTML specifics, and the server components, that could help with shader templating (pure strings) have no reactive support. Overall most of the concepts are contained within svelte but not optimally adapted. Bothering the community with support for a 3rd type would be a big risk.
I'm working on two front to make it more chewable considering my modest abilities and resources.
1/ compiler
This is a tough one, I'm browsing the whole svelte code to extract what ever inspiration I can get. By the way good job because the whole compiling thing is very complex. I'm leveraging the same parser, walker, but I'm literally at 1% completion.
My Idea is to do static analysis of the app to extract a tree of possibility with all the static stuff pre-compiled, including 3D content data but also GL api calls, and all the reactive stuff built to run fast, with near 0 overhead.
Some elements would have parameters only if some values are variable. We can consider a webgl app to be :
string of calls
Depending on the moment (initial, rendering, updating) some parts of the string need to be run. The order of elements is the key part. because some scenes require multiple draw (one for each different object / material). This graph needs to be ultra optimized for runtime performance, by being made using pre-compiled primitives that assemble for cheap (filtering,ordering,looping). We can think of the vinyl turntable head as an analogy for the control. Because disks are linear but we can take the head and switch track.
detached elements
Some elements are detached because they are triggered by state update. When you move a 3d object, 2 things need to happened, modify the uniform containing the matrix, then render. The render is part of the string (it's the tail end) but the uniform change sometimes takes a different shape than the uniform creation which was part of the initial setup part of the string. In the case of a scene with a single object/material, even with instances or using batching (one draw call for multiple objects/materials), you update nothing between frames rendered, only the single bit that changed needs to be updated. Only perfect optimization can reach the full performance in this case, only invalid data needs to be re-uploaded, that's where svelte is strong. For data viz we would need such a feat to achieve good lightness.
pitfalls
Even if pre-compiled, some things can change drastically so many magic tricks need to exist to avoid limiting content creators. The context(s) may need to be merged, swapped and assigned during the string execution to allow some specific features. Even the string of element could have some dynamic part, but only if necessary.
shaders
The shader requires some templating, here are the main elements that are needed :
Looks familiar right ;-), From my limited knowledge, Svelte template syntax elements can be added to basic glsl parsers. I played with a very simple one that I forked to add the values but using the js string literal syntax : https://github.com/rbenzazon/glslx-tokenizer which relies on https://github.com/rbenzazon/glslx-parser it's amateurish, but I feel that it's achievable to have a language extension and tooling that goes with it. (lint/format/minimize/bundle,...) There are some alternative that I don't prefer like glsl in js kind. I feel that the talent of writing shader deserve respect and ergonomic so we should have plain glsl files with all the goodies added to it
state / context
I played with some higher order function with context in my svelte runtime POC, seems realistic. Some part of the context is shared along the string, some part is program specific (each program represent a unbatched/uninstanced different object/material). I want to avoid at all cost a virtual dom situation. These value containers should be almost inert. It's only because the chain of function calls is the most efficient that we need them, to be compared to the full state book keeping representation that threejs has.
Main language
I don't think that using XML like syntax has any benefits at all. I do feel the temptation to always go to XML for declarative but in this case I don't get it. If the game engines were written in XML I would maybe think about it, but really here it's not the way. I was putting xml everywhere in the past (ah the good old e4x days) but today I consider it off topic for 3D apps. JS is the main dish. We still need components, but not only one form, we need shader components => JS + shader templates, object components (composition, dependency injection but made simple thanks to the compiler), and many other kinds as well (app/scene,...)
2/runtime
I'm working on a POC using bare svelte 4 to run some simplistic webgl apps compiled by svelte. At this point it leverages a svelte store structure (sry svelte 5 don't want to run for me using vite or rollup $effect is null or something). The point is to map how reactive statements and stores can run a webgl app efficiently. This map would serve as guide to design the compiler and runtime. Webgl is a different animal than HTML, not a lot in common. It's procedural, not declarative, and the GPU state is write only, you can get back some render buffer but it's not the usual flow, and the state is not permanent (I still don't grasp some basic details), but the app should be partially declarative and reactive, this POC has to bridge the two worlds while being the most efficient way to call the gl api as well as low overhead state management. This lib could still be useful in a niche way.
Conclusion
I would love to work on prototyping/modeling and brainstorming with like minded people. I'm not even hoping to be part of a team but even some shared idea would be nice.
Greetings