Getting this all started

[peabrainiac]

Let's maybe keep the most important points of discussion about how we are planning on doing this here for now instead of on discord, so we can use markdown and this doesn't get buried too quickly. Unless one of you has a better idea, of course.

general vision

Just to be sure we're all on the same page: the goal of this is to build a scratch-like, block-based shader programming environment with similar capabilities as shadertoy, and possibly even hook it up to the shadertoy API later on. right?

The probably simplest / best way to do that would be to fork scratch (repos can be found here) and adapt it to fit our needs, We could in theory also fork blockly - the library scratch uses under the hood - but I feel like that is not only uglier than scratch, but also would be way more work to adapt, since with scratch we already have a full well-designed editor that takes care of having different code spaces (sprites, in our case buffers), a stage on which the project can be seen, controls for it, an ui bar at the top etc., while blockly basically only has the block editor itself.

The first long-term goal should probably be to get an editor to work than can load simple shadertoy shaders from json files like you can get by exporting them there, run them, display their code in the form of blocks, allow you to edit those blocks, and save them to a shadertoy-readable json file again. Starting things like a backend, further shadertoy integration or advanced features now would not make much sense, since they would end up as a waste of time if it later turns out that we can't even get a simple editor to work.

One thing to keep in mind though regarding possible shadertoy integration: while the Shadertoy API is open for anyone to use, I think it can only be used to fetch shaders, not upload them. It also requires us to get an API key, which should probably be kept private, so just accessing shadertoy from within the browser is most likely not an option. So, just keep in mind that even if this whole editor thing works out as planned, we'll still be limited in that regard.

scratch repo structure

Now, regarding what I have found out about the scratch codebase so far:

• scratch-gui: this is probably the most important repository for us for now. It contains/loads the code for the scratch editor as seen here, i.e. without integration into the scratch website, but with all features relevant to the editor intact. It would thus be the entry point for getting a copy of the editor to run locally and making changes to it. Its readme file also describes how to set this and other scratch repositories up locally. Besides that, it contains the code for for example the project viewer gui (i.e. the stage wrapper and controls), managing and switching between sprites, switching between the code, image and sound tabs, the block backpack, the ui bar at the top and stuff like that.
• scratch-blocks: this seems to be (I say seems because I don't quite understand that repo yet) the component for the actual drag-and-drop code editor, i.e. the editor in the middle and the sidebar you can get the blocks from. It seems to contain some sort of type definitions for the different input types scratch blocks can have as well as descriptions of how some certain blocks are supposed to look in the two different layout styles, but not an extensive list of all the scratch blocks there are. It seems to be mostly build around blockly, and also uses a different license and build process than the other repositories because of that.
• scratch-vm: this is where things get really interesting. At first glance it seems to be mostly about running projects and managing their state - which is something we would need to completely replace with our own code anyways, since we want to compile blocks to glsl and run that, not interpret them in some virtual machine - however, it also seems to contain the extensive list of block declarations I was already looking for in scratch-blocks, and all the extension blocks as well. Its readme file also states that it manages the abstract syntax tree for the project, which I too would have expected scratch-blocks to do.
  • scratch-render: this is probably the repo I am the most familiar with, but also one of the ones that are of the least interest to us here. It contains the code for rendering sprites and the pen layer to a webgl canvas, which we of course won't need here. We can probably safely discard this repo entirely and replace it with our own shader renderer.
  • scratch-www: this contains the code for large parts of the actual scratch website, though only the frontend. If we ever want to hook our editor up to some sort of backend and have features like sharing, exploring and liking shaders, we could probably adapt this to do so.
  • scratch-desktop: this basically takes scratch-gui and bundles it together with electron as a desktop app, so if we get this whole thing to work as a web app, we can use this to get a desktop version too without much effort. Something to note though is that it also contains some telemetry which we should probably remove.
  • scratch-parser: this seems to be for loading and validating scratch project files. If we ever want to save projects in a similar format we could maybe try to adapt this, but I think saving them as glsl in the same format shadertoy uses might be the easiest way for now.
  • scratch-l10n: translations and stuff. I don't think we'll need to worry about translating stuff any time soon, but still, might be good to keep in mind.

Those are at least the repositories that I think might be important for our work here; all of the other ones, like scratch-paint, scratch-audio and scratch-svg-renderer can probably be ignored.

licensing

All of those repositories are licensed under the BSD-3-Clause License, except for scratch-blocks, which is licensed under the Apache License 2.0. If I unterstand those correctly, that means we're allowed to modify, redistribute and use the code theoretically even commercially, as long as we keep the original license and copyright notice. However, we can't just continue to use the copyrighted name "scratch" for those modifications; I'm not sure if the derived name ShaderScratch would be ok or if we need to come up with something else, so someone should probably look into that further.

next steps

Based on all of this, I think the next logical steps would be:

1. Fork scratch-gui, scratch-blocks, scratch-vm and scratch-render and see if we can get these to work for each of us locally.
2. Discuss how exactly we want the blocks for standard glsl functions too look like. For example:
   • how are we going to deal with the fact that glsl is strictly typed, while scratch isn't? just show an error message when trying to run a project where the types don't match up, or try to make it impossible to fit those blocks together in the first place?
   • what about the huge variety of constructors glsl has to offer? are we going with tons of different blocks for different functions like vec2(float), vec2(float,float), vec2(vec2), vec3(float) etc., or something simpler, which may though possibly be harder to impose type-checks on?
   • glsl has not only mat2, mat3 and mat4, but also mat2x3 and other asymmetric matrices. how will we support those?
   • what about having custom blocks / functions return things? scratch doesn't have that functionality, but the underlying blockly engine has; could we maybe reenable it somehow? if so, how exactly do we want it to work?
   • what about inout and out parameters? if we want to be compatible with shadertoy we should probably support those (especially since every single main function there already depends on those), but how exactly should that look?
   • how are we going to scope variables? scratch doesn't do that, but glsl does, so we'll have to come up with our own solution
   • what about structs?
3. Once we have that cleared up (or already while discussing that, to try out different things), try to replace the block definitions in scratch-vm and scratch-blocks with our own ones. This will obviously break project execution and not be runnable yet - the goal at this point should just be to have the blocks we want and to be able to edit them exactly in the ways we want, but not in other ways (for example feeding functions wrong types).
4. At the same time, start working on replacing scratch-render with something that can render shadertoy-like shader projects when given those.
5. Modify scratch-vm to compile our blocks to glsl and render those with our new renderer, throwing out most of the original code of the scratch virtual machine. The goal at this point should be to be able to create and render simple single-buffer shaders.
6. Use the code we already have for compiling blocks to glsl to save projects in the json format used by shadertoy, and try to create a transpiler in the other direction too to be able to import projects exported from shadertoy. This will make debugging later on much easier.
7. Start working on more advanced features like multiple buffers, keyboard and mouse inputs etc., and removing parts of the user interface we don't need. Only much, much later on, start working on things like having a glsl tab next to the block view (where currently the costumes tab is in scratch, maybe) or adapting scratch-www for some other backend.

The order of the steps here is of course somewhat arbitrary, and many of them can of course be worked on simultaniously. However, I think aside from such smaller deviations, the order outlined here is what would make the most sense.

To be honest, I kind of doubt we'll ever make it past step 3 - but of course, it doesn't hurt to try. I'm still sort of busy with university right now, but I'll see if I can spend some more time on this during the next semester break. And of course, having this many or even more people potentially working on the project doesn't hurt our chances of actually managing to do this too.

[ahhhh6980]

Well we could have types be a dropdown on a variable block, and have structs be defined as a block holding variable blocks, right? And just program it to define that as a struct rather than a list/array/vector of those values. For mat, we could just have fields the user inputs to, and if the mat size requested doesn't exist just throw an error? I'm still unsure about how to go about the vec functions.

[peabrainiac]

Regarding step 1: after 3 hours of working through weird errors, I've now finally managed to set up and build those four repositories locally. Build instructions are here now, so you'll hopefully have an easier time with this than I just had: https://github.com/ShaderScratch/organisation#setup

[ArvinSKushwaha]

Perhaps we could use vector components and then parse adjacent components as a swizzled vector? This would allow us to retain a non-dynamic vector block.