WebGL renderer?

[adroitwhiz]

EDIT: Tracking progress:

• [x] Bitmap + vector sprites
• [x] Speech bubbles
• [x] Pen layer
• [x] Sprite is touching mouse
• [x] Sprite is touching sprite
• [x] Effects
• [x] Layer ordering
• [x] Rotation style
• [x] Convex hull generation / "if on edge, bounce"
• [x] Sprite is touching color
• [x] Color is touching color
• [x] Resizable stage
• [ ] Subpixel positioning of SVG viewboxes

Incorporating WebGL in some form is necessary to implement graphic effects at a reasonable speed.

I was thinking of rewriting the renderer to use WebGL at its core, similar to scratch-render. This could potentially take care of a number of currently missing blocks in one fell swoop:

• Effect blocks
• Layering
• Touching color
• If on edge, bounce (requires a "precise" bounding box, and hence convex hull generation, for Scratch-compatible results)

There are a couple of implementation details I'm not sure about, though:

In regular Scratch, each sprite has a fixed number of costumes, and you can only create costumes by clicking a button in the editor. In scratch-js, on the other hand, you can dynamically instantiate as many Costume objects as you like.

This presents a problem because WebGL textures are never garbage-collected. In Scratch, the textures that back a costume are deleted when the costume is. In scratch-js, though, you could (as far as I can tell) do something silly like create and render a new Costume object every frame.

I can think of three ways to deal with this:

1. Add a destroy method to Costumes that you require API consumers to call.
2. Maintain a list of every created texture in the Renderer. Every rendering step, mark every texture that ends up getting rendered, then sweep away those that aren't.
3. Give each texture a refcount. At every call site that adds/removes textures (e.g. the Sprite.costume setter), update the refcount and delete the texture if the count is 0.

I'm leaning heavily towards the second option myself--it seems like the least complex one.

The other implementation detail that might be affected is pen color, which is also being discussed in #22. While the canvas API accepts any CSS color string, WebGL requires colors in an RGBA format. That may factor into the pen color discussion somewhat, as every accepted pen color format will need to be converted into RGB(A).

I'd love to hear your thoughts on all of this--this is a really promising project!

[PullJosh]

This would be fantastic! I don't know anything about WebGL (other than, at a very high level, what it is), but I would love to learn!

I had pretty much written off most graphic effects (except ghost and color) as well as "if on edge, bounce", but you've reminded me that being ambitious is way more fun. I'm all in. 😃

---

Now, the specifics:

In the first implementation, I would focus on keeping things simple. For that reason, let's start by destroying and recreating all costumes every frame. This is obviously a terrible design decision, but I think it's the right move to help gain momentum and get the project started.

Once that's finished, I agree that it makes sense to implement option two.

[bates64]

I would wholeheartedly recommend PixiJS as your WebGL renderer. Maintaining shaders and other fancy WebGL stuff isn't really worth the development and maintainance time within the scope of ScratchJS.

[PullJosh]

Interesting. How heavy is PixiJS? I really like the idea of keeping scratch-js super lightweight.

I also hadn't considered that switching to WebGL makes the source code more difficult for an intermediate-level javascripter to understand. Initially I had hoped to keep scratch-js very minimal (ideally without a build step), and it might be worth trying to preserve that simplicity. A strike against WebGL?

[adroitwhiz]

I'd hope not--I've already started in on it!

If you look inside Renderer.mjs and the renderer folder, you can get a feel for the complexity WebGL adds. You're correct that it's somewhat more difficult to understand than the canvas API, especially if you've never seen it before, just because it's "closer to the metal", but I think that it's a fair tradeoff given how much more you can do with it.

I've tried to abstract away the more boilerplate-y stuff (in classes like ShaderManager) and I'll probably clean it up more as I progress. I also haven't yet heavily commented the code.

If this is your first time looking at WebGL code, I'd recommend looking at WebGL Fundamentals or the MDN tutorial to get a feel for how the API functions.

The reasons I don't want to go with Pixi are: 1) It's a pretty big library (360 KB minified). 2) It contains a lot of APIs that scratch-js will never use. 3) You'd have to manually maintain the correspondence between Pixi APIs and scratch-js APIs. This includes chores like copying all transform attributes from scratch-js Sprites to PIXI.Sprites, and managing the creation and deletion of PIXI.Textures (which, to be fair, you also need to do in WebGL).

[PullJosh]

Yo! This is great! I'm not at a computer right now, so I can't run the code (easily), but everything that I can see looks awesome.

Your code looks great, and at this rate I don't think complexity will get too bad. Plus, even the worst case scenario, we can always have a default canvas-based renderer that is a little crippled with the option to import and use the WebGL renderer separately. (Although if possible, sticking to just one renderer would be even better.)

[adroitwhiz]

To support layer ordering, we'll need to do one of two things:

• Change Project.sprites from an object (which has no explicit order) to an Array, and add a getSprite or sprite method to access a sprite by name.
• Give each sprite a layerOrder property, and use some complex logic like Scratch does to keep them in order in the renderer. Some of this logic (e.g. sorting sprites by layer order) would have to be done each frame, as we don't maintain a "draw list", instead just drawing whatever Project.sprites contains.

[PullJosh]

For point one, we could also use a Map, which gives us most of the benefits of both an Object and an Array. Not sure if it's the right choice, but it's at least worth considering.

[adroitwhiz]

I considered Maps too, but they can't be reordered which kinda defeats the purpose of using them for layering.

[PullJosh]

Interestingly, the WebGL renderer is much slower for the project on the homepage than the canvas-based renderer. This more accurately matches the behavior of Scratch, but that's not necessarily what we want. :P

[PullJosh]

The Chrome profiler seems to indicate that drawing pen lines strains the GPU pretty badly.

[PullJosh]

@adroitwhiz What would it take to add support for "if touching edge" and "if on edge, bounce"? The fact that these blocks are missing seems to be tripping some people up.

I can give it an attempt if you are unable, but you certainly know more than I do. :P

[adroitwhiz]

In order to calculate tight bounding boxes around sprites, costumes' convex hulls will need to be generated. Because the shape of the convex hull depends on the effects applied to the sprite, it'll have to be stored per-sprite. I'll need to add some renderer-specific attributes to the Sprite class, if that's okay, or just add a WeakMap mapping sprites to those attributes in the renderer itself.

[PullJosh]

I'll need to add some renderer-specific attributes to the Sprite class, if that's okay.

That's okay! :)