[p5.js 2.0 RFC Proposal]: Renderer system refactor

[limzykenneth]

Increasing access

A more flexible renderer system enables p5.js to be used in a larger variety of rendering situation that a user may need, eg. to be print medium, controlling plotter/robot arm, SVG, and more. This enables context in which the default provided renderers of p5.js may not meet some user's specific needs (such as a renderer for Braille display perhaps?)

Which types of changes would be made?

• [ ] Breaking change (Add-on libraries or sketches will work differently even if their code stays the same.)
• [x] Systemic change (Many features or contributor workflows will be affected.)
• [ ] Overdue change (Modifications will be made that have been desirable for a long time.)
• [ ] Unsure (The community can help to determine the type of change.)

Most appropriate sub-area of p5.js?

• [ ] Accessibility
• [ ] Color
• [X] Core/Environment/Rendering
• [ ] Data
• [ ] DOM
• [ ] Events
• [ ] Image
• [ ] IO
• [ ] Math
• [ ] Typography
• [ ] Utilities
• [ ] WebGL
• [ ] Build process
• [ ] Unit testing
• [ ] Internationalization
• [ ] Friendly errors
• [ ] Other (specify if possible)

What's the problem?

p5.js 1.0 is bundled with two renderers: 2D and WebGL. They corresponds to the HTML Canvas 2d and webgl context respectively. However, there had been requests over the years to add additional renderers such as an SVG renderer or a renderer with scene graph capabilities. As the web evolve, we are also seeing new a possible standard renderer being developed, ie. WebGPU.

As currently implemented, adding a new renderer to p5.js is not an easy task which involves many parts that expects to behave differently depending on whether the current sketch is in 2D or WebGL mode.

createCanvas(400, 400, WEBGL);

The constant value to determine whether a canvas is in 2D or WEBGL mode is also not easily extendable by addon libraries.

What's the solution?

With p5.js 2.0, the renderer system is redesigned and tweaked with a few key points.

1. p5.Renderer class which both p5.Renderer2D and p5.RendererGL classes inherit from will now act more like an abstract class that it is meant to be.
   • The p5.Renderer class will determine a set of basic properties and methods any renderer class inheriting from it should implement, while extra functionalities can still be implemented on top. (eg. all renderers should implement the ellipse() method but the WebGL renderer will also implement a sphere() method that 2D renderers don't need).
   • The p5.Renderer class should never be instantiated directly.
2. The core will not have implicit knowledge of what renderers are available. Previously the two modes supported (P2D and WEBGL) are harded coded into functions like createCanvas() making creating a new rendering mode difficult without also modifying core functionalities.
   • A list of renderers should be kept under the p5.renderers object with value being the class object of the renderer (that inherits from p5.Renderer class).

     p5.renderers = {
     };

For an addon library to create a new renderer to work with p5, it will need to first create a class that inherits and implements the p5.Renderer abstract class, then register the renderer under the p5.renderers object.

(function(p5){
  class MyRenderer extends p5.Renderer {
    ellipse(x, y, w, h) {
      // ...
    }

    // ...
  }

  p5.registerAddon((p5, fn, lifecycles) => {
    p5.renderers.myRenderer = MyRenderer;
  });
})(p5);

When a sketch author wants to use the addon provided renderer above, they can use the following code when creating a canvas.

function setup(){
  createCanvas(400, 400, 'myRenderer');
}

For usage that are more similar to p5.js' own renderers, constants registration can be exposed to addon library authors as well. In this case, it is recommended to make the constant value a Symbol matching behavior in the core library itself.

Core question

Part of the motivation for this proposal is to enable a leaner build of the library for users who only use the 2D renderer but not the WebGL renderer and vice versa. If someone uses only the 2D canvas, there is no need for most if not all of the WebGL components to be included.

This creates a question, should p5.js still be bundled with both renderers for distribution? What about new renderers in the future? There are a few options for this:

• Bundling both 2D and WebGL renderers, bundling also any new renderers if they are included in the library directly.
• Bundle only 2D renderer. Log a warning message if the sketch author tries to create a WebGL canvas without loading in the WebGL renderer as an addon.
• Bundle no renderer. All and any renderers should be included as addons.

The third options is probably too extreme and probably should not be considered. Either of the first two options are open for discussions.

Pros (updated based on community comments)

• It will be much easier to maintain and extend/add renderers to p5.js with the tweaked renderer system
• Addon libraries will be able to add their own renderers while the users will still be able to use mostly the same p5.js sketching API

Cons (updated based on community comments)

• Some refactoring on the renderer side of things will be required
• A more standardized approach to defining what a renderer should include (ie. the p5.Renderer abstract class) will be needed.

Proposal status

Under review

[davepagurek]

I think this is going to be one of the more important things to get out of 2.0! A couple things that probably need clarifying though:

• How would we want a renderer to provide functionality specific to it? e.g. how WebGL has orbitControl()
• How do you think we should structure p5.Shader? It's mostly used by WebGL, but 2D mode also has the ability to run filter shaders (currently by making a WebGL graphic under the hood.) Would this only be possible if you import both renderers, or is there a minimal WebGL context that both 2D and WebGL mode can make use of?
• Some other proposals such as batching of rendering (https://github.com/processing/p5.js/issues/6805) could potentially apply to multiple renderers, but may not make sense for all future renderers. Is there a good design pattern we can use to have some shared strategies between the renderers? e.g. the vertex refactor proposal (https://github.com/processing/p5.js/issues/6560) proposes a visitor pattern that renderers can use to inspect shapes, does something like that apply here too?

[limzykenneth]

@davepagurek

How would we want a renderer to provide functionality specific to it? e.g. how WebGL has orbitControl()

The p5.Renderer abstract class will define the basic drawing functions that all renderers should (or are highly recommended to) implement, so things like line(), ellipse(), rect(), beginShape(), endShape(), vertex(), etc, that forms some of the most commonly used functions in p5.js. These should have the same function signatures where possible to maximize general compatibility amongst renderers. For renderer specific functions likeorbitControl()` they can directly be implemented as the class's own method.

How do you think we should structure p5.Shader?

This can possibly be its own importable module separate from the more complete WebGL renderer (which will include p5.Shader as well). The WebGL instancing code will probably need to be split out and shared by both the WebGL renderer and p5.Shader to avoid duplication.

Some other proposals such as batching of rendering

For batch rendering I still need to better understand the trade off for implementing it to know for sure. Ideally I would like implementing a renderer by following p5.Renderer abstract class to be comprehensive and easy to do. More complex things may need to be implemented either by individual renderer itself or by a shared implementation.

For vertex functions, a visitor pattern is my current preference, although my understanding of the pattern mainly comes from Rust's Serde crate so I'm not 100% sure that is the idea being proposed. Perhaps this may need to be worked out in practice or in a proof of concept.

[davepagurek]

Currently, renderer methods aren't directly made public, with p5.prototype functions being the ones that define the user-facing API and then pass the implementation to the renderer. Renderers could potentially do the same, by implementing their internal methods, and then extending p5.prototype like an addon would in order to add functionality? (Or maybe via a slightly different API like how we register a renderer, so that we can also have some internal logic to dispatch to the current renderer if multiple renderers have naming collisions in their extensions.) Or alternatively, we can have some logic where we automatically make all public renderer methods available?

[limzykenneth]

Most of what I'm thinking of in terms of defining a renderer is done in the proof of concept dev-2.0 branch. The renderer will be similar to this where it imports the abstract Renderer class and extend it, then simply export it.

To register the renderer, we need to add a function to the addon library API to add the custom renderer class to this list and it will be instantiated here. Additional magic will need to be used to attach all public methods of the custom renderer to p5.prototype and I would like this part to be handled by the core library instead of by the renderer themselves, so towards your latter idea.

[davepagurek]

Additional magic will need to be used to attach all public methods of the custom renderer to p5.prototype and I would like this part to be handled by the core library instead of by the renderer themselves

Sounds good! We could always enforce the convention that underscore-prefixed things are private, or maybe just forward all enumerable keys if we can use something like private methods for other stuff. (Can build systems transpile it to regular non enumerable properties if we want compatibility?)

[limzykenneth]

For truly private things I would prefer to use native private methods as you linked. It is pretty well supported so we may not even need to transpile it to something else.

[davepagurek]

Btw, today I came across this project: https://github.com/humanbydefinition/p5js-ascii-renderer (demo: https://editor.p5js.org/humanbydefinition/full/ibclfMqlk)

It got me thinking, if one wanted to make something like this as a renderer, would that mean having to also reimplement the transformation push/pop stack? Currently, that's also part of what each renderer has to provide, since 2D mode piggybacks on the native canvas's stack for some state. It's probably not necessary for v1, but just another thing to think about -- maybe there's a nice abstracting for that too in there somewhere.

[limzykenneth]

I think it would be good to be able to generalize an implementation for push/pop as well but it may bring a lot of complexity into core that the 2D renderer will likely still want to overwrite with native canvas stack in its own implementation still.

[mvicky2592]

@limzykenneth I think users will be happy with this new approach!

Your option 2 would be the best imo, having WebGL rendering as an addon. Would help with minimizing the default bundle size.

But how would that work exactly? If functions like rect (with a canvas 2d implementation) are the default would the WebGL addon override them?

[davepagurek]

I think it's a little more like the strategy design pattern: the p5 instance has a renderer object (the "strategy") and in all of the core methods like rect(), it passes off the implementation to the same method on its renderer. So something like:

rect(...args) {
  this.renderer.rect(...args)
}

So the renderer is an object that contains all the implementations. It'll be a little more advanced than that because we'll need to have a hook when you set up a renderer for it to register new methods specific to that renderer (e.g. 2d mode doesn't have sphere()) but this is how I'm imagining core methods will work.

2d mode isn't structurally special, it'd just be a strategy we bundle by default. This also means that in theory you could make webgl-only builds if you wanted, or a build with no renderers registered yet, for as modular as possible of a build.

[davepagurek]

Earlier we were talking about the push/pop stack and leaving that out from the 2D renderer. Do you think it makes sense to break the WebGL renderer into some more modular chunks that could maybe be reused, like the custom push/pop functionality and 3D transformations? I think the current code organization of the WebGL module has been a barrier to newcomers so it could use some work, but I agree that it's maybe not general enough to be part of this wider rendering system refactor.

As a minimal way to do something like that, maybe we could have these pieces work sort of like mixins that define the bits they want to help with, e.g.:

class TransformationHandler {
  constructor(pInst) {
    this.pInst = pInst
    this.matrix = new p5.Matrix()
  }
  translate(x, y, z = 0) {
    this.matrix.translate(x, y, z)
  }
  rotate(angle) {
    this.matrix.rotate(this.pInst.radians(angle))
  }
  // etc
}

class RendererGL extends p5.Renderer {
  constructor(pInst) {
    this.transforms = new TransformationHandler(pInst)
  }
  translate(x, y, z) {
    this.transforms.translate(x, y, z)
  }
  rotate(angle) {
    this.transforms.rotate(angle)
  }
  // etc
}

My thinking being:

• For one renderer it's a bit more verbose but hopefully leads to more cohesive self-contained files
• It would be easier for future renderers (I'm thinking about WebGPU here) to reuse
• By manually calling a mixin's methods, it makes it clearer what order multiple mixins are called in and how they would interact rather than something automatic or implicit (like long inheritance chains, or having it automatically call these methods)

[limzykenneth]

@davepagurek I think having a pattern where things can potentially be reused is not a bad plan. I'm not entirely sure about the design pattern of an object oriented mixin, especially when it also needs to have access to the p5 instance, ideally it would be independent as much as possible.

Or if the idea is to delegate this kind of generic functionality to the extended Renderer class perhaps the generic implementation can be done in the Renderer class and if the implemented renderer need to override it it can, otherwise use the default implementation?

[davepagurek]

Just having it all in the base class definitely simplifies the object-oriented structure that contributors would have to have in their heads. So in this case, that would mean having a full matrix implementation in the base class, but then 2D mode would override it and ignore it in favour of the native canvas's implementation? That would also work.

If we do have mixins of some sort, so far I was thinking the instance would be needed for accessing state like angle mode and to hook into the stuff that gets saved/restored by push/pop. I think both could be avoided if we have a contract where all conversion happens before passing data into a mixin or possibly even into the renderer at all, if it happens at the p5 global method level?), and by having some API to get the state to be saved, e.g.:

save() {
  return this.matrix.copy()
}
restore(state) {
  this.matrix = state
}

[limzykenneth]

In terms of instance states, it probably should be firewalled early, the mixin should be as pure as possible and the wrapper will be responsible for normalizing inputs to it. At least that's how something like p5.color using external library will need to do it anyway.

[limzykenneth]

Although thinking about this a bit more, another note is that for base class implementation, we probably should not have too complex stuff in there if the likely hood of it being shared are low, so for example I think in this case transformation via matrix may not be in the base class implementation as it can be quite complex and not sure how reusable it is.

[davepagurek]

Agreed about mixins being stateless, if the data flow between a renderer and a mixin gets too complicated, you probably end up with something that resembles the current WebGL renderer with stuff being passed back and forth between files a bunch, which is part of what I want to improve upon.

Another option for base classes is to make a BaseRenderer3D that has some common things that WebGL does that a future WebGPU will also need, such as management of its own matrices and triangulation of curves.

[limzykenneth]

BaseRenderer3D sounds like a very good idea here so essentially the renderer implementations will be a tree and additional renderers can possibly extend existing renderers to use their implementation instead of writing from scratch or depend on the base renderer having an implementation.

[davepagurek]

Sounds good to me! I can look into splitting up RendererGL a bit once the base Renderer class has stabilized, let me know when you feel it's in a good place!