[proposal] Make the screen/viewport capable of using layers

[jwx]

Context

The canvas provides limited help when it comes to creating UI such as buttons, input fields, text and other things used in menus, HUDs and similar. At the same time, the browser provides a great set of tools for this with html+css in the DOM. It'd be beneficial to be able to easily select which tool, canvas or DOM, to use for which task.

Proposal

Make the screen/viewport capable of having different layers of different types and help keeping them in sync.

The concept comes from peasy-viewport in which providing the configuration

viewport.addLayers([
  // Parallaxing background layers
  { name: 'city', image: 'background/layer_08.png', size: { x: 1920, y: 1080 } },
  { name: 'city', parallax: 0.97, image: 'background/layer_07.png', size: { x: 1920, y: 1080 }, position: { x: 960, y: 0 } },
  { name: 'city', parallax: 0.85, image: 'background/layer_06.png', size: { x: 1920, y: 1080 }, repeatX: true },
  { name: 'city', parallax: 0.8, image: 'background/layer_05.png', size: { x: 1920, y: 1080 }, repeatX: true },
  { name: 'city', parallax: 0.7, image: 'background/layer_04.png', size: { x: 1920, y: 1080 }, repeatX: true },
  { name: 'city', parallax: 0.5, image: 'background/layer_03.png', size: { x: 1920, y: 1080 }, repeatX: true },
  { name: 'city', parallax: 0.25, image: 'background/layer_02.png', size: { x: 1920, y: 1080 }, repeatX: true },
  { name: 'city', parallax: 0, image: 'background/layer_01.png', size: { x: 1920, y: 1080 }, repeatX: true },

  // Player (blue box) layer
  { name: 'world', parallax: 0, size: { x: 0, y: 0 }, position: { x: 0, y: 242 } },
  // Effects (gray circles) layer
  { name: 'effects', canvasContext: '2d', scaling: true },
  // HUD layer
  { name: 'HUD', id: 'HUD' },

  // Parallaxing foreground layer
  { name: 'city', parallax: -0.2, image: 'background/layer_02.png', size: { x: 1920, y: 1080 }, position: { x: 0, y: 270 }, repeatX: true },
]);

results in this

In this example, peasy-viewport automatically manages and synchronizes layers based on the position and zoom of the camera used in the canvas layer.

While the above example might be a bit overkill for Excalibur to implement, it would be nice to have some support for layers. Syntax could be similar, but as an optional layers option when creating the Engine. Management of the Layers would be the responsibility of the Screen.

[eonarheim]

@jwx Thanks for the proposal! I'm pretty sold on the concept, I've spoken to other maintainers and we're excited.

This definitely fits into our desire to support HTML based UI.

Questions:

• Would layers be both HTML and Excalibur Actors/Entities?
  • Maybe an ex.HTMLLayer that creates an absolute div and expose the element to attach to?
  • Ultimately I'd like people to BYO frontend framework to render HTML in the way that makes them happiest
• Excalibur already has a concept of z index on a per actor basis, how do layers fit in with the layer concept?
  • We could use the new inherited z from entities to make this work. Entities are sortable within a layer, but layers are always drawn in a specific order?
• We'd probably want unique layer names in excalibur right?
• Do we use a declarative like above API to assert the order (first is drawn first and so on)?
  • How do we want to handle an imperative api layer ordering (if any?) scene.addLayer("name1", ...)

Potential implementation thoughts:

• We kind of have 2 implicit layers in Excalibur already with CoordPlane.World and CoordPlane.Screen, I picture this new layer concept as generalizing these into N possible layers
• Excalibur named layers might exist on Scene's instead of the Screen (but maybe defined on the Engine/Screen up front? Not sure yet)
  • Internally we could treat layers as special Entity's where any Actors/Entities added are parented to the "layer root" entity, this would allow the existing ECS ex.ParallaxComponent to just work and so on. Maybe a potential bad idea for an mvp is class Layer extends Entity?
  • Maybe a base class Layer extends Entity but specialized ex.HTMLLayer or ex.ParallaxLayer?
  • We'd have a default named layer for world space and another screen that capture the existing concept.
• For an ex.HTMLLayer we could do some fancy stuff with clip path and with the current screen size so we don't have to do these hacks anymore

@mattjennings @kamranayub @jedeen Let me know what you think as well

[jwx]

@jwx Thanks for the proposal! I'm pretty sold on the concept, I've spoken to other maintainers and we're excited.

This definitely fits into our desire to support HTML based UI.

Great to hear!

• Would layers be both HTML and Excalibur Actors/Entities?

Not 100% sure with what you mean by this but in peasy-viewport (I'll use it as a reference when it might make things easier to reason about, not because Excalibur should necessarily do it the same way) there are more than one type of layer, including Canvas, DOM element (a div), Image, and Other, to mention some. I suppose a layer could also be an Actor/Entity, but the way I've made them is as parts of the Viewport/Screen and not the Game.

• Maybe an ex.HTMLLayer that creates an absolute div and expose the element to attach to?

Yeah, that's basically what a DOM element layer is.

• Ultimately I'd like people to BYO frontend framework to render HTML in the way that makes them happiest

Me too and this is exactly how it is in peasy-viewport.

• Excalibur already has a concept of z index on a per actor basis, how do layers fit in with the layer concept?

  • We could use the new inherited z from entities to make this work. Entities are sortable within a layer, but layers are always drawn in a specific order?

Layers are drawn in a specific order and what's drawn within a layer and how it's z-ordered is up to the layer. In peasy-viewport I've got the basics of a mechanic I sometimes call "layer jumping" the creates a shared z-ordering between layers, but it's a somewhat specialized feature.

• We'd probably want unique layer names in excalibur right?

In peasy-viewport layer names are in the user domain. The Viewport method getLayers(name), well, gets all layers of a specific name.

• Do we use a declarative like above API to assert the order (first is drawn first and so on)?

  • How do we want to handle an imperative api layer ordering (if any?) scene.addLayer("name1", ...)

In peasy-viewport you can add and remove layers to the viewport to get the render (z) order you want between layers. It also exposes the layers property for manual handling. Here's an example that changes layers to a specified level

public async setLevel(name: string) {
  // Create a fade layer and run a fade in effect on the created element
  const fade = this.viewport.addLayers({ name: 'fade', before: this.hudLayer })[0];
  await this.fadeIn(fade.element, 300);

  // Remove layers with the same name as current level
  this.viewport.removeLayers(this.viewport.getLayers(this.level));

  // Update the insertion point for the local level background and foreground data (layers) 
  this._backgrounds[name].forEach(layer => layer.before = this.worldLayer);
  this._foregrounds[name].forEach(layer => layer.before = fade);

  // Add the background and foreground layers for the new level
  this.viewport.addLayers([...this._backgrounds[name], ...this._foregrounds[name]]);
  this.level = name;

  // Fade out the fade layer and remove it
  await this.fadeOut(fade.element, 300);
  this.viewport.removeLayers(fade);
}

Potential implementation thoughts:

• We kind of have 2 implicit layers in Excalibur already with CoordPlane.World and CoordPlane.Screen, I picture this new layer concept as generalizing these into N possible layers

Yeah, sounds reasonable.

• Excalibur named layers might exist on Scene's instead of the Screen (but maybe defined on the Engine/Screen up front? Not sure yet)

Is the drawing context(s) on the Screen or Scene? Feels like that might be a clue to where layers should be.

• Internally we could treat layers as special Entity's where any Actors/Entities added are parented to the "layer root" entity, this would allow the existing ECS ex.ParallaxComponent to just work and so on. Maybe a potential bad idea for an mvp is class Layer extends Entity?
• Maybe a base class Layer extends Entity but specialized ex.HTMLLayer or ex.ParallaxLayer?

In peasy-viewport I've intentionally kept the layers out of the game. (Also because there's no actual usage (game) in peasy-viewport.) Parallaxing components, if I understand them correctly, might either use in-layer parallax (adjusting position as I guess it does now) or through parallaxing layers. I'm not sure there's a gain for parallaxing to make the layers Actors/Entities. Are there additional reasons to add the layers into the game?

• We'd have a default named layer for world space and another screen that capture the existing concept.
  • For an ex.HTMLLayer we could do some fancy stuff with clip path and with the current screen size so we don't have to do these hacks anymore

Yeah, layers can definitely be used to get around some coordinate transformations.

[mattjennings]

In peasy-viewport I've intentionally kept the layers out of the game. (Also because there's no actual usage (game) in peasy-viewport.) Parallaxing components, if I understand them correctly, might either use in-layer parallax (adjusting position as I guess it does now) or through parallaxing layers. I'm not sure there's a gain for parallaxing to make the layers Actors/Entities. Are there additional reasons to add the layers into the game?

To add some insight to this, we've had needs in the past to manage layers at a scene level (y-sorted z-indexes for isometric games, where you might have verticality and want to ensure it's always above the layer below). I thought it would be a good idea to have layers as entities because in its simplest form a layer can just be an entity with children (children are positioned relative to their parent, so if layer is at 0,0 then children's coordinates are unchanged). To make it parallax, you add the ex.ParallaxComponent and it should parallax all of its children, and that fits in really nicely with existing Excalibur APIs.

I think I see this proposal was more about having literal DOM node layers as each view port, such that you could even have multiple canvases? I believe it would be better if the layers conceptually existed within the one canvas, i.e the game, and additional HTML layers would be created as a side effect of an ex.HTMLLayer entity for example.

Some pseudocode implementations:

class Layer extends ex.Entity {}

class BackgroundLayer extends ex.Entity {
    constructor({ x, y, image, parallax }) {
        super()      

        // (init transform and graphics components)

        this.graphics.use(image)

       if (parallax) {
           this.addComponent(new ex.ParallaxComponent(...))
       }
    }
}

Here's what an HTMLLayer could look like (using some code I've used in the past to create an HTML element that overlays the canvas):

class HTMLLayer extends ex.Entity {
  constructor({
    tag = 'div',
    id,
  }: {
    tag?: string
    id?: string
  } = {}) {
    super()
    this.htmlElement = document.createElement(tag)
    this.htmlElement.style.position = 'absolute'
    this.htmlElement.style.pointerEvents = 'none'

    if (id) {
      this.htmlElement.id = id
    }

    this.resizeObserver = new ResizeObserver(() => {
      this.resizeToCanvas()
    })
    this.resizeObserver.observe(document.body)
  }

  onInitialize(engine: Engine<any>): void {
    this.engine = engine
    this.parentElement = engine.canvas.parentElement!
    this.parentElement.appendChild(this.htmlElement)
    this.resizeToCanvas()

    const scene = this.scene!

    scene.on('activate', () => {
      this.show()
    })

    scene.on('deactivate', () => {
      this.hide()
    })
  }

  show() {
    this.htmlElement.removeAttribute('hidden')
  }

  hide() {
    this.htmlElement.setAttribute('hidden', '')
  }

  resizeToCanvas() {
    if (this.htmlElement && this.engine?.canvas) {
      this.emit('resize')

      const { width, height, left, top, bottom, right } =
        this.engine.canvas.getBoundingClientRect()

      const scaledWidth = width / this.engine.drawWidth
      const scaledHeight = height / this.engine.drawHeight
      this.scale.x = scaledWidth
      this.scale.y = scaledHeight

      this.htmlElement.style.top = `${top}px`
      this.htmlElement.style.left = `${left}px`
      this.htmlElement.style.bottom = `${bottom}px`
      this.htmlElement.style.right = `${right}px`
      this.htmlElement.style.overflow = 'hidden'

      this.htmlElement.style.width = `${this.engine.drawWidth}px`
      this.htmlElement.style.height = `${this.engine.drawHeight}px`
      this.htmlElement.style.transform = `scale(${scaledWidth}, ${scaledHeight})`
      this.htmlElement.style.transformOrigin = '0 0'
    }
  }
}

[jwx]

I don't know Excalibur that well yet, but I think this is two different use cases. (Which I think you're getting at.) The first use case is where you've got Excalibur Actors/Entities in the canvas that you want to parent to something else in the canvas (I think it'd be more appropriate to parent it to another Actor/Entity) that has parallax. The second is where you want to add layers, probably HTML that's vanilla or a framework of choice, behind or on top of the canvas (simplified). I'm (so far) only talking about the second one.

I believe it would be better if the layers conceptually existed within the one canvas, i.e the game, and additional HTML layers would be created as a side effect of an ex.HTMLLayer entity for example.

How do you get fully functional HTML layers into the canvas?

[mattjennings]

Gotcha, yeah, my gut feeling is we don't want to introduce layering as a concept outside of the canvas, with the exception being the HTMLLayer.

How do you get fully functional HTML layers into the canvas?

It'd just be a way to create and expose an overlaaying html element, so that you insert your own HTML into (framework or vanilla html or whatever). So there'd have to be some additional API for a user to hook into it, as even though it's called an ex.HTMLLayer it's actually not a layer in the scene (so maybe this isn't the right spot for it...)

[mattjennings]

(but I'll defer to Erik on this idea of layers outside of canvas, just giving my 2c)

[jwx]

In this proposal I'm primarily talking about HTML layers being the usage. Outside the canvas. But more than one, if someone wants to. And, probably, as a conceptual part of the Screen/Viewport rather than the Scene. Unless the canvas is considered part of the Scene rather than the Screen?

[eonarheim]

Great discussion!

I think there are maybe 2 independent ideas forming:

1. Entity/Actor layering in Scenes (which I do want now after talking about it haha). For now let's put a pin in this discussion for the purpose of this proposal. I'll write up a separate proposal for this concept with @mattjennings

2. HTML UI overlaying the Engine canvas (maybe we use different terminology than layers in1 to disambiguate?).

HTML Overlays

Ideally I still would love an easier way to handle building HTML UI's that can be integrated with Excalibur more tightly. In this concept perhaps it does make sense to have these HTML overlays exist on the Screen, at minimum they are probably fairly coupled. But I could see a world where folks want separate overlays on a per Scene basis, hence the pull towards scenes as well.

Current problems in excalibur I'd love to solve with HTML overlays:

• We currently build HTML UI without much formalism, it'd be nice to have a path
• Easy translation of Excalibur coordinates to HTML space, and vice-versa
• Solve screen scaling of HTML UI hacks
• Easy hooks to BYO frontend framework
• Easy ways to interact with UI from excalibur code (I hesitate to suggestion some form of pluggable state)

[jwx]

I think there are maybe 2 independent ideas forming:

Yeah.

1. Entity/Actor layering in Scenes (which I do want now after talking about it haha). For now let's put a pin in this discussion for the purpose of this proposal. I'll write up a separate proposal for this concept with @mattjennings

Sounds good. Just so that I understand, this is for sorting in-game layering challenges like for example the bridge problem? If so, they can mostly be solved with parenting and/or additional z-ordering. (Here's a proof of concept from a DOM only, non-canvas game lib I tinkered with a few years ago.)

2. HTML UI overlaying the Engine canvas (maybe we use different terminology than layers in1 to disambiguate?).

I think that's a good idea.

HTML Overlays

Ideally I still would love an easier way to handle building HTML UI's that can be integrated with Excalibur more tightly. In this concept perhaps it does make sense to have these HTML overlays exist on the Screen, at minimum they are probably fairly coupled. But I could see a world where folks want separate overlays on a per Scene basis, hence the pull towards scenes as well.

Great! That keeps this proposal alive. I think the layers belong with the Viewport/Screen (but I might just be biased). Where is the canvas drawing context, on Screen or Scene? However, if more than one Scene can be shown/rendered at the same time, it does make sense to connect them to the Scene. If not, we can always give devs a way to switch (some) layers when they switch scenes.

Current problems in excalibur I'd love to solve with HTML overlays:

• We currently build HTML UI without much formalism, it'd be nice to have a path

Will be resolved.

• Easy translation of Excalibur coordinates to HTML space, and vice-versa

Can be resolved.

• Solve screen scaling of HTML UI hacks

Can probably be resolved (I need to understand it more before a more definitive answer).

• Easy hooks to BYO frontend framework

Will be resolved.

• Easy ways to interact with UI from excalibur code (I hesitate to suggestion some form of pluggable state)

Yes, but I think, at least right now, that HTML manipulation should be outside of Excalibur (in line with previous point).

Any preferences on how we proceed?

[eonarheim]

Where is the canvas drawing context, on Screen or Scene? However, if more than one Scene can be shown/rendered at the same time, it does make sense to connect them to the Scene. If not, we can always give devs a way to switch (some) layers when they switch scenes.

Good questions, currently the graphics drawing context abstraction lives on the engine and screen. Screen is basically responsible for coordinating layout and positioning in the DOM, and handling any fullscreen/resize/resolution change/viewport change.

Only 1 Scene can be drawn at a time (and that won't change), but Scene switches can happen at any time

Yes, but I think, at least right now, that HTML manipulation should be outside of Excalibur (in line with previous point).

Totally fair, I think I agree we should stay out of this for now

Any preferences on how we proceed?

@jwx Let's sketch out a possible Excalibur API for the HTML Overlays, maybe a small low effort prototype to prove out the concept?

@mattjennings anything you'd like to see?

[jwx]

I'd say we want to support adding layers both when initializing and during the game.

For initialization, I see two options (or a mix of them):

a) instantiated objects

const game = new ex.Engine(
  ...,
  layers: [ new ex.CanvasLayer(canvasLayerOptions), new ex.HTMLayer(htmlLayerOptions) ],
  ...
  );

b) configurations

const game = new ex.Engine(
  ...,
  layers: [ { canvasLayerOptions }, { htmlLayerOptions } ],
  ...
  ];

During the game, maybe just stick to the instantiated

game.add(new HTMLLayer(htmlLayerOptions,) { before: existingLayer });

And of course, also

game.remove(layer);
game.getLayer(layerName);

Thoughts?

[Autsider666]

I'm interested in this feature (the canvas layer) as well, but let me ask the same question that I asked myself when I first ran into the "lack" of multiple drawing layers in EX: What would this feature add for the average developer using EX?

I couldn't answer it in a satisfactory way myself, so I chose to create a plugin-like setup for it, using existing features of EX.

100% personal opinion: The idea to add HTML layers as well feels even more out of scope for a core EX feature with all the alternative options available in existing 3th party libraries.