Open jwx opened 1 month ago
@jwx Thanks for the proposal! I'm pretty sold on the concept, I've spoken to other maintainers and we're excited.
This definitely fits into our desire to support HTML based UI.
Questions:
ex.HTMLLayer
that creates an absolute div and expose the element to attach to?scene.addLayer("name1", ...)
Potential implementation thoughts:
CoordPlane.World
and CoordPlane.Screen
, I picture this new layer concept as generalizing these into N possible layersScene
's instead of the Screen
(but maybe defined on the Engine/Screen up front? Not sure yet)
Entity
's where any Actors/Entities added are parented to the "layer root" entity, this would allow the existing ECS ex.ParallaxComponent
to just work and so on. Maybe a potential bad idea for an mvp is class Layer extends Entity
?class Layer extends Entity
but specialized ex.HTMLLayer
or ex.ParallaxLayer
?ex.HTMLLayer
we could do some fancy stuff with clip path and with the current screen size so we don't have to do these hacks anymore@mattjennings @kamranayub @jedeen Let me know what you think as well
@jwx Thanks for the proposal! I'm pretty sold on the concept, I've spoken to other maintainers and we're excited.
This definitely fits into our desire to support HTML based UI.
Great to hear!
- Would layers be both HTML and Excalibur Actors/Entities?
Not 100% sure with what you mean by this but in peasy-viewport
(I'll use it as a reference when it might make things easier to reason about, not because Excalibur should necessarily do it the same way) there are more than one type of layer, including Canvas, DOM element (a div), Image, and Other, to mention some. I suppose a layer could also be an Actor/Entity, but the way I've made them is as parts of the Viewport/Screen and not the Game.
- Maybe an
ex.HTMLLayer
that creates an absolute div and expose the element to attach to?
Yeah, that's basically what a DOM element layer is.
- Ultimately I'd like people to BYO frontend framework to render HTML in the way that makes them happiest
Me too and this is exactly how it is in peasy-viewport
.
Excalibur already has a concept of z index on a per actor basis, how do layers fit in with the layer concept?
- We could use the new inherited z from entities to make this work. Entities are sortable within a layer, but layers are always drawn in a specific order?
Layers are drawn in a specific order and what's drawn within a layer and how it's z-ordered is up to the layer. In peasy-viewport
I've got the basics of a mechanic I sometimes call "layer jumping" the creates a shared z-ordering between layers, but it's a somewhat specialized feature.
- We'd probably want unique layer names in excalibur right?
In peasy-viewport
layer names are in the user domain. The Viewport method getLayers(name)
, well, gets all layers of a specific name.
Do we use a declarative like above API to assert the order (first is drawn first and so on)?
- How do we want to handle an imperative api layer ordering (if any?)
scene.addLayer("name1", ...)
In peasy-viewport
you can add and remove layers to the viewport to get the render (z) order you want between layers. It also exposes the layers
property for manual handling. Here's an example that changes layers to a specified level
public async setLevel(name: string) {
// Create a fade layer and run a fade in effect on the created element
const fade = this.viewport.addLayers({ name: 'fade', before: this.hudLayer })[0];
await this.fadeIn(fade.element, 300);
// Remove layers with the same name as current level
this.viewport.removeLayers(this.viewport.getLayers(this.level));
// Update the insertion point for the local level background and foreground data (layers)
this._backgrounds[name].forEach(layer => layer.before = this.worldLayer);
this._foregrounds[name].forEach(layer => layer.before = fade);
// Add the background and foreground layers for the new level
this.viewport.addLayers([...this._backgrounds[name], ...this._foregrounds[name]]);
this.level = name;
// Fade out the fade layer and remove it
await this.fadeOut(fade.element, 300);
this.viewport.removeLayers(fade);
}
Potential implementation thoughts:
- We kind of have 2 implicit layers in Excalibur already with
CoordPlane.World
andCoordPlane.Screen
, I picture this new layer concept as generalizing these into N possible layers
Yeah, sounds reasonable.
- Excalibur named layers might exist on
Scene
's instead of theScreen
(but maybe defined on the Engine/Screen up front? Not sure yet)
Is the drawing context(s) on the Screen or Scene? Feels like that might be a clue to where layers should be.
- Internally we could treat layers as special
Entity
's where any Actors/Entities added are parented to the "layer root" entity, this would allow the existing ECSex.ParallaxComponent
to just work and so on. Maybe a potential bad idea for an mvp isclass Layer extends Entity
?- Maybe a base
class Layer extends Entity
but specializedex.HTMLLayer
orex.ParallaxLayer
?
In peasy-viewport
I've intentionally kept the layers out of the game. (Also because there's no actual usage (game) in peasy-viewport
.) Parallaxing components, if I understand them correctly, might either use in-layer parallax (adjusting position as I guess it does now) or through parallaxing layers. I'm not sure there's a gain for parallaxing to make the layers Actors/Entities. Are there additional reasons to add the layers into the game?
- We'd have a default named layer for world space and another screen that capture the existing concept.
- For an
ex.HTMLLayer
we could do some fancy stuff with clip path and with the current screen size so we don't have to do these hacks anymore
Yeah, layers can definitely be used to get around some coordinate transformations.
In peasy-viewport I've intentionally kept the layers out of the game. (Also because there's no actual usage (game) in peasy-viewport.) Parallaxing components, if I understand them correctly, might either use in-layer parallax (adjusting position as I guess it does now) or through parallaxing layers. I'm not sure there's a gain for parallaxing to make the layers Actors/Entities. Are there additional reasons to add the layers into the game?
To add some insight to this, we've had needs in the past to manage layers at a scene level (y-sorted z-indexes for isometric games, where you might have verticality and want to ensure it's always above the layer below). I thought it would be a good idea to have layers as entities because in its simplest form a layer can just be an entity with children (children are positioned relative to their parent, so if layer is at 0,0 then children's coordinates are unchanged). To make it parallax, you add the ex.ParallaxComponent
and it should parallax all of its children, and that fits in really nicely with existing Excalibur APIs.
I think I see this proposal was more about having literal DOM node layers as each view port, such that you could even have multiple canvases? I believe it would be better if the layers conceptually existed within the one canvas, i.e the game, and additional HTML layers would be created as a side effect of an ex.HTMLLayer
entity for example.
Some pseudocode implementations:
class Layer extends ex.Entity {}
class BackgroundLayer extends ex.Entity {
constructor({ x, y, image, parallax }) {
super()
// (init transform and graphics components)
this.graphics.use(image)
if (parallax) {
this.addComponent(new ex.ParallaxComponent(...))
}
}
}
Here's what an HTMLLayer could look like (using some code I've used in the past to create an HTML element that overlays the canvas):
class HTMLLayer extends ex.Entity {
constructor({
tag = 'div',
id,
}: {
tag?: string
id?: string
} = {}) {
super()
this.htmlElement = document.createElement(tag)
this.htmlElement.style.position = 'absolute'
this.htmlElement.style.pointerEvents = 'none'
if (id) {
this.htmlElement.id = id
}
this.resizeObserver = new ResizeObserver(() => {
this.resizeToCanvas()
})
this.resizeObserver.observe(document.body)
}
onInitialize(engine: Engine<any>): void {
this.engine = engine
this.parentElement = engine.canvas.parentElement!
this.parentElement.appendChild(this.htmlElement)
this.resizeToCanvas()
const scene = this.scene!
scene.on('activate', () => {
this.show()
})
scene.on('deactivate', () => {
this.hide()
})
}
show() {
this.htmlElement.removeAttribute('hidden')
}
hide() {
this.htmlElement.setAttribute('hidden', '')
}
resizeToCanvas() {
if (this.htmlElement && this.engine?.canvas) {
this.emit('resize')
const { width, height, left, top, bottom, right } =
this.engine.canvas.getBoundingClientRect()
const scaledWidth = width / this.engine.drawWidth
const scaledHeight = height / this.engine.drawHeight
this.scale.x = scaledWidth
this.scale.y = scaledHeight
this.htmlElement.style.top = `${top}px`
this.htmlElement.style.left = `${left}px`
this.htmlElement.style.bottom = `${bottom}px`
this.htmlElement.style.right = `${right}px`
this.htmlElement.style.overflow = 'hidden'
this.htmlElement.style.width = `${this.engine.drawWidth}px`
this.htmlElement.style.height = `${this.engine.drawHeight}px`
this.htmlElement.style.transform = `scale(${scaledWidth}, ${scaledHeight})`
this.htmlElement.style.transformOrigin = '0 0'
}
}
}
I don't know Excalibur that well yet, but I think this is two different use cases. (Which I think you're getting at.) The first use case is where you've got Excalibur Actors/Entities in the canvas
that you want to parent to something else in the canvas
(I think it'd be more appropriate to parent it to another Actor/Entity) that has parallax. The second is where you want to add layers, probably HTML that's vanilla or a framework of choice, behind or on top of the canvas
(simplified). I'm (so far) only talking about the second one.
I believe it would be better if the layers conceptually existed within the one canvas, i.e the game, and additional HTML layers would be created as a side effect of an ex.HTMLLayer entity for example.
How do you get fully functional HTML layers into the canvas
?
Gotcha, yeah, my gut feeling is we don't want to introduce layering as a concept outside of the canvas, with the exception being the HTMLLayer.
How do you get fully functional HTML layers into the canvas?
It'd just be a way to create and expose an overlaaying html element, so that you insert your own HTML into (framework or vanilla html or whatever). So there'd have to be some additional API for a user to hook into it, as even though it's called an ex.HTMLLayer
it's actually not a layer in the scene (so maybe this isn't the right spot for it...)
(but I'll defer to Erik on this idea of layers outside of canvas, just giving my 2c)
In this proposal I'm primarily talking about HTML layers being the usage. Outside the canvas
. But more than one, if someone wants to. And, probably, as a conceptual part of the Screen/Viewport rather than the Scene. Unless the canvas
is considered part of the Scene rather than the Screen?
Great discussion!
I think there are maybe 2 independent ideas forming:
Entity/Actor layering in Scenes (which I do want now after talking about it haha). For now let's put a pin in this discussion for the purpose of this proposal. I'll write up a separate proposal for this concept with @mattjennings
HTML UI overlaying the Engine canvas (maybe we use different terminology than layers in1 to disambiguate?).
Ideally I still would love an easier way to handle building HTML UI's that can be integrated with Excalibur more tightly. In this concept perhaps it does make sense to have these HTML overlays exist on the Screen
, at minimum they are probably fairly coupled. But I could see a world where folks want separate overlays on a per Scene
basis, hence the pull towards scenes as well.
Current problems in excalibur I'd love to solve with HTML overlays:
I think there are maybe 2 independent ideas forming:
Yeah.
- Entity/Actor layering in Scenes (which I do want now after talking about it haha). For now let's put a pin in this discussion for the purpose of this proposal. I'll write up a separate proposal for this concept with @mattjennings
Sounds good. Just so that I understand, this is for sorting in-game layering challenges like for example the bridge problem? If so, they can mostly be solved with parenting and/or additional z-ordering. (Here's a proof of concept from a DOM only, non-canvas
game lib I tinkered with a few years ago.)
- HTML UI overlaying the Engine canvas (maybe we use different terminology than layers in1 to disambiguate?).
I think that's a good idea.
HTML Overlays
Ideally I still would love an easier way to handle building HTML UI's that can be integrated with Excalibur more tightly. In this concept perhaps it does make sense to have these HTML overlays exist on the
Screen
, at minimum they are probably fairly coupled. But I could see a world where folks want separate overlays on a perScene
basis, hence the pull towards scenes as well.
Great! That keeps this proposal alive. I think the layers belong with the Viewport/Screen (but I might just be biased). Where is the canvas drawing context, on Screen
or Scene
? However, if more than one Scene
can be shown/rendered at the same time, it does make sense to connect them to the Scene
. If not, we can always give devs a way to switch (some) layers when they switch scenes.
Current problems in excalibur I'd love to solve with HTML overlays:
- We currently build HTML UI without much formalism, it'd be nice to have a path
Will be resolved.
- Easy translation of Excalibur coordinates to HTML space, and vice-versa
Can be resolved.
- Solve screen scaling of HTML UI hacks
Can probably be resolved (I need to understand it more before a more definitive answer).
- Easy hooks to BYO frontend framework
Will be resolved.
- Easy ways to interact with UI from excalibur code (I hesitate to suggestion some form of pluggable state)
Yes, but I think, at least right now, that HTML manipulation should be outside of Excalibur (in line with previous point).
Any preferences on how we proceed?
Where is the canvas drawing context, on Screen or Scene? However, if more than one Scene can be shown/rendered at the same time, it does make sense to connect them to the Scene. If not, we can always give devs a way to switch (some) layers when they switch scenes.
Good questions, currently the graphics drawing context abstraction lives on the engine and screen. Screen is basically responsible for coordinating layout and positioning in the DOM, and handling any fullscreen/resize/resolution change/viewport change.
Only 1 Scene
can be drawn at a time (and that won't change), but Scene switches can happen at any time
Yes, but I think, at least right now, that HTML manipulation should be outside of Excalibur (in line with previous point).
Totally fair, I think I agree we should stay out of this for now
Any preferences on how we proceed?
@jwx Let's sketch out a possible Excalibur API for the HTML Overlays, maybe a small low effort prototype to prove out the concept?
@mattjennings anything you'd like to see?
I'd say we want to support adding layers both when initializing and during the game.
For initialization, I see two options (or a mix of them):
a) instantiated objects
const game = new ex.Engine(
...,
layers: [ new ex.CanvasLayer(canvasLayerOptions), new ex.HTMLayer(htmlLayerOptions) ],
...
);
b) configurations
const game = new ex.Engine(
...,
layers: [ { canvasLayerOptions }, { htmlLayerOptions } ],
...
];
During the game, maybe just stick to the instantiated
game.add(new HTMLLayer(htmlLayerOptions,) { before: existingLayer });
And of course, also
game.remove(layer);
game.getLayer(layerName);
Thoughts?
I'm interested in this feature (the canvas layer) as well, but let me ask the same question that I asked myself when I first ran into the "lack" of multiple drawing layers in EX: What would this feature add for the average developer using EX?
I couldn't answer it in a satisfactory way myself, so I chose to create a plugin-like setup for it, using existing features of EX.
100% personal opinion: The idea to add HTML layers as well feels even more out of scope for a core EX feature with all the alternative options available in existing 3th party libraries.
Context
The
canvas
provides limited help when it comes to creating UI such as buttons, input fields, text and other things used in menus, HUDs and similar. At the same time, the browser provides a great set of tools for this with html+css in the DOM. It'd be beneficial to be able to easily select which tool,canvas
or DOM, to use for which task.Proposal
Make the screen/viewport capable of having different layers of different types and help keeping them in sync.
The concept comes from
peasy-viewport
in which providing the configurationresults in this![multi-layered viewport](https://github.com/user-attachments/assets/094b219a-f8c6-40e8-913c-b42abc1d1e5d)
In this example,
peasy-viewport
automatically manages and synchronizes layers based on the position and zoom of the camera used in the canvas layer.While the above example might be a bit overkill for Excalibur to implement, it would be nice to have some support for layers. Syntax could be similar, but as an optional
layers
option when creating the Engine. Management of the Layers would be the responsibility of the Screen.