Sprite Atlas Slicing Improvement

[deanljohnson]

I've been working on an editor plugin called Pixmap Slicer that can auto-slice a sprite sheet and allows the definition of rectangles with the mouse rather than by manually entering coordinates.

After a discussion with @ilexp it was decided that we should integrate this into Duality directly. Additionally, it was noted that changes coming with v3.0 affect the integration of this plugin.

Features (current and planned)

• Auto-slice sprite sheets
• Add/delete/edit rectangles with mouse
• Specify the order of rectangles in the atlas
• Slice the Pixmap with a grid from the slicing window
• Zoom and pan the Pixmap
• Naming of animation sequences or individual sprites

All of these features can be implemented without changing anything in the rest of Duality except for the naming of animation sequences.

Integration Plan

1. [x] Cleanup current features and get a working baseline.
2. [x] Move changes into a forked Duality branch. PR to Duality master.
3. [x] Implement features onto Duality master that do not require atlas changes
4. [x] Move the slicer plugin to v3.0 branch and integrate with existing 3.0 changes
5. [ ] Improve atlas definitions to add support for named sequences, see issue #697 and PR #708
6. [ ] Implement additional features on the slicer given the named sequence support

[deanljohnson]

Progress

• Implemented add, delete, modify of rects
• Finished auto-slicing
• Improved zoom
• Added the ability to define the order of rectangles by clicking on them in the desired order
• Made the whole design much more extensible
• Documented current baseline and general cleanup

I'm going to start working on moving this to a forked Duality branch now. I want to get the initial PR in sometime soon so I can get some feedback on it.

@ilexp You had said this should be placed in the EditorBase Duality project. I'm looking at the folder structure there and I'm not sure where many of these things should go and don't want to start creating a bunch of new folders in that project. Here is a summary of the files I currently have and where they could go:

1x editor action -> EditorActions folder The slicing form itself -> a new Forms folder 4x states (different modes of interaction with the slicing form, kind of like CamViewState) -> not sure where these should go. Maybe a Forms/PixmapSlicer folder? 1x event args file -> either new EventArgs folder or Forms/PixmapSlicer 2x utility files -> not sure

Thoughts?

[ilexp]

I'm going to start working on moving this to a forked Duality branch now. I want to get the initial PR in sometime soon so I can get some feedback on it.

Sounds good 👍

Here is a summary of the files I currently have and where they could go:

I think your list makes a lot of sense. Here's the list, adjusted as I'd probably approach it:

• EditorActions --> that editor action you're defining
• Forms/PixmapSlicer --> The form itself, its states, event args
• Depending on what exactly the utility files do, either put them with the rest of the supporting classes inside the slicer directory, or create a new Utility folder if they're general-purpose enough to warrant that.

That way, we'll have grouped together the most strongly connected parts as a distinct group, with the interfacing editor action part separately.

It might make sense to integrate the slicer into the inspector as well by extending the PixmapPropertyEditor to show a button that would bring up the slicer tool to expose it a bit more beyond the editor action - doesn't have to be in first iteration though.

[ilexp]

Progress

• Concluded work on the v2.X PR for the Pixmap Slicer, merged back to master.
• Triggered a binary release - it's out there now 🙂
• Merged the most recent master into develop-3.0 and adjusted all parts of the code that required changes for v3.0.

[deanljohnson]

From my testing it seems like everything is working well on the 3.0 branch. Unless you see something else that needs to be done I think we can start discussing the other idea mentioned in this issue: named sequences.

A named sequence would be defined by a name and then a series of indices associated with that sequence. This would make including multiple animations within a single sprite sheet easier.

The approach I'm kind of favoring when implementing this is to build a new Resource class, lets call it Animation. This Animation class would then store a reference to the Pixmap that it is based off of, the sequence of animation frames, and either a duration or time values for each frame. Additionally (maybe a later iteration) this class could support animation events - callbacks at a certain point within the animation.

I think we should also want to consider if we want to try and build a generalized animation component that can animate public properties of objects besides just sprite indices.

[ilexp]

Sounds great 👍 Let's come up with a solid concept on how to proceed.

I'd like to split this topic into multiple parts, one being atlas improvements and another one being animation features. That last one is, I think, big enough to warrant its own issue, maybe even its own set of issues. Like you mentioned, animation can be much much more than sprite flipbooks, and a generalized resource should probably address that.

Atlas Use Cases

Let's gather some use cases.

Flipbook Animation

First, we have the regular sprite animation that we're talking about. This is one thing that works differently in v2.x and v3.0, as rendering sprites has been decoupled from animating sprites by introducing the ICmpSpriteRenderer interface.

This interface is implemented by renderers like SpriteRenderer or ActorRenderer in order to provide a standard way to change the displayed graphic, and even cross-fade between two distinct / indexed graphics when supported. The renderer doesn't care about the sequence of images it displays, or their associated behavior, it just needs to know which index is active right now - or which two indices are active, as well as their fade value.

On the other side of that deal is the SpriteAnimator component, which cares only about the sequence of indices, but not rendering. It can animate any ICmpSpriteRenderer and will provide crossfading info, so renderers can make use of it when supported, but it's more or less providing the same limited animation capabilities the AnimSpriteRenderer did in v2.x.

A more complex implementation can be found in the ActorAnimator from the Tilemaps sample, which defines an ActorAnimation class that provides direction specific sprite index animations. Though it can also animate any ICmpSpriteRenderer, it's still far from a general case and very specialized, but it gets the job done and does exactly what is required in the sample project. Similar to this, other projects might use other implementations.

One thing the animation directionality highlights is that animation can live in a multi-dimensional world, with multiple parameters affecting them, and changing or even blending the used sprite index. Even a flipbook animation isn't necessarily a fixed sequence of indices - or maybe it is, but there is an optional higher level entity that can switch or blend between multiple simultaneously running flipbook animations.

Texturing

Besides animations, an atlas can also be used as just that: Defining regions in a Pixmap / Texture which are mapped to a bunch of vertices when rendering. It is already used by Fonts and Tilesets to look up UV coordinates for each glyph / tile, but with a little shader help for tiling support, it could be used in freely painted polygon terrains and similar. As a typical optimization step, it's also an option to bake hundreds of sprites into a single Pixmap and address them using an atlas index to reduce the number of required state changes for rendering a 2D scene.

9-Patches

For UI support, but also more flexible / advanced polygon texturing, it would be nice if there was a way to define 9-patches in a Pixmap. A 9-patch is like an atlas rect that defines an optional border width for each of the four sides, splitting the rect into nine smaller rects, where the three horizontal center rects can stretch vertically, and the three vertical center rects can stretch horizontally. It's a super neat feature to define, for example, the look of a UI button or window as a single graphic and still allow it to resize arbitrarily. With a clever algorithm, the same 9-patch can also be used to texture an arbitrary polygon, which can be pretty powerful in 2D sidescrollers and the like.

And of course, all of the above use cases could be combined in a single Pixmap / Texture after doing a bake step during build in order to optimize.

Atlas API

So there are two spots where atlases are currently used, one being Pixmap (pixel coordinates) and the other being Texture (texture / UV coordinates). Aside from differing coordinate spaces they behave exactly the same and should provide the same featureset. Right now, they're just a List<Rect>, but with new features to be added, it might make sense to create an actual ImageAtlas class that is used by both Pixmap and Texture, and exposed using their Atlas property.

Since atlas lookups happen a lot on a per-frame basis (every text glyph, every tile, every sprite), performance is a priority - probably not a problem, just something to keep in mind. For example, fast-path atlas indexing using an int for some internal array is a feature we might want to keep regardless of other lookup methods.

On the top of my head, I'd probably go with the following featureset for a potential ImageAtlas:

• Fast-path atlas rect retrieval via direct indexing. Probably still via int that can go directly to some internal array.
• The possibility to tag individual rects, or groups of rects, with a string. Could be done manually in the slicer, in the object inspector, or automatically during import when importing sprite sheets from other applications.
• Lookup of an index or index list via tag string. Doesn't need to be super-fast, as it's probably more of a one-off-per-object operation.
• Optional definition of rect borders for each rect side for 9-patch support, but could be used for other purposes as well.
• Optional definition of a rect pivot, allowing renderers to position the sprite relative to its own position. Would grant some extra robustness for animations, but also simplify regular SpriteRenderer setups, so we don't need to specify the exact rect / offset for every single renderer, and adjust that everywhere when it changes at some point.

("Optional" meaning we define default values that behave as if the feature wasn't there)

With the above features, we do get some low-key animation support, but don't actually go too far into that territory. Most of the features that can be used for easing flipbook animation workflows might as well be used for different cases and are (hopefully) general purpose enough to not feel weird in a general image atlas definition.

---

I don't have a nice summary for this, just some general thoughts right now. Would be interested in what you think.

Btw, should we move this to a different issue and close this one? Something like "Extend Pixmap / Texture Atlas"?

[SirePi]

Sorry to jump in, but did someone say 9-Patches? They are basically the starting block of my UI plugins so if it can be of help all my code is available for use _{and maybe integrate a simple UI in the core?}

[ilexp]

Sorry to jump in, but did someone say 9-Patches? They are basically the starting block of my UI plugins so if it can be of help all my code is available for use

Sounds good! Can you throw in a link for reference?

and maybe integrate a simple UI in the core?

One step at a time! Integrating a UI framework into the core would be a great thing at some point™️, but probably not now. Need to keep the number of simultaneous construction sites manageable, and UI will be a pretty big one 🙂

[SirePi]

Full, direct IDrawDevice drawing - rotation, zoom, etc.. supported; requires NonPowerOfTwo to be set if the Texture is non-power-of-two-sized) https://github.com/SirePi/duality-frozen/blob/master-nuget/UI/Widgets/SkinnedWidget.cs#L211

Canvas based, no support for rotation, actually manages properly non-power-of-two textures without the need to set the property https://github.com/SirePi/duality-ui/blob/master/SnowyPeak.Duality.Plugins.YAUI/Controls/Control.cs#L148

[deanljohnson]

I definitely think we need to approach these things as separate issues and work on them one at a time.

I would say we first address the creation of an ImageAtlas class with the following goals:

• Optimized Rect lookup by index
  • Can't really beat what we have now, but lets just not make it worse 😏
• Rect Pivots
  • Need to then adjust existing sprite/texture rendering to take this pivot into account
• Tag individual rects by name
  • Care should be taken to avoid memory overhead when this feature is not heavily used

I'm not sure if the ImageAtlas level is the appropriate place to tag groups of sprites - to me that seems like a higher level of abstraction... perhaps a future Animation resource supports that or something similar.

[ilexp]

I'm not sure if the ImageAtlas level is the appropriate place to tag groups of sprites - to me that seems like a higher level of abstraction... perhaps a future Animation resource supports that or something similar.

Good point. Idea for a compromise: Have API for both retrieving "the first" / a single rect index via tag, and also for retrieving all rect indices with a tag, sorted from lowest to highest. That way, they'd really be just simple, "dumb" tags without any higher level / animation-related assumptions, but still viable to be used as part of a very simple flipbook animation workflow. Could also be useful aside from animations.

[raycrasher]

Slightly tangential, but I found that it is easier to just have another panel display the available sprites an atlas has (an "Atlas Viewer"). Dragging the sprite from the viewer to the scene editor will create a SpriteRenderer.

Please see below sample. Note that this is a working example, and am using it for a few months now.

[ilexp]

@raycrasher This looks super useful, thanks for posting! Is this editor extension by any chance available as open source?

[raycrasher]

Yes.