Automap: Allow explicitly specifying Empty and Wildcard tiles

[eishiya]

tl;dr: Add special tiles that users can use to represent Empty, Any Tile, or Wildcard (empty or any tile) in their Automap rules, to make creating automapping rules more intuitive.

--

A common task when creating Automapping rules, especially for sidescrollers, is to make rules that match certain arrangements of various tiles and empty tiles. In the simplest scenario, such as a grass-top tile that replaces an empty tile that is above a grass-base tile, this isn't very difficult to achieve - if you remember that StrictEmpty is a thing. However, using StrictEmpty means that you've now given up the ability to match any non-empty tile in that rule. It's also just difficult to figure out (it still trips me up after years of making Automapping rules!), and it's not very intuitive.

Similarly, leaving a tile empty to denote "any non-empty tile" is not very intuitive. And when you need a rule that needs to combine wildcards and empty tiles, e.g. "change empty tile on top of a Grass tile into Flower, but only if there is a Tree two tiles away", it gets very messy.

It would make creating Automap rules more intuitive and simpler if Empty Tiles, Any Tile (non-empty wildcards), and Full Wildcards (matching empty and non-empty) could be specified explicitly in Automap rules, with visual representations. This poses a challenge since Automap rules are just regular map files with a few special properties.

I've described them in the context of input layers so far, but the Empty Tile could also be used on output layers to explicitly erase a tile, which is currently challenging to do. The wildcards probably don't make much sense on output layers and could be disregarded.

I can think of two options to implement this:

• A special user-defined layer that contains the three tiles, in order. This would be even better as some special properties, but Tiled has no way to reliably set a tile as the value of a special property. If only 1 or 2 tiles are in this layer, then no tile will ever match the remaining match meaning(s).

  • This option has the benefit of letting users easily define their own visuals. However, in most cases, I suspect this is more of a burden than a bonus.
  • This option doesn't do much to make Automapping more accessible to new users, since there's still a lot of setup required to get these special tiles working, and it may lead to confusion if users aren't sure whether defining these tiles is required.
  • The special layer would have to be added to every single rule file, adding tedium to an already tedious process. The docs encourage one rule per file in scenarios where order matters, so for many users,t hat's a lot of files.

• A special "automap" mode you can enable when creating maps that adds a special helper tileset with Empty, Any Tile, and Wildcard "tiles".

  • The visuals for these would be part of Tiled, and perhaps they can be floating symbols so that they will look alright regardless of map orientation. Unlike regular tiles, the art for these would need to be scaled according to the map's tile size.
  • The symbols could be a red X, white , and white on top of a red X perhaps, all with thick black outlines for good visibility in any scenario.
  • This would be more difficult to code because of the special tileset, but I think it would be a much better user experience, since it wouldn't require the user to make placeholder tiles, mess with custom properties, or add special layers. Having these tiles immediately available would probably help new users understand Automapping quicker, too.
  • Having an "Automap" map mode users can enable would also help with the occasional confusion about the fact that Automap rules are just maps, since now they can be distinguished in Tiled, if only with a checkbox.

Also, I think if special tiles like this are added, then it would be a lot easier to eliminate the need for regions layers as suggested in #1918, since there would be way fewer scenarios where empty tiles are an intended part of the region.

[bjorn]

This makes a lot of sense, and indeed would go very nicely along with obsoleting the need to define regions. I think it leaves no scenario where regions would still be required. I've tried to summarize the meaning of the special tiles for each type of layer here:

	input	inputnot	output
Empty	Match empty cell	Match non-empty cell	Erase cell
Non-Empty	Match non-empty cell	Match empty cell	ignored
Wildcard	ignored	ignored	ignored
Normal Tile	Match given tile	Match anything except the given tile	Place tile

It looks like the Wildcard is only useful in extending the region to make sure otherwise disconnected pieces form a single rule.

In the initial implementation I will not do any editor UI changes, though I do agree the UI will need to make this all more accessible. We can first implement the above based on custom tile properties. This could be in the form of a number of bools, but it looks like it might make sense to use an enum here due to the exclusivity. Of course the latter opens the question how a user-defined enum could be interpreted by Tiled.

In C++ the enum could look like:

enum MatchType {
    Tile,
    Empty,
    NonEmpty,
    Ignore
};

One could even imagine adding a "NotTile" value to the enum, which could provide an alternative to "inputnot" layers, though it's not really convenient since as a tile property that would mean duplicating the tileset for tiles with "NotTile" match type.

I think I'll implement it with such an enum in C++, but to not require the user to actually set up a matching enum in their project I would also support the bool tile properties MatchEmpty, MatchNonEmpty and MatchIgnored, where setting the first two to true would also imply the third.

It'll be a bit of work to implement this, since so far no per-tile properties are supported. For efficiency reasons, we'll want to avoid reading out custom properties each time we're matching a tile. So the data structures used by the automapping code would need to be changed from using TileLayer instances directly to using a custom Grid instance that allows for the addition of the per-cell match type.

[eishiya]

I think the end goal should be a dedicated UI and special "tiles", but in the interim, perhaps Tiled can use custom properties on tiles in a Tileset? The custom properties could be something like "Automap:TileType" and have values 1-3 or strings ("empty", "nonempty", "wildcard"). Any tiles that don't have this property set or have an invalid value are treated as normal Tiles.

This would even allow Tiled to ship with a small tileset in /examples/ that has these properties already set. Users can modify the draw offset on this tileset to make it work with their maps. I could make such a tileset if you'd like, perhaps 8x8 so that it can work with maps of most tile sizes. And of course, users would still be free to set these properties on their own tiles.

Edit: Here's an 8x8 tileset with empty, non-empty, and wildcard tiles.

[eishiya]

I have another concern: How would this new approach, lacking region definitions, deal with input layers of varying size? For example, what if I have a rule with these three input layers (all have the same name)? (red = "empty", white = regular tiles)

My intuition is that "undefined" cells like the bottom row and right column of the bottom two layers just don't contribute at all to the match options. This means the only matching values in those cells would be the solid white tiles defined in the top "input_light". So, this rule would match only solid white tiles in the bottom row and right column, and any of those slopes or empty tiles in the top left 3 tiles.

If it works this way, it would make it a lot easier to deal with rulemaps that have multiple rules that have differing numbers of input variants, some of those layers could just be left empty for the rules that don't need them. At present, those layers have to be filled with repeating tiles to avoid matching [any other tile] or (if StrictEmpty) empty.

[bjorn]

My intuition is that "undefined" cells like the bottom row and right column of the bottom two layers just don't contribute at all to the match options. This means the only matching values in those cells would be the solid white tiles defined in the top "input_light". So, this rule would match only solid white tiles in the bottom row and right column, and any of those slopes or empty tiles in the top left 3 tiles.

Right, effectively any empty tiles can be assumed to be total wildcards. The only reason to actually place wildcards explicitly, would be to make sure different pieces of the same rule are actually recognized as one rule, which is quite rare.

Of course, that changes the meaning of empty tiles, but I guess this is fine from a compatibility perspective since we can change the meaning based on whether any explicit region layers are defined.

Speaking of the meaning of empty tiles, traditionally (without "StrictEmpty") they mean "any tile that is not empty and that is also not any of the other tiles used in the rule". I guess this presents another possible value in our "MatchType" enum, though I find it rather challenging to name it intuitively.

[eishiya]

Right, effectively any empty tiles can be assumed to be total wildcards. The only reason to actually place wildcards explicitly, would be to make sure different pieces of the same rule are actually recognized as one rule, which is quite rare.

I think you may have misunderstood what I meant. I'm saying that empty input tiles should behave very differently from being wildcards, In the version I described, an empty tile doesn't add any valid matches to that cell. In the example above, the only valid matches for those bottom cells are the solid white tile, and nothing else. Not other tiles, not empty tiles. Wildcards would therefore be useful for more than just connecting regions.

-- I haven't actually found any use for the current default meaning of empty tiles ("other non-empty tiles"). I wonder if anyone else has?

[eishiya]

I haven't actually found any use for the current default meaning of empty tiles ("other non-empty tiles"). I wonder if anyone else has?

I came across at least one person who finds this feature useful, so an additional "any other non-empty tile" wildcard would probably be a good idea, to keep this behaviour accessible. As the documentation says, it does seem a good way to avoid making a bunch of inputnot layers.

[bjorn]

I think you may have misunderstood what I meant. I'm saying that empty input tiles should behave very differently from being wildcards, In the version I described, an empty tile doesn't add any valid matches to that cell. In the example above, the only valid matches for those bottom cells are the solid white tile, and nothing else. Not other tiles, not empty tiles.

You're right, describing those empty tiles as wildcards was not entirely accurate. The behavior here is actually "Ignore", so they can't make the match fail, but it also doesn't make a match succeed where other input layers are used to define what are valid inputs.

Wildcards would therefore be useful for more than just connecting regions.

I don't understand this part though. What behavior are you looking for exactly, and when would it be useful?

[eishiya]

I don't understand this part though. What behavior are you looking for exactly, and when would it be useful?

Clarity. While just leaving the region out of the input would achieve the same result, it's sometimes much more clear to the person building or reading the rule that a tile is deliberately "anything goes", rather than merely irrelevant. When a tile is empty, it's easy to think, "ah, that's going to match empty rather than anything" even when you know that's not the case, at least when you're used to working with sidescrollers where empty is important xP

The purpose of changing the Automapper to work this way isn't to change or improve its functionality (though it can do that), it's to make it more clear to the user what their rules are doing exactly. and I think a Wildcard helps with that. In the implementation, you could even make it so that aside from connecting regions, the tiles in these Wildcard cells are ignored, so they don't cause a performance problem.

[bjorn]

@eishiya Ah, makes sense.

Alright, so the only additional mode we need is the any-other-non-empty-tile (maybe we call it "unused tile"):

	input	inputnot	output
Normal Tile	Match given tile	Match anything except the given tile	Place tile
Empty	Match empty cell	Match non-empty cell	Erase cell
Non-Empty	Match non-empty cell	Match empty cell	ignored
Unused Tile	Match any tile that's not used in this rule	Match any tile used in this rule	ignored
Wildcard	ignored	ignored	ignored

The Unused Tile on an inputnot layer provides a new behavior, where I have no idea whether it is useful in any way. I guess somebody will find a use for it. I'm doubting a little about whether it should match the empty tile as well, to make it an exact negative.

The C++ enum could look like:

enum MatchType {
    Tile,
    Empty,
    NonEmpty,
    UnusedTile,
    Ignore
};

Or, when we translate inputnot layers on load into the matching data grid, we could unify the input layer handling when we'd have:

enum MatchType {
    Tile,
    NotTile,
    Empty,
    NonEmpty,
    UnusedTile,
    UsedTile,
    Ignore
};

[eishiya]

Unused Tile on inputnot actually sounds useful for some of my rules! I have a few rules that include "match any of these tiles in any of these cells", requiring a layer for each tile, filling the input region with that tile. With this addition, I could have 1-2 input layers that just include each desired matching tile once, and then have one inputnot layer filled with "Unused Tile", so it'll match any of those tiles in any position. This could cut down the number of input layers I need for some rules. Having not Unused Tile include empty would be a problem for this potential use case, so I think it should just be "any tile used in this rule".

In the case that the rule includes special tiles, I think "any tile used in this rule" should ignore them, and only react to actual specific tiles, because including any of the wildcard-types just makes this the "any tile used in this rule" equivalent to the broadest one. The only potentially useful one to include is Empty, but I think it's clearer to have to specify that one explicitly. Adding extra input layers just for one cell is going to be much easier now that you don't have to worry about empty tiles adding potential matches :]

In the enums, I wonder if Empty/NonEmpty could be unified with Tile/NotTile, with a null Tile? Or would that just make the code harder to read?

[bjorn]

In the enums, I wonder if Empty/NonEmpty could be unified with Tile/NotTile, with a null Tile? Or would that just make the code harder to read?

No, I think that's a good idea. The only problem I see is that flipped tiles count as different, and I'd want Empty and NonEmpty to work regardless of flipped state, whereas Tile and NotTile should probably keep checking also the flipped state. Though, I guess only for non-empty cells anyway, so yeah, we can omit the special modes for empty ones:

enum MatchType {
    Tile,
    NotTile,
    UnusedTile,
    UsedTile,
    Ignore
};

[eishiya]

From Discord, another special tile idea for this system: "out of bounds", so that we can make rules that specifically affect tiles near the edge of the map. Naturally, this would never match anything on infinite maps. This would allow rules to target tiles based on their location within the map, which can be useful in scenarios such as:

• generating a decorative frame around a map of any size
• making sure the map is always impassable at the edges, e.g. by making sure the edges are always water or cliffs
• treating the bottom edge as solid while treating other edges as empty for the purpose of automatically placing sidescroller collision tiles

I am not sure whether this tile should match anything when the rule map is set to MatchOutsideMap. I guess it should.

On an inputnot layer, it should match any in-bounds cell, including empty.

[bjorn]

I am not sure whether this tile should match anything when the rule map is set to MatchOutsideMap. I guess it should.

I guess when an out-of-bounds tile is specified or used, it implies that MatchOutsideMap should be true, similar to how WrapBorder and OverflowBorder automatically set that to true. There's two ways we could implement the "out of bounds". We could have a special matching tile, that matches out of bound locations, or we could specify a particular tile, which will get matched against for any out of bounds location (instead of the empty tile).

[eishiya]

I guess when an out-of-bounds tile is specified or used, it implies that MatchOutsideMap should be true,

I disagree. I think these are independent things. If anything, when making rules that use this special tile, my gut instinct would be to make sure MatchOutsideMap is OFF, because I want to focus on tiles I know are in-bounds, whereas MatchOutsideMap feels more like "I don't care about map bounds, pretend they're not there". This special tile is, in terms of end user logic, more like a "map edge" tile - it's the map edge we care about, not the "tile" out of bounds. But since "edge" is not something we can address, the cell just beyond that edge is the next best thing.

I think a special tile that matches out of bounds would be more in keeping with the style of input in this new system. Specifying a particular tile as the map edge might be a good way to backport this feature to the current/old style.

[bjorn]

If MatchOutsideMap is false, the region where the rule is applied is reduced such that its entire input region stays within the map. As such, it makes no sense for a rule to have any tiles matching "out of bounds" when MatchOutsideMap is false. It would never match.

[eishiya]

Ah, of course, makes sense. That's unfortunate since it's a little unintuitive (again, due to this being "the map edge" rather than out of bounds as such), but it's something that's easy enough to work around.

I'm not sure it's a great idea to set MatchOutsideMap automatically just because the Map Edge tile is present, since it can mess up other rules. I think it would be better to show a warning when first loading the rules, if MatchOutsideMap is false but some rules use the Map Edge tile. This way, all other rules don't over-apply, and the user is told why their edge-matching rules aren't working.

[bjorn]

Closing this issue now that there is also a built-in tileset provided with the special tiles.

[eishiya]

Unfortunately, I find most of the tiles in the built-in tileset visually unintuitive. Here's a mock-up with some tweaks that I think would help:

Ignore: Just a dotted outline reads as "empty tile" to me, so I added a * to give it visual solidity, and because * often represents a wildcard, which is what Ignore is. I kept the outline dotted because this tile matches empty too.

NonEmpty: Made the border solid. This always matches a non-empty tile, so I don't think a dotted border makes sense. Because the border is thin, away from the tile edge, and rounded, I don't think the solid border has a risk of being too solid, I don't think it'll block too much art. I also moved the ? over a little to centre it better.

Empty: Removed the red X, and instead made the dotted outline red. A dotted outline suggests empty to me, and red reinforces that. The tile is now visually nearly empty, which I think is fitting. I have no issue with the red X in terms of iconography, but it just feels too solid for a tile that should read as empty. In addition, when I use the Empty tile in my input layers, I often have other layers below that I want to see, too - with StrictEmpty, visibility was never an issue, but with the red X, it is. I think a dotted outline with an empty centre strikes a good balance.

Other: Another solid border, since this also always matches a non-empty tile. I don't think this one's icon is clear, but I don't have any better suggestions. In my personal tileset, I made it look like the NonEmpty tile, but gave it a different colour from anything else, so it stands out as "something else".

In addition, I think the tiles should be rearranged. Forbid shouldn't be at the start, since it's not likely to be a very frequently-needed tile, and since its behaviour is complex. I recommend Empty, Ignore, NonEmpty, Other, Forbid, or, if you'd prefer Empty and NonEmpty together, NonEmpty, Empty, Ignore, Other, Forbid.

Consider renaming "Forbid" to "Negate" or "Not", since it completely reverses the meaning of the other layers for the target layer rather than merely forbidding the tiles below.

[bjorn]

@eishiya Thanks a lot for the feedback! I've decided to implement it exactly as such, in 95af3a7db05836994afaa94da7c674981d8d4a58 and b24024c2462db94783877f94b8c2fe79803f7f6f.

[eishiya]

Looks like you didn't reorder the tiles though, though on Discord you agreed that it would be a good idea to put Forbid/Negate at the end xP

[bjorn]

Looks like you didn't reorder the tiles though, though on Discord you agreed that it would be a good idea to put Forbid/Negate at the end xP

I did, but not in the image since that would break existing rules (in autotests and by whoever tested early builds). The tiles have been reordered in the tsx file. :-)