Fix TXT formats to be proper TOML

[Des-Nerger]

I wanted to develop an alternative flare-engine implementation that I'd feel comfortable to play Diablo II maps in. And be as much compatible with the data formats of the original flare-engine implementation, as I possible can. So far, I've basically rewritten flare-engine v0.01 in Rust and intended to continue developing it evolutionally: using the oldest FLARE versions as references of what I must implement at each step: flare v0.01, flare v0.05, flare v0.06 and so on. However, doubts completely paralyzed me for some months. Including the doubts about the TXT formats: I don't feel like relying on something that doesn't have a 1-to-1 mapping to a hashmap and/or requires outsiders to write their own parsers in order to read/write the content from flare-games... just recently I also thought about ditching Tiled as a tile editor of choice, in order (among the other troubles with Tiled) to avoid the annoyance of maintaining two .pngs: 1 for Tiled and 1 for flare-engine itself... Which means I have nowhere to run anymore: I must be compatible with the flare-engine's TXTs.

The TOML format was developed for this exact purpose: to be a hashmap-corresponding version of INI files people like us love. Would you generally be positive about accepting a cosmetical redesign of the TXT formats that turned them into proper TOML files? If so, I would discuss more specifically.

[dorkster]

An interesting proposal, but I'm going to say no. Here's why:

• The APPEND and INCLUDE directives in our parser don't have an equivalent in TOML (understandably). These allow us to merge files while parsing, and are very important to how our modding system works. So even if we adopted TOML, we would need to extend it in a non-standard way and write our own parser.
• We must maintain compatibility with existing mods, many of which will never be updated. We could modify our existing parsing system to output the same data structure as the TOML parser, but we'd still need to maintain both parsers.
• TOML integers can only be int_64t, but we use uint64_t for some data. More specifically, we use unsigned 64-bit integers for our XP tables. We also use this type for saving the player's XP and time played, but this could be worked around by offsetting by -0x8000000000000000 and casting to a signed value.

There are some appealing things in TOML when compared to our existing format, such as the handling of nested arrays. But I don't think the advantages are big enough to jump ship.

[Des-Nerger]

Fair enough. The change is too radical.

Btw, isn't the APPEND directive technically unnecessary when there's INCLUDE already? I mean, couldn't:

APPEND
# to engine/resolutions.txt

virtual_height=480,600,768

be as well expressed as:

INCLUDE engine/resolutions.txt

virtual_height=480,600,768

?

// p.s. Speaking of the C programming language, by itself it doesn't define any #include directive either. And yet, we use it all the time in there. :thinking:

[dorkster]

That example does not work.

INCLUDE will use the highest priority copy of a file and nothing more. So if engine/resolutions.txt contained

INCLUDE engine/resolutions.txt

it would loop infinitely (this has been fixed in v1.14.34: https://github.com/flareteam/flare-engine/commit/251389d9bcdbe339a3293583a8c3dc0fbd51be2f)

APPEND, on the other hand, will walk down the mod list by priority and combine the files.

We could probably just treat the self-include as an append. But I'd rather the syntax be distinct.

[Des-Nerger]

🤔 Another alternative for me for not relying on the "insecure" "fragile" TXT format is to try to use Godot's .tscn / .tres for that purpose (but keeping all the names / conventions I can from flare-engine), which is also kind of INI-like... If I'm gonna use the Godot's TileMap editor (not their game engine itself!) instead of Tiled anyway, then that would be a reasonable alternative, I think. The parsers for the Godot scene format are probably gonna be quite popular / maintained across different programming languages' repositories. Maybe as maintained as TOML parsers; maybe not quite as, but still well enough. 🤔.

[clintbellanger]

the "insecure" TXT format

Not sure if this is your concern, but this reminds me. Some people see TXT format as a drawback because players can easily open the files to cheat or learn spoilers. But this is the whole reason I originally used TXT format, so that it opens by default for anyone poking around. The engine was designed to be open source and open data. I personally first learned how to make games by opening plain data files and breaking things. I invite others to learn this way too.

TXT is not specifically necessary, just about any human readable and editable format would probably be fine. Early Flare version config files were very simple and few, and I was too entrenched to change once we started storing more complex data. I could understand future version of Flare preferring formats that are easy for developers to reimplement/reuse instead of easiest for end players to edit. That could open up an ecosystem of tools, etc.

There's a line somewhere, I'd say that formats like json or xml are inconvenient for regular people to edit. If one stray character can break the whole data file it's just harder for people to learn, or to make a whole game with.

[Des-Nerger]

Oh, sorry. I guess I've used the "insecure" word in an unconventional way. I didn't mean I want to hide or complicate-editing any sensitive game data or something. Quite contrary: I very much appreciate the open and easy-to-edit nature of the flare-engine's formats. What bothers me is the reliability of parsers and easyness of extending / redesigning the data structures showing up in those TXTs. For instance, I thought in the future it might be necessary for me to introduce some things inspired by Diablo II formats (which are binary, but can be easily reimagined as INI-like TOML files), familiar to people from its modding community:

[header]
version = [7, 6]
tileHeadersPointer = 276

[[tile]]
direction = 2
roofHeight = 0
materialFlags = [0, 0]
height = -160
width = 160
orientation = 2
mainIndex = 0
subIndex = 21
rarityOrFrameIndex = 0
unknown = [255, 0, 255, 0]
subtileFlags = [0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0]
blockHeadersPointer = 9012
blocksDataLength = 5584
usuallyZeros = [0, 0, 0, 0]

[[tile.block]]
x = 96
y = -128
gridX = 0
gridY = 0
format = [1, 16]
length = 105
fileOffset = 260

[[tile.block]]
x = 64
y = -128
gridX = 0
gridY = 0
format = [1, 16]
length = 459
fileOffset = 365

# ...

// I think my main problem is that I'm still not exactly sure how close I want to be to Diablo II and how close to FLARE. It all may be even an impossible wish to materialize...