Created unified text file tokenizer

[ohlidalp]

We have multiple formats with very similar syntax: rig def (truck), odef, tobj, character (see #2942). This new parser supports all these formats and adds new features:

• Support for "quoted strings with spaces". I first implemented them in the new .character fileformat and I decided I want them everywhere, notably .tobj so that procedural roads can have custom names and descriptions for a GPS-like navigation system.
• Simple editing in memory. The document is a vector of Tokens (types: linebreak, comment, keyword, string, number, boolean.) which keeps the exact sequence as in the file. Tokens can be easily added/modified/removed without writing any extra custom code.
• Serializability - saving the (possibly modified) file is as simple as looping through the Tokens array and writing them to a file. No custom code needs to be written for any file format.
• Ease of binding to AngelScript: a single API can modify any fileformat. All the operations the user needs is 1. insert token 2. modify token 3. delete token.

Update: I added AngelScript bindings and extended the bundled 'demo_script.as' to showcase them. There's a new button "View document" (changes to "Close document" after pressing) which opens separate window with a syntax highlighted truck file. Update2: all glitches were fixed. It's ready for review.

New script API:

• class GenericDocumentClass - loads/saves a document from OGRE resource system
• class GenericDocReaderClass - traverses document tokens
• enum TokenType (TOKENTYPE*)
• enum GenericDocumentOptions (GENERIC_DOCUMENTOPTION*)
• void ImGui::AlignTextToFramePadding()
• string Terran::getTerrainFileName()
• string Terrain::getTerrainFileResourceGroup()

To test, open game console and say loadscript demo_script.as.

[ohlidalp]

When I'm looking at the serialization loop I wrote, I realize that the way you tell good code isn't about diving into the details, but rather looking from bird's perspective where you can barely read the characters, then observing the overall shape and using just common sense to figure out what it probably does. If the above journey is a pleasure to do and the conclusion you arrive at turns out to be pretty correct, you've met good code. Yes, I've had a 🤓 moment, judge me all you want.

[ohlidalp]

Purpose: to be able to read, edit and export any fileformat (TRUCK DEF, ODEF, TOBJ, CHARACTER) directly in game using AngelScript.

Estimate: Blocked by #2930. As soon as #2930 is done, this will take at most 1 man day to finish.

Work to be done:

• Export elements in 'GenericFileFormat.h' to AngelScript.
• Add AngelScript API Document@ game.loadDocument(filename, rg_name) and void game.saveDocument(Document@) that will check if the file is unzipped and if yes, load/save the document.
• Extend the 'demo_script.as' to showcase this new feature.

[CuriousMike56]

Viewing truck file of any 'complex' vehicle -> ~30 fps drop: No frame drops with the DAF Semi. Also much of the truck file doesn't match the actual file:

[ohlidalp]

@CuriousMike56 Thanks for testing.

Viewing truck file of any 'complex' vehicle -> ~30 fps drop:

Makes sense, it's because the script traverses the entire document every frame, so it's combined overhead of DearIMGUI and string allocation by angelscript. I'd have to optimize it like in Console UI - artificially scale the scrollbar and only populate the visible part of the document. But I'd like to do it in another PR since it's just a demo.

Also much of the truck file doesn't match the actual file:

Interesting, I see 3 problems:

• [fixed] For each number, only first digit is displayed - probably a tokenizer bug.
• [fixed] set_*_defaults are omitted entirely - absolutely no idea why. UPDATE: it was a tokenizer bug, keywords containing underscore got discarded.
• Empty lines are not preserved - this is a script issue, it relies on imgui not doing SameLine() which works for single linebreaks but squashes multiple linebreaks.

[ohlidalp]

Truck definition format parsing is complete.

Note the option 'ALLOW_NAKED_STRINGS' must be on at all times, because otherwise the parser requires all strings to be in quotes. Also see these special cases:

1. The first line is the actor name, with spaces and special characters, possibly including multiple quotes. I added an option "FIRST_LINE_IS_TITLE" which parses it as one string.
2. in forset, single node numbers are parsed as NUMBER. Node ranges are not valid numbers so they decay to STRINGs.
3. The axle and interaxle keyword have a quirky abc(1 2) syntax. I added an option 'PARENTHESES_CAPTURE_SPACES' which make the whole expression parse as one string.

[ohlidalp]

Originally I only planned to cover fileformats which are similar to each other (truck, odef, tobj) but now I wanted the demo script to display TOBJ files too, but I didn't want to clog the Terrain API with getFileWhatever() funcs. I wanted to use the GenericDocument to parse the TERRN2 file, get TOBJ filenames from there and parse those as well. It turned out to be pretty straightforward, the tokenizer is already robust so it could take a few extra options.

• Parsing TERRN2 format needs these special options: "ALLOW_HASH_COMMENTS", "ALLOW_SEPARATOR_EQUALS", "ALLOW_BRACED_KEYWORDS" and of course "ALLOW_NAKED_STRINGS".

• Parsing TOBJ format needs just "ALLOW_NAKED_STRINGS" and "ALLOW_SLASH_COMMENTS".