xNVSE / NVSE

New Vegas Script Extender (NVSE)
https://git.io/JfSAo
702 stars 55 forks source link

Add ScriptTokenizer to fix variable type retrieval #159

Closed Demorome closed 1 year ago

Demorome commented 1 year ago

Fixes 2 bugs:

  1. Comments were being ignored by GetDeclaredVariableType, which parses a script to see if one of its declared variables exists, and what type it is declared as. This lead to the following issue:
    
    scn SomeQuestScript
    ;array_var test
    int test

scn SomeUDF begin Function { } SomeQuest.test = 1 ; DOES NOT COMPILE! ; Assumes the type of test is array_var, and an array can't be set to a number ; This happens because the token-matching code is blind to comments. end


The solution I propose is to use the `ScriptTokenizer` class I've added, which properly ignores tokens that are within `;` and `/* */`-style comments.

My concern with the solution is, well, `ScriptTokenizer`'s code is kinda spaghetti, but I did test a good couple of cases with the newly added unit tests.

2. Variables declared in quest scripts using one-line multiple variable declarations couldn't be referenced in other scripts, unlike regularly defined vars.
This only required a slight change in how the tokenizer was being used.