Closed jgersti closed 2 years ago
Indeed, this one is a bit of a doozy looks like.
Let me give this one some thought. This sort of hints at the 2 pass approach being insufficient.
Perhaps we need to introduce parsing rules, with priority ordering, similar to how the parameter processor & engine work. Doesn't need to be near as complicated though.
Perhaps a delegate Processor taking List
First phase pulls out firstPassTokens Second phase can look for floating point numbers Third phase can parse other primitive types fourth phase can parse secondPassTokens
That approach would let us add more parsing rules iteratively as needed. Hopefully can be done without too many characters. Since only one line is parsed per tick (and is processed in a different tick) i don't think the additional steps for processing the tokens will cause script parsing complexity issues, but guess we'll see.
Thoughts?
My suggestion to keep changes pretty minimal is to split the tokenizing into the following steps:
firstPassTokens
and tokenize matches<, >, =, &, |, ?, :, +, -
and tokenize matches.
and tokenize matchesedit: +
and -
need to also be seperated in step 4.
Currently the following steps are performed:
firstPassTokens
and tokenize matchessecondPassTokens
As a site note: maybe call these functions TokenizeXXX
since that is what they actually do.
Ya I agree the .
would need to be separated last, after checking for primitives.
If we remove Vector parsing of floats from parse primitive (which I think we should do) then this simplifies even more, as you don't need to do the primitive tokenization in step 3.
.
).
Thoughts?
Tokenize seems fine as a name.
When tokenizing a floating point number which is directly preceded or followed by an character from
separateTokensSecondPass
the floating point number is decomposed.Print 2.0+1
is tokenized asPrint "2" Dot "0" Add "1"
.In #169 this situation is even worse since the component delimiter
:
is tokenized in the second pass. This means a vector containing variables and floating point numbers cannot be defined and the dot product between two such vectors inside an the success branch of an ternary condition might be ambiguous.Print pi:0.5:1
is tokenized asPrint "pi" : "0" Dot "5" : "1"