Significant rewrite of the tokens sheet.

[rpitasky]

Closes #13 and closes #12, see those issues for details. Conversion was done via script, so there may (but shouldn't) be errors.

[rpitasky]

We (I?) still need to devise a nice API for the reference parser that captures what you're supposed to do with these files; the main idea is that if you need to reference it more than just once in a program, you should extract whatever you want while parsing the file and stick it in a suitable data structure for your task. The existing reference parser is just dreadful in nearly every way, not to mention incompatible with this rewrite.

[rpitasky]

Nice catch on the colors, I'll fix that.

Your proposal for checking display vs accessible doesn't really work because <accessible> represents what you'd type in SC3 or TokenIde; there are tokens like the sequence variables that have ASCII displays but are not typed using exactly their display.

[kg583]

there are tokens like the sequence variables that have ASCII displays but are not typed using exactly their display.

I did not realize those exist, but they make sense. Perhaps you could still run a check and visually inspect which are different to make sure they're as expected?

[rpitasky]

I ran the suggested check; the only conflicts are tokens that have changed (such as sin), tokens in the class I explained above, and the color tokens.

[adriweb]

post-merge: can we discuss whether we (still) need to have a two-level depth for 1/2-byte tokens? i.e. why can't we just have one level and the value attribute will just have one or two byte in its value, and the program reading the XML will know this when it reads the value anyway (> 0xFF means it's a two-byte token)

[rpitasky]

I don't see any compelling reason to do that; this is nice because you effectively get a byte-by-byte parser for free and in editors you can collapse token ranges you don't want.

[kg583]

and the program reading the XML will know this when it reads the value anyway (> 0xFF means it's a two-byte token)

Sure, but the other way around becomes harder. Parsing an 8xp using a flat map requires a check against the whole sheet to see if a two-byte sequence is oncoming (wherein you need to read the next byte ahead of time), while the non-flat variant simply requires the next byte, from which you can index one layer down for a two-byte.

[adriweb]

No worries, I'll just convert to my internal format for processing this input. Anyway, merge? :P