TI-Toolkit / tokens

TI-BASIC token information XMLs for inclusion in other projects
8 stars 0 forks source link

Add more models and utilities to the scripts #15

Closed kg583 closed 1 year ago

kg583 commented 1 year ago

Implements three main additions to the scripts:

  1. Previously missing models, particularly all the .fr variants, recognizable by the OsVersion struct. None of the added models appear in the sheets, but can now be used in conjunction with those that do.
  2. A new OsVersions enum to hold INITIAL and LATEST helper constants. Applications can extend this enum to include other useful version markers like CSE 4.0 or CE 5.6.0.
  3. A new TokenTrie class which can construct a trie from a Tokens object. The trie can be used to get all tokens which could be a prefix of a given string more efficiently than looping over Tokens directly. Most parsers will use maximal prefixes, so a helper method is included for just the longest token as well. Both methods include the remaining part(s) of the input string in their returns.