Closed Utodev closed 5 years ago
I have partially done this, but to avoid having to recalculate all token numbers, in case a given token it's not used, drb will be dumping it in the tokens table as a one-character-token. The interpreter won't use it anyway, as there is no text containing that token, so the toke text it's not important, but we will be saving as much bytes as the token legth minus 1.
I keep the issue open so I can go for the original target .
Most people will use the built-in tokens, so there is a chance some token is unused. Make a first pass to find not used tokens, and delete them before making final compression.