search

mattgodbolt / owlet-editor

A modern BBC BASIC editor inspired by the BBC Micro Bot (https://bbcmicrobot.com)
https://bbcmic.ro
54 stars 4 forks source link

Improve tokeniser performance #50

Closed mattgodbolt closed 3 years ago

mattgodbolt commented 3 years ago

@8bitkick points out the editor is feeling a little sluggish. A super quick profile shows we're spending a bunch of time in the tokeniser, unsurprisingly probably due to the giant regex we use to tokenise. Most "sensible" languages use whitepsace to terminate tokens, and monaco is optimised for that. We have to disable it though as PRINTLEN is two tokens without whitespace :)

mattgodbolt commented 3 years ago

Alright...I had a good poke about but can't see any obvious way to improve, if indeed this is the cost. The regex isn't that bad:

STRING\$\(|RIGHT\$\(|LEFT\$\(|RENUMBER|ENVELOPE|INSTR\(|OPENOUT|POINT\(|INKEY\$|MID\$\(|ENDPROC|RESTORE|OPENIN|OPENUP|DELETE|REPEAT|REPORT|RETURN|COLOUR|ERROR|TAB\(|LOMEM|HIMEM|ADVAL|COUNT|FALSE|INKEY|CHR\$|GET\$|STR\$|LOMEM|HIMEM|SOUND|CHAIN|CLEAR|CLOSE|GOSUB|INPUT|LOCAL|PRINT|TRACE|UNTIL|WIDTH|OSCLI|LINE|STEP|ELSE|THEN|PAGE|TIME|BGET|EVAL|TRUE|VPOS|AUTO|LOAD|LIST|SAVE|PAGE|TIME|BPUT|CALL|DATA|DRAW|GOTO|GCOL|MODE|MOVE|NEXT|PLOT|PROC|READ|STOP|AND|DIV|EOR|MOD|OFF|SPC|PTR|ABS|ACS|ASC|ASN|ATN|COS|DEG|ERL|ERR|EXP|EXT|GET|INT|LEN|LOG|NOT|POS|RAD|RND|SGN|SIN|SQR|TAN|USR|VAL|EOF|NEW|OLD|PTR|CLG|CLS|DEF|DIM|END|FOR|LET|VDU|REM|RUN|OR|FN|LN|PI|TO|IF|ON

...it's not ideal, but I looked through a bunch of other monaco languages and couldn't find a way around this.