[Experimental] Better math shift handling

[Aerijo]

Closes #13

This PR does away with math_shift, instead introducing inline_math_shift, display_math_shift, and implicit_math_shift. These are applied by the external scanner for finer control.

It also adds a new test file error_recovery, which is ideally used to test behaviour for invalid source (or seemingly invalid, like with unclosed $).

I don't think we can reliably test Tree-sitter's own error recovery decisions, as they will change as the grammar expands. Having said that, I do add a couple of tests for unmatched groups inside of math. It seems to be acting rationally currently, but we may want to remove those tests (or store them somewhere else that isn't auto checked) if they break every time something changes.

There is one behaviour I think can still be improved: foo $$ bar $ baz $$ quux is currently scoped as

(document 
  (text_mode 
    (text) 
    (tex_display_math 
      (math_shift) (math_shift) 
      (math_mode (text)) 
      (math_shift) (MISSING)) 
    (text))
  (ERROR 
    (math_shift) (math_shift) 
    (text)))

This PR changes it (somewhat unintentionally) to

(document
  (text_mode
    (text)
    (tex_display_math
      (display_math_shift)
      (math_mode (text)) 
      (MISSING))
    (tex_inline_math
      (inline_math_shift)
      (math_mode (text))
      (inline_math_shift))
    (tex_inline_math
      (inline_math_shift)
      (math_mode (text))
      (MISSING))))

Ideally, it would pick up the single $ as invalid and continue.

I've also explicitly allowed $$$$ as an empty double math shift.

• Which testing just now turns out to be perfectly legal. How about that.

[yitzchak]

@Aerijo Overall I think this is probably a good idea. Let's wait on merging until we merge or reject #11 since there is a lot of scanner overlap. Rebasing will probably be difficult.

[Aerijo]

I think a version of this was added at some point