Add talon syntax highlights

[Nova38]

I'm not sure if this would better be its own repo or included in here, but it would be nice to get syntax highlighting working in the sphinx generation. I have done a little reading and I am not sure that we can directly use the vscode texmate Grammer or the treesitter gramer to create the highlighting. It looks like sphinx uses pygments as it's engine. I will have some time this coming few weeks to do some work on porting the Grammer's over if you think it would be worthwhile

[wenkokke]

Should be pretty doable to implement a Pygments lexer which implements get_tokens_unprocessed by calling tree sitter's parse and then folds the AST to produce the stream of tokens.

[Nova38]

Ah that would be a lot simpler than reconstructing it. That is a good direction to go

[wenkokke]

A good example of how to fold over the syntax tree would be in talonfmt.formatter.

As this feature isn't unique to talondoc, it would be good to upstream it to wenkokke/py-tree-sitter-talon, so that it's available at the earliest relevant package.

I'm happy to do the upstreaming from a PR to this repository, though.

[Nova38]

I was gonna ask where it would be best to include the additions, but that seems like a better place for it. When I get a chance this week I'll make a issue and once done a pull request upstream for the implementation of the highlighter. Do you wanna keep this pull request open for integrating those changes in Talondoc (setting up the configuration and registering the syntax highlighter with sphinx) or should we make a new one for those changes?

[wenkokke]

If Sphinx uses Pygments, then registering the lexer with Pygments should be enough.

[Nova38]

Ah I just reread the stack overflow threads I had been looking at about registering the lexer. It was for including it as its own extentions for sphinx. As Talondoc is already an extentions registering it should also let it know about the lexer added to the the tree sitter library.