Chat highlighting

[pythops]

use https://github.com/trishume/syntect

[mkroman]

Responses that include code are returned using markdown code blocks.

This unfortunately makes it much more difficult to determine when a code block ends, and also requires a modular markdown parser.

The pulldown-cmark crate seems like a good candidate with its push parsing architecture.

But there's still an edge case that can always happen in LLM context windows.. It's possible for the reply to include the beginning of a code block, some code, and then it stops halfway without closing the block, because it hit the context limit.

[pythops]

Maybe I'm missing something but how is that related to highlighting ? this crate is for markdown parsing and rendering to html

[mkroman]

Ah, I only just realized that you're using termimad to render the markdown as rich text and not just writing plain text :facepalm:

My comment was made with the assumption that it still needed to be enriched.

Unfortunately it's quite an old feature request: https://github.com/Canop/termimad/issues/1

I'm curious how efficient termimad is, particularly in relation to #12.

[pythops]

I'm curious how efficient termimad is, particularly in relation to https://github.com/pythops/tenere/issues/12.

Need to be tested to see.

There is also a potential to use https://github.com/sharkdp/bat as well

[pythops]

with the release of bat 0.24, it should be straightforward to add syntax highlighting

[pythops]

Done