(EAI-349) [UI] Disable setext headings in LGMarkdown

[nlarew]

Jira: (EAI-349) [UI] Disable setext headings in LGMarkdown

Changes

• Right now we have an issue where for one token's worth of streaming we'll render a line as heading. This works around that by disabling setext header parsing via remark - instead we treat the text as a paragaraph.

Overview

Given this markdown text:

# This is an H1 (ATX)

Lorem ipsum.

## This is an H2 (ATX)

Dolor sit amet.

### This is an H3 (ATX)

consectetur adipiscing elit, sed do eiusmod tempor.

This is an H1 (Setext)
-------

This is an H2 (Setext)
=======

Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris.

We can see in a before and after that ATX style headings are rendered as headings whlie setext style headings are just paragraphs.

Demo

Before

https://github.com/mongodb/chatbot/assets/15657698/1d552219-164e-4f2d-b33b-1a10ccbd2a23

After

https://github.com/mongodb/chatbot/assets/15657698/88c4fe54-3087-4fbb-b6f0-1fbaf8c8a547

[nlarew]

should this change be added to the lg-chat package at some point?

I think it's better left to implementers for now. If more people have streaming markdown cases that encounter this we could package up this remark plugin for them but I think lg-markdown should support as much of "standard" markdown as possible out of the box.