Sentence splitting and lists

[dchest]

In plain text and Markdown it's common to use lists like this:

This is a simple text. It uses lists:
- This is a list item.
- This is a second list item.

Unfortunately, the model doesn't consider such list items as separate sentences. Is there a possibility for improvement here? For example, consider a single line break as an indicator that the sentence could end? I assume there's a collision with some other sentence structure that makes it necessary to consider those as a single sentence?

Sentences:

1. This is a simple text.
2. It uses lists: - This is a list item. - This is a second list item.

Two line breaks, however, make it work:

This is a simple text. It uses lists:

- This is a list item.

- This is a second list item.

Sentences:

1. This is a simple text.
2. It uses lists:
3. - This is a list item.
4. - This is a second list item.

Thank you!

[rachnachakraborty]

Hi @dchest

We appreciate your contribution in explaining the issue with relevant examples. You are right about collision with some other sentence structures which might not yield an accurate sentence breakdown.

In our current eng-lite model, Markdown format training hasn't been included. Adding it to our list of enhancements.

Shall keep you posted once the MD trained model is ready.

Many thanks, Rachna