Hard crash parsing certain markdown files

I have been using tree_sitter_languages to parse markdown. Some of my md files are causing a hard crash of the parser.parse() call:

Assertion failed: (i == length), function deserialize, file scanner.cc, line 79.
Abort trap: 6

I have isolated a sample which can trigger the crash. I binary searched the file to find a single offending line. Then, I gradually replaced all the characters with X until the crash went away.

Some notes:

There is a single astrisk * remaining. The parser no longer crashes if you remove it.
There are still underscores _ remaining. Turning them into Xs resolves the crash.
Reducing the line length, or splitting the text onto multiple lines also seems to avoid the crash.

I have filed this issue with both of these projects, as I am not sure which is most likely to be able to resolve it:


code = '''
XXXXXX_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX X XXXX X XXXXX *X XXXXXXXX XXXXXXX X XXXX X XXXXX XXXXXXX_XXXXX XXXXXXXXXXX X XXXX X XXXXX XXXXXXX_XXXXXXXXX XXXXX X XXXX X XXXXX XX_XXXXX XXXX X XXXX X XXXXX XXXXXX XXXXX X XXXX X XXXXX XXXXXXXXXX XXXXXXXXX X XXXX X XXXXX XXXXXXXX_XX_XXXXXXX XXXX X XXXX X XXXXX XX_XXXXXXXXX XXXX X XXXX X XXXXX XXX_XXXXXXXXX XXXXXXXXXXXXXXX X XXXX X XXXXX XXXXXX_XXXXXXXX XXXXXXXXXXXXXXXX X XXXX X XXXXX XXXXXX XXXXXXXXX X XXXX X XXXXX XXXXX_XXXXXX XXXXXXXXXX X XXXX X XXXXX XXXXXXX XXXXXXXXXXXXXXXXXXXX X XXXXX XXXXXXX_XXXXXXX_XXXXXXXX_XXXXXX XXXXXXXXXXXX X XXXX X XXXXX XXXXX_XXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXX X XXXX X XXXXX XXXX_XXXXX_XXX_XXXX_XXXXXXX XXXXXXXXXXXX X XXXX X XXXXX XXXXXXXX XXXXXXXXXXXXXXXX X XXXX X XXXXX XXXXXXXXXX XXXXXXXXXXXX X XXXX X XXXXX XXXXXX_XXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXX X XXXX X XXXXX XXXX_XXXXXX XXXXXXXXXXXX X XXXX X XXXXX XXXXXX_XXXXXXX_XXXXXXX XXXXXXXXXXXX X XXXX X XXXXX XXXXXX_XXXXXX_XXXXX XXXXXXXXXXXX X XXXX X XXXXX XXXXXXXX_XXXXX_XXXXXXX XXXXXXXXXXXX X XXXX X XXXXX XXXXXXXX_XXXXX_XXXXXXX XXXXXXXXXXXX X XXXX X XXXXX XXXXXXXX_XXXXXX XXXXXXXXXXXXX X XXXX X XXXXX XXXX_XXXXXXXX XXXXXXXXXXX X XXXX X XXXXX XXXXXXXXXX XXXXXXXXX X XXXX X XXXXX XXXXXXXXXX XXXXXXXXX X XXXX X XXXXX XXXXX_XXXXXXXX XXXXXXXXXXXX X XXXX X XXXXX XXXXX_XXXXXXXXXXX XXXXXXXXXXXXXXX X XXXX X XXXXX XXXXXXXXXXXX XXXXXXXXXXXXXXXX X XXXX X XXXXX XXXXXXX_XXXXX_XXX_XXXXX XXX X XXXX X XXXXX XXXXXX_XXXXXX_XXXXXXX XXXXXXXXXXXX X XXXX X XXXXX XXXXXX_XXXXXXXX XXXXXXXXXXXX X XXXX X XXXXX XXXXXX_XXXX_XX_XXXXXXX XXXXXXXXXXXX X XXXX X XXXXX XXXXXXXX_XXXXXXX XXX X XXXXXXXXXXXX XXXX X XXXX X XXXXX XXXXXXXXX XXX X XXXXXXXXXXXX XXXX X XXXX X XXXXX XXXXXXXX XXX X XXX XXXXXX_XXXXXXXX XXXX X XXXXXX XXX_XXXX XXXXXXXXXXXX XXXXX X XXXX X XXXX
'''

import tree_sitter_languages

print(tree_sitter_languages.__version__) # 1.7.0

parser = tree_sitter_languages.get_parser('markdown')
parser.parse(bytes(code, "utf8"))

ikatyang / tree-sitter-markdown

Hard crash parsing certain markdown files #59