Parsing markdown with mix of ordered and unordered list runs into RecursionError

[ketanbhatt]

Hello 👋🏼

I am using mistune==3.0.1.

When I try to parse a markdown that is one big list of ordered and unordered list, mistune runs into Recursion Error. I suspect it might be taking it as one big nested list.

Here is a sample markdown:

- Fruits
1. Apple
2. Orange

- Vegetables
1. Eggplant
2. Onion

If this markdown continued like this to have more lines than the sys.getrecursionlimit(), parsing this markdown would result in a recursion error.

Here is some code that reproduces the error:

import mistune
import sys

list_to_repeat = """
- Fruits
1. Apple
2. Orange
"""

breaking_md = ""

# Get number of list items such that we hit the recursion limit
repeat_times = sys.getrecursionlimit() // len(list_to_repeat.splitlines()) + 10
print(f"Recursion Limit: {sys.getrecursionlimit()}")
print(f"Repeat times: {repeat_times}")

for idx in range(repeat_times):
    breaking_md += list_to_repeat

print("\nTesting breaking markdown...")
print(f"Number of lines in breaking markdown: {len(breaking_md.splitlines())}")

try:
    mistune.create_markdown(renderer="ast")(breaking_md)
except RecursionError as exc:
    print(f"RecursionError: {exc}")

non_breaking_md = ""

for idx in range(repeat_times):
    non_breaking_md += list_to_repeat
    non_breaking_md += "\n---\n"

print("\nTesting non-breaking markdown...")
print(f"Number of lines in non-breaking markdown: {len(non_breaking_md.splitlines())}")

mistune.create_markdown(renderer="ast")(non_breaking_md)
print("No error")

Output:

Recursion Limit: 1000
Repeat times: 260

Testing breaking markdown...
Number of lines in breaking markdown: 1040
RecursionError: maximum recursion depth exceeded while calling a Python object

Testing non-breaking markdown...
Number of lines in non-breaking markdown: 1560
No error

Here is the stacktrace when the recursion error is hit

I set the recursion limit to `100` so I can get a smaller full stacktrace. The stacktrace is similar with limit 1000. ``` Traceback (most recent call last): File "/Users/ketanbhatt/Library/Application Support/JetBrains/PyCharm2023.1/scratches/scratch_3.py", line 24, in mistune.create_markdown(renderer="ast")(breaking_md) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/markdown.py", line 110, in __call__ return self.parse(s)[0] File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/markdown.py", line 85, in parse self.block.parse(state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 446, in parse end_pos = self.parse_method(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method return func(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list return parse_list(self, m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list groups = _parse_list_item(block, bullet, groups, token, state, rules) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item end_pos = block.parse_method(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method return func(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list return parse_list(self, m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list groups = _parse_list_item(block, bullet, groups, token, state, rules) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item end_pos = block.parse_method(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method return func(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list return parse_list(self, m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list groups = _parse_list_item(block, bullet, groups, token, state, rules) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item end_pos = block.parse_method(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method return func(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list return parse_list(self, m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list groups = _parse_list_item(block, bullet, groups, token, state, rules) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item end_pos = block.parse_method(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method return func(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list return parse_list(self, m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list groups = _parse_list_item(block, bullet, groups, token, state, rules) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item end_pos = block.parse_method(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method return func(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list return parse_list(self, m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list groups = _parse_list_item(block, bullet, groups, token, state, rules) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item end_pos = block.parse_method(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method return func(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list return parse_list(self, m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list groups = _parse_list_item(block, bullet, groups, token, state, rules) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item end_pos = block.parse_method(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method return func(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list return parse_list(self, m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list groups = _parse_list_item(block, bullet, groups, token, state, rules) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item end_pos = block.parse_method(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method return func(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list return parse_list(self, m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list groups = _parse_list_item(block, bullet, groups, token, state, rules) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item end_pos = block.parse_method(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method return func(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list return parse_list(self, m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list groups = _parse_list_item(block, bullet, groups, token, state, rules) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item end_pos = block.parse_method(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method return func(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list return parse_list(self, m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list groups = _parse_list_item(block, bullet, groups, token, state, rules) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item end_pos = block.parse_method(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method return func(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list return parse_list(self, m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list groups = _parse_list_item(block, bullet, groups, token, state, rules) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item end_pos = block.parse_method(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method return func(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list return parse_list(self, m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list groups = _parse_list_item(block, bullet, groups, token, state, rules) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item end_pos = block.parse_method(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method return func(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list return parse_list(self, m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list groups = _parse_list_item(block, bullet, groups, token, state, rules) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item end_pos = block.parse_method(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method return func(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list return parse_list(self, m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list groups = _parse_list_item(block, bullet, groups, token, state, rules) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item end_pos = block.parse_method(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method return func(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list return parse_list(self, m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list groups = _parse_list_item(block, bullet, groups, token, state, rules) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item end_pos = block.parse_method(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method return func(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list return parse_list(self, m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list groups = _parse_list_item(block, bullet, groups, token, state, rules) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item end_pos = block.parse_method(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method return func(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list return parse_list(self, m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list groups = _parse_list_item(block, bullet, groups, token, state, rules) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item end_pos = block.parse_method(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method return func(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list return parse_list(self, m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list groups = _parse_list_item(block, bullet, groups, token, state, rules) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item end_pos = block.parse_method(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method return func(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list return parse_list(self, m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list groups = _parse_list_item(block, bullet, groups, token, state, rules) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item end_pos = block.parse_method(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method return func(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list return parse_list(self, m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list groups = _parse_list_item(block, bullet, groups, token, state, rules) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item end_pos = block.parse_method(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method return func(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list return parse_list(self, m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list groups = _parse_list_item(block, bullet, groups, token, state, rules) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item end_pos = block.parse_method(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method return func(m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list return parse_list(self, m, state) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list groups = _parse_list_item(block, bullet, groups, token, state, rules) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 91, in _parse_list_item text, continue_width = _compile_continue_width(text, leading_width) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 205, in _compile_continue_width text = expand_tab(text) File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/util.py", line 18, in expand_tab return _expand_tab_re.sub(repl, text) RecursionError: maximum recursion depth exceeded while calling a Python object ```

Thank you for your work : )

[lepture]

Hi, thanks for your reporting. This issue has been resolved.