Simplify string tokenization regexes

psf / black

The uncompromising Python code formatter

https://black.readthedocs.io/en/stable/

MIT License

37.6k stars 2.38k forks source link

Simplify string tokenization regexes #4331

Closed JelleZijlstra closed 1 month ago

JelleZijlstra commented 1 month ago

Fixes #4330

I found that the repro case from #4330 was hanging in applying some of the regexes used in tokenization. These regexes have a few inefficient features:

Nested and adjacent groups instead of a single level; this leads to backtracking
Some of the groups for "normal" characters were missing an exclusion for backslash, which was also leading to backtracking.

github-actions[bot] commented 1 month ago

diff-shades reports zero changes comparing this PR (315ea67669aab217659e23c4561bd72b235dd573) to main (5683242fd432d93c9c37540948a39afd06ea4f7d).

What is this? | Workflow run | diff-shades documentation