flake8 crash (string index out of range) on python 3.12

[matejsp]

how did you install flake8?

pip install flake8==6.1.0 flake8-future-import==0.4.7 flake8-import-order==0.18.2 flake8-import-single==0.1.5 flake8-pep3101==2.1.0 flake8-quotes==3.3.2

unmodified output of flake8 --bug-report

{
  "platform": {
    "python_implementation": "CPython",
    "python_version": "3.12.1",
    "system": "Darwin"
  },
  "plugins": [
    {
      "plugin": "flake8-future-import",
      "version": "0.4.7"
    },
    {
      "plugin": "flake8-import-order",
      "version": "0.18.2"
    },
    {
      "plugin": "flake8-import-single",
      "version": "0.1.5"
    },
    {
      "plugin": "flake8-pep3101",
      "version": "2.1.0"
    },
    {
      "plugin": "flake8-quotes",
      "version": "3.3.2"
    },
    {
      "plugin": "mccabe",
      "version": "0.7.0"
    },
    {
      "plugin": "pycodestyle",
      "version": "2.11.1"
    },
    {
      "plugin": "pyflakes",
      "version": "3.1.0"
    }
  ],
  "version": "6.1.0"
}

describe the problem

what I expected to happen

running flake8 -j 8 should just work on python 3.12 like it works on 3.11.

However it crashes:

(.venv) ➜  test flake8 -j 8 .            
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/local/Cellar/python@3.12/3.12.1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
                    ^^^^^^^^^^^^^^^^^^^
  File "/Users/myuser/projects/test/.venv/lib/python3.12/site-packages/flake8/checker.py", line 82, in _mp_run
    ).run_checks()
      ^^^^^^^^^^^^
  File "/Users/myuser/projects/test/.venv/lib/python3.12/site-packages/flake8/checker.py", line 526, in run_checks
    self.process_tokens()
  File "/Users/myuser/projects/test/.venv/lib/python3.12/site-packages/flake8/checker.py", line 511, in process_tokens
    self.handle_newline(token_type)
  File "/Users/myuser/projects/test/.venv/lib/python3.12/site-packages/flake8/checker.py", line 541, in handle_newline
    self.run_logical_checks()
  File "/Users/myuser/projects/test/.venv/lib/python3.12/site-packages/flake8/checker.py", line 438, in run_logical_checks
    comments, logical_line, mapping = self.processor.build_logical_line()
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/myuser/projects/test/.venv/lib/python3.12/site-packages/flake8/processor.py", line 238, in build_logical_line
    comments, logical, mapping_list = self.build_logical_line_tokens()
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/myuser/projects/test/.venv/lib/python3.12/site-packages/flake8/processor.py", line 219, in build_logical_line_tokens
    previous_text = self.lines[row_index][column_index]
                    ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
IndexError: string index out of range
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/myuser/projects/test/.venv/bin/flake8", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/myuser/projects/test/.venv/lib/python3.12/site-packages/flake8/main/cli.py", line 23, in main
    app.run(argv)
  File "/Users/myuser/projects/test/.venv/lib/python3.12/site-packages/flake8/main/application.py", line 198, in run
    self._run(argv)
  File "/Users/myuser/projects/test/.venv/lib/python3.12/site-packages/flake8/main/application.py", line 187, in _run
    self.run_checks()
  File "/Users/myuser/projects/test/.venv/lib/python3.12/site-packages/flake8/main/application.py", line 103, in run_checks
    self.file_checker_manager.run()
  File "/Users/myuser/projects/test/.venv/lib/python3.12/site-packages/flake8/checker.py", line 235, in run
    self.run_parallel()
  File "/Users/myuser/projects/test/.venv/lib/python3.12/site-packages/flake8/checker.py", line 204, in run_parallel
    self.results = list(pool.imap_unordered(_mp_run, self.filenames))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/python@3.12/3.12.1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/pool.py", line 873, in next
    raise value
IndexError: string index out of range
...

#### sample code

couldn't pinpoint the exact code posing a problem.

#### commands ran

```console
$ flake8 -j 8 .
...

[matejsp]

I think I have located the problematic part of code:

print(
    '''Id,ProgramId,ProgramName,ProfileId,IpAddress,EventDate,MediaId,MediaName,AdId,AdName,AdType
        "clickId1",14006,Test,"profileId1",ip1,2022-08-20T13:27:35+0200,1,"Joe Doeツ",adId1,Online,1
        "clickId2",14006,Test,"profileId1",ip1,2022-08-20T13:39:51+0200,1,"Joe Doeツ",adId2,Online,2'''
)

save this under somename bla.py and run flake8 bla.py

Moving last ) to previous line ... fixes the problem (but black is not happy).

[asottile]

there's a bug in python 3.12.0 and 3.12.1 with multiline strings containing non-ascii characters -- would it be too much to ask if your error is fixed with the unreleased cpython main branch?

[matejsp]

Yes I have tried and it works.

3.12.1 -> borked with IndexError: string index out of range

pyenv install 3.12-dev
3.12-dev -> runned OK

Any idea how to work around this or handle this error gracefully on every 3.12 version? Like ignoring the error or skipping the checks for such strings.

[sigmavirus24]

@matejsp when the underlying language is so broken that all prior versions and newer versions work, why would we attempt to fix such a narrow use-case when people are likely to get the fixed version? We can't possibly maintain tiny papercut fixes for every fundamental language flaw

[matejsp]

The only reason that comes to my mind is that you can't get a working version of 3.12. Well not yet anyway. Not to mention waiting for distros to catchup. It will take another month to be relesed (every two months for bugfix release). And couple of months for all the distros.

I understand that you can't maintain papercut fixed for language flaws. But this is official and latest and current release of python. This is hinders the upgrade to latest python version (just my opinion anyway). If you would get something like false positive or negative it would be fine. But it completely crashes flake8 making it unusable. I see it more about making flake8 code more resiliant (even for unexpected input than maintaining papercut fixes).

Can you point me to upstream bug? Tried searching https://github.com/python/cpython/issues but was not able to find the issue. Is this specific to flake8 ast analysis or poses more deep runtime problem so whole python 3.12 should be avoided as a whole? If it would be so critial they would probably scrape the 3.12 release or make immediate hotfix.

[asottile]

https://github.com/python/cpython/issues/112943

[asottile]

this isn't the first time this has happened. 3.8.0 was also completely broken in some very niche cases that we can't really work around either