ValueError: source code string cannot contain null bytes

spaceone commented 2 years ago

how did you install flake8?

pre-commit

unmodified output of `flake8 --bug-report`

{
  "platform": {
    "python_implementation": "CPython",
    "python_version": "3.7.3",
    "system": "Linux"
  },
  "plugins": [
    {
      "plugin": "mccabe",
      "version": "0.7.0"
    },
    {
      "plugin": "pycodestyle",
      "version": "2.9.1"
    },
    {
      "plugin": "pyflakes",
      "version": "2.5.0"
    }
  ],
  "version": "5.0.4"
}

describe the problem

what I expected to happen

Proper error handling telling which file is broken.
allow source code to contain binary data (at least if file encoding is stated as ISO8859-1)

sample code

https://github.com/univention/univention-corporate-server/blob/5.0-2/services/univention-ldb-modules/buildtools/bin/waf-svn

commands ran

$ pre-commit run -a
flake8...................................................................Failed
- hook id: flake8
- exit code: 1

multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/home/user/.cache/pre-commit/repoyq5o3_en/py_env-python3/lib/python3.7/site-packages/flake8/checker.py", line 621, in _run_checks                                                                                                     
    return checker.run_checks()
  File "/home/user/.cache/pre-commit/repoyq5o3_en/py_env-python3/lib/python3.7/site-packages/flake8/checker.py", line 531, in run_checks                                                                                                      
    self.run_ast_checks()
  File "/home/user/.cache/pre-commit/repoyq5o3_en/py_env-python3/lib/python3.7/site-packages/flake8/checker.py", line 425, in run_ast_checks                                                                                                  
    ast = self.processor.build_ast()
  File "/home/user/.cache/pre-commit/repoyq5o3_en/py_env-python3/lib/python3.7/site-packages/flake8/processor.py", line 218, in build_ast                                                                                                     
    return ast.parse("".join(self.lines))
  File "/usr/lib/python3.7/ast.py", line 35, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
ValueError: source code string cannot contain null bytes
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/user/.cache/pre-commit/repoyq5o3_en/py_env-python3/bin/flake8", line 8, in <module>
    sys.exit(main())
  File "/home/user/.cache/pre-commit/repoyq5o3_en/py_env-python3/lib/python3.7/site-packages/flake8/main/cli.py", line 22, in main                                                                                                            
    app.run(argv)
  File "/home/user/.cache/pre-commit/repoyq5o3_en/py_env-python3/lib/python3.7/site-packages/flake8/main/application.py", line 336, in run                                                                                                    
    self._run(argv)
  File "/home/user/.cache/pre-commit/repoyq5o3_en/py_env-python3/lib/python3.7/site-packages/flake8/main/application.py", line 325, in _run                                                                                                   
    self.run_checks()
  File "/home/user/.cache/pre-commit/repoyq5o3_en/py_env-python3/lib/python3.7/site-packages/flake8/main/application.py", line 229, in run_checks                                                                                             
    self.file_checker_manager.run()
  File "/home/user/.cache/pre-commit/repoyq5o3_en/py_env-python3/lib/python3.7/site-packages/flake8/checker.py", line 250, in run                                                                                                             
    self.run_parallel()
  File "/home/user/.cache/pre-commit/repoyq5o3_en/py_env-python3/lib/python3.7/site-packages/flake8/checker.py", line 217, in run_parallel                                                                                                    
    for ret in pool_map:
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 354, in <genexpr>
    return (item for chunk in result for item in chunk)
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 748, in next
    raise value
ValueError: source code string cannot contain null bytes

asottile commented 2 years ago

this appears to be a limitation in cpython itself

>>> import ast
>>> contents = open('waf-svn?raw=true', encoding='ISO-8859-1').read()
>>> ast.parse(contents)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.8/ast.py", line 47, in parse
    return compile(source, filename, mode, flags,
ValueError: source code string cannot contain null bytes

I suspect the fix here is to also catch ValueError in places where ast gets parsed and handle that "gracefully" -- in the meantime I'd recommend not linting that file as it appears to be not actual source but a hacked-together binary ?

spaceone commented 2 years ago

Yes, I can live with ignoring it (the upstream file is not meant to be changed anymore anytime). But catching the ValueError in flake8 would save a little time in manually debugging it. Maybe it can just be reported as E999.

asottile commented 1 year ago

python 3.12 is converting this to a SyntaxError which will then be handled via the usual pathway -- going to just defer that until that's fixed -- https://github.com/python/cpython/issues/96670

PyCQA / flake8