Incorrect choice of encoding Cyrillic characters resulting in UnicodeDecodeError

[AlexArcPy]

Description

I have:

• Windows 10
• Clean conda Python 3.6 env

I ran pip install flake8-bandit

and got Successfully installed GitPython-2.1.11 PyYAML-3.13 bandit-1.5.1 flake8-3.5.0 flake8-bandit-1.0.2 flake8-polyfill-1.0.2 gitdb2-2.0.4 mccabe-0.6.1 pbr-4.3.0 pycodestyle-2.3.1 pyflakes-1.6.0 six-1.11.0 smmap2-2.0.4 stevedore-1.29.0

I have a Python module with the following contents:

# -*- coding: UTF-8 -*-
"""Module docstring."""
print('hello') # э

I run flake8 C:\Development\Flake8PluginDev\foo.py

an error is produced.

(BanditDebug) C:\Users\user>flake8 C:\Development\Flake8PluginDev\foo.py
Traceback (most recent call last):
  File "c:\anaconda3\envs\banditdebug\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\anaconda3\envs\banditdebug\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Anaconda3\envs\BanditDebug\Scripts\flake8.exe\__main__.py", line 9, in <module>
  File "c:\anaconda3\envs\banditdebug\lib\site-packages\flake8\main\cli.py", line 16, in main
    app.run(argv)
  File "c:\anaconda3\envs\banditdebug\lib\site-packages\flake8\main\application.py", line 396, in run
    self._run(argv)
  File "c:\anaconda3\envs\banditdebug\lib\site-packages\flake8\main\application.py", line 384, in _run
    self.run_checks()
  File "c:\anaconda3\envs\banditdebug\lib\site-packages\flake8\main\application.py", line 310, in run_checks
    self.file_checker_manager.run()
  File "c:\anaconda3\envs\banditdebug\lib\site-packages\flake8\checker.py", line 321, in run
    self.run_serial()
  File "c:\anaconda3\envs\banditdebug\lib\site-packages\flake8\checker.py", line 305, in run_serial
    checker.run_checks()
  File "c:\anaconda3\envs\banditdebug\lib\site-packages\flake8\checker.py", line 579, in run_checks
    self.run_ast_checks()
  File "c:\anaconda3\envs\banditdebug\lib\site-packages\flake8\checker.py", line 486, in run_ast_checks
    checker = self.run_check(plugin, tree=ast)
  File "c:\anaconda3\envs\banditdebug\lib\site-packages\flake8\checker.py", line 435, in run_check
    return plugin['plugin'](**arguments)
  File "c:\anaconda3\envs\banditdebug\lib\site-packages\flake8_bandit.py", line 35, in __init__
    self._load_source()
  File "c:\anaconda3\envs\banditdebug\lib\site-packages\flake8_bandit.py", line 73, in _load_source
    self.source = f.read()
  File "c:\anaconda3\envs\banditdebug\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 68: character maps to <undefined>

To Reproduce

If I have

# -*- coding: UTF-8 -*-
"""Module docstring."""
print('hello') # й

I run flake8 C:\Development\Flake8PluginDev\foo.py

and get C:\Development\Flake8PluginDev\foo.py:3:15: E261 at least two spaces before inline comment

It works fine.

Now if we change the Cyrillic # й comment into either the # э, # я, or # с (with all three characters typed with Russian keyboard layout, mind Russian C) and run flake8 C:\Development\Flake8PluginDev\foo.py

an error is produced.

It seems as bandit has problems with only three Russian letters эяс because when running the flake8 with the comment containing all Russian letters except эяс, it works fine:

# -*- coding: UTF-8 -*-
"""Module docstring."""
print('hello') # йцукенгшщзхъфывапролджё

Expected behavior Bandit should handle all Cyrillic characters without throwing UnicodeDecodeError error.

Bandit version

(BanditDebug) C:\Users\user>bandit --version
bandit 1.5.1
  python version = 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 11:21:07) [MSC v.1900 32 bit (Intel)]

[AlexArcPy]

@sobolevn, you are impacted by this as well in wemake-python-styleguide

[sobolevn]

@AlexArcPy what happens if you remove coding magic comment?

[AlexArcPy]

@sobolevn , same error!

[ericwb]

I don't see Bandit in the traceback. Is this a problem just with flake8-bandit? Can you recreate the problem using just Bandit?

https://github.com/tylerwince/flake8-bandit

[AlexArcPy]

@ericwb , you are right, the issue is not with the bandit, but rather with the flake8-bandit. I'll post an issue in their repo, but they do not seem to be very active, so not sure if this is something anyone could look at.

Running (BanditDebug) C:\Users\user>bandit "C:\Development\Flake8PluginDev\foo.py" produces

[main]  INFO    profile include tests: None
[main]  INFO    profile exclude tests: None
[main]  INFO    cli include tests: None
[main]  INFO    cli exclude tests: None
[main]  INFO    running on Python 3.6.6
Run started:2018-10-04 21:58:32.449122

Test results:
        No issues identified.

Code scanned:
        Total lines of code: 2
        Total lines skipped (#nosec): 0

Run metrics:
        Total issues (by severity):
                Undefined: 0.0
                Low: 0.0
                Medium: 0.0
                High: 0.0
        Total issues (by confidence):
                Undefined: 0.0
                Low: 0.0
                Medium: 0.0
                High: 0.0
Files skipped (0):

[sobolevn]

Related: https://github.com/tylerwince/flake8-bandit/issues/4