CPU usage at 100% on all cores

[stuaxo]

Hi, I added flake8-black to poetry, it's been running for over a 5 minutes on my small project and 8 workers on 100% CPU - I'm guess this is not expected behaviour ?

So - I guess my config is wrong and it's stuck in a loop - this is probably something that still shouldn't be possible though ?

[peterjc]

That does sound wrong. I take it flake8 and your other plugins worked fine?

Is your small project online? That would be the easiest way for me to try to solve this.

Can you share your flake8 and black configuration files, and the command line used to run flake8?

[stuaxo]

Thanks, I'm 99.999% it's wrong configuration on my part - I'll get out it up later when I'm in front of that computer.

[peterjc]

If you can work out what was wrong, please do let me know - I might be able to guard against it. This is a very poor way for the plugin to fail (even if given bad input - which may not be the case).

[stuaxo]

Hi, The nox branch of my project here demonstrates the issue https://github.com/shoebot/toybot-scenegraph/tree/feature/nox

It's possible to trigger it, by running flake8.

S

[peterjc]

Frustratingly it works for me, at least with my local copy of flake8 and its plugins:

$ git clone https://github.com/shoebot/toybot-scenegraph.git
...
$ cd toybot-scenegraph.
$ git checkout -t origin/feature/nox
Branch 'feature/nox' set up to track remote branch 'feature/nox' from 'origin'.
Switched to a new branch 'feature/nox'

$ flake8
./noxfile.py:1:1: D100 Missing docstring in public module
./noxfile.py:8:11: Q000 Remove bad quotes
./noxfile.py:9:25: Q000 Remove bad quotes
./noxfile.py:10:1: E265 block comment should start with '# '
./noxfile.py:10:2: BLK100 Black would make changes.
...

This could be down to the versions, and/or a conflict with another plugin. I used:

$ flake8 --version
3.7.9 (assertive: 1.2.1, black: 0.1.1, flake8-blind-except: 0.1.1, flake8-bugbear: 19.8.0, flake8-comprehensions: 3.1.4, flake8-docstrings: 1.5.0, pydocstyle: 5.0.1, flake8-pie: 0.4.2, flake8-sfs: 0.0.2, flake8_implicit_str_concat: 0.1.0, flake8_quotes: 2.1.1, mccabe: 0.6.1, pycodestyle: 2.5.0, pyflakes: 2.1.1, rst-docstrings: 0.0.13) CPython 3.7.3 on Darwin

Using flake8 --bug-report is even more detailed. What version do you have? I could have a guess from your [tool.poetry.dev-dependencies] setting in pyproject.toml but easier to confirm.

[stuaxo]

Here is the output of flake8 --bug-report.

I tried getting it to recreate the virtualenv and was able to reproduce it.

The biggest difference is probably that I am running on Linux and you on OSX. ?

{
  "dependencies": [
    {
      "dependency": "entrypoints",
      "version": "0.3"
    }
  ],
  "platform": {
    "python_implementation": "CPython",
    "python_version": "3.7.5",
    "system": "Linux"
  },
  "plugins": [
    {
      "is_local": false,
      "plugin": "black",
      "version": "0.1.1"
    },
    {
      "is_local": false,
      "plugin": "flake8-annotations",
      "version": "1.1.3"
    },
    {
      "is_local": false,
      "plugin": "flake8-bandit",
      "version": "2.1.2"
    },
    {
      "is_local": false,
      "plugin": "flake8-bugbear",
      "version": "20.1.3"
    },
    {
      "is_local": false,
      "plugin": "flake8-docstrings",
      "version": "1.5.0, pydocstyle: 5.0.2"
    },
    {
      "is_local": false,
      "plugin": "import-order",
      "version": "0.18.1"
    },
    {
      "is_local": false,
      "plugin": "mccabe",
      "version": "0.6.1"
    },
    {
      "is_local": false,
      "plugin": "pycodestyle",
      "version": "2.5.0"
    },
    {
      "is_local": false,
      "plugin": "pyflakes",
      "version": "2.1.1"
    }
  ],
  "version": "3.7.9"
}

[peterjc]

(Updated - original comment has a typo in the Python version and was missing one plugin)

Still on macOS, with Python 3.7.4, but with matching versions of flake8 and its plugins:

$ flake8 --version
3.7.9 (black: 0.1.1, flake8-annotations: 1.1.3, flake8-bandit: 2.1.2, flake8-bugbear: 20.1.3, flake8-docstrings: 1.5.0, pydocstyle: 5.0.2, import-order: 0.18.1, mccabe: 0.6.1, pycodestyle: 2.5.0, pyflakes: 2.1.1) CPython 3.7.4 on Darwin

$ flake8
./noxfile.py:1:1: D100 Missing docstring in public module
./noxfile.py:10:1: E265 block comment should start with '# '
./noxfile.py:10:2: BLK100 Black would make changes.
...

That works for me. Time to try Linux....

[peterjc]

Running out of ideas now, Linux with Python 3.7.6:

$ flake8 --version
3.7.9 (black: 0.1.1, flake8-annotations: 1.1.3, flake8-bandit: 2.1.2, flake8-bugbear: 20.1.3, flake8-docstrings: 1.5.0, pydocstyle: 5.0.2, import-order: 0.18.1, mccabe: 0.6.1, pycodestyle: 2.5.0, pyflakes: 2.1.1) CPython 3.7.6 on Linux

$ flake8
./noxfile.py:1:1: D100 Missing docstring in public module
./noxfile.py:10:1: E265 block comment should start with '# '
./noxfile.py:10:2: BLK100 Black would make changes.
...

Could be the exact flavour of Linux. Could be the exact version of Python 3.7. Neither seems likely.

Could it be the filesystem? I have had flake8 stall on our cluster when trying to examine lots of files and the storage layer was under stress.

Could you double check locally on a fresh clone and checkout in git (just in case there are some local files which are triggering this)?

[peterjc]

The verbose logging has highlighted that each worker loads the black settings.

This macOS machine has 4 cores; so did this Linux machine - so it used four workers:

$ flake8 -v
flake8.plugins.manager    MainProcess    537 INFO     Loading entry-points for "flake8.extension".
flake8.plugins.manager    MainProcess    862 INFO     Loading entry-points for "flake8.report".
flake8.plugins.manager    MainProcess   1157 INFO     Loading plugin "B" from entry-point.
flake8.plugins.manager    MainProcess   1277 INFO     Loading plugin "S" from entry-point.
flake8.plugins.manager    MainProcess   2613 INFO     Loading plugin "C90" from entry-point.
flake8.plugins.manager    MainProcess   2621 INFO     Loading plugin "TYP" from entry-point.
flake8.plugins.manager    MainProcess   2676 INFO     Loading plugin "D" from entry-point.
flake8.plugins.manager    MainProcess   2955 INFO     Loading plugin "F" from entry-point.
flake8.plugins.manager    MainProcess   3111 INFO     Loading plugin "pycodestyle.ambiguous_identifier" from entry-point.
flake8.plugins.manager    MainProcess   3111 INFO     Loading plugin "pycodestyle.bare_except" from entry-point.
flake8.plugins.manager    MainProcess   3111 INFO     Loading plugin "pycodestyle.blank_lines" from entry-point.
flake8.plugins.manager    MainProcess   3111 INFO     Loading plugin "pycodestyle.break_after_binary_operator" from entry-point.
flake8.plugins.manager    MainProcess   3111 INFO     Loading plugin "pycodestyle.break_before_binary_operator" from entry-point.
flake8.plugins.manager    MainProcess   3111 INFO     Loading plugin "pycodestyle.comparison_negative" from entry-point.
flake8.plugins.manager    MainProcess   3111 INFO     Loading plugin "pycodestyle.comparison_to_singleton" from entry-point.
flake8.plugins.manager    MainProcess   3111 INFO     Loading plugin "pycodestyle.comparison_type" from entry-point.
flake8.plugins.manager    MainProcess   3111 INFO     Loading plugin "pycodestyle.compound_statements" from entry-point.
flake8.plugins.manager    MainProcess   3111 INFO     Loading plugin "pycodestyle.continued_indentation" from entry-point.
flake8.plugins.manager    MainProcess   3111 INFO     Loading plugin "pycodestyle.explicit_line_join" from entry-point.
flake8.plugins.manager    MainProcess   3111 INFO     Loading plugin "pycodestyle.extraneous_whitespace" from entry-point.
flake8.plugins.manager    MainProcess   3111 INFO     Loading plugin "pycodestyle.imports_on_separate_lines" from entry-point.
flake8.plugins.manager    MainProcess   3111 INFO     Loading plugin "pycodestyle.indentation" from entry-point.
flake8.plugins.manager    MainProcess   3112 INFO     Loading plugin "pycodestyle.maximum_doc_length" from entry-point.
flake8.plugins.manager    MainProcess   3112 INFO     Loading plugin "pycodestyle.maximum_line_length" from entry-point.
flake8.plugins.manager    MainProcess   3112 INFO     Loading plugin "pycodestyle.missing_whitespace" from entry-point.
flake8.plugins.manager    MainProcess   3112 INFO     Loading plugin "pycodestyle.missing_whitespace_after_import_keyword" from entry-point.
flake8.plugins.manager    MainProcess   3112 INFO     Loading plugin "pycodestyle.missing_whitespace_around_operator" from entry-point.
flake8.plugins.manager    MainProcess   3112 INFO     Loading plugin "pycodestyle.module_imports_on_top_of_file" from entry-point.
flake8.plugins.manager    MainProcess   3112 INFO     Loading plugin "pycodestyle.python_3000_async_await_keywords" from entry-point.
flake8.plugins.manager    MainProcess   3112 INFO     Loading plugin "pycodestyle.python_3000_backticks" from entry-point.
flake8.plugins.manager    MainProcess   3112 INFO     Loading plugin "pycodestyle.python_3000_has_key" from entry-point.
flake8.plugins.manager    MainProcess   3112 INFO     Loading plugin "pycodestyle.python_3000_invalid_escape_sequence" from entry-point.
flake8.plugins.manager    MainProcess   3112 INFO     Loading plugin "pycodestyle.python_3000_not_equal" from entry-point.
flake8.plugins.manager    MainProcess   3112 INFO     Loading plugin "pycodestyle.python_3000_raise_comma" from entry-point.
flake8.plugins.manager    MainProcess   3112 INFO     Loading plugin "pycodestyle.tabs_obsolete" from entry-point.
flake8.plugins.manager    MainProcess   3112 INFO     Loading plugin "pycodestyle.tabs_or_spaces" from entry-point.
flake8.plugins.manager    MainProcess   3112 INFO     Loading plugin "pycodestyle.trailing_blank_lines" from entry-point.
flake8.plugins.manager    MainProcess   3112 INFO     Loading plugin "pycodestyle.trailing_whitespace" from entry-point.
flake8.plugins.manager    MainProcess   3112 INFO     Loading plugin "pycodestyle.whitespace_around_comma" from entry-point.
flake8.plugins.manager    MainProcess   3112 INFO     Loading plugin "pycodestyle.whitespace_around_keywords" from entry-point.
flake8.plugins.manager    MainProcess   3113 INFO     Loading plugin "pycodestyle.whitespace_around_named_parameter_equals" from entry-point.
flake8.plugins.manager    MainProcess   3113 INFO     Loading plugin "pycodestyle.whitespace_around_operator" from entry-point.
flake8.plugins.manager    MainProcess   3113 INFO     Loading plugin "pycodestyle.whitespace_before_comment" from entry-point.
flake8.plugins.manager    MainProcess   3113 INFO     Loading plugin "pycodestyle.whitespace_before_parameters" from entry-point.
flake8.plugins.manager    MainProcess   3113 INFO     Loading plugin "BLK" from entry-point.
flake8.plugins.manager    MainProcess   3622 INFO     Loading plugin "I" from entry-point.
flake8.plugins.manager    MainProcess   3673 INFO     Loading plugin "default" from entry-point.
flake8.plugins.manager    MainProcess   3697 INFO     Loading plugin "pylint" from entry-point.
flake8.plugins.manager    MainProcess   3697 INFO     Loading plugin "quiet-filename" from entry-point.
flake8.plugins.manager    MainProcess   3697 INFO     Loading plugin "quiet-nothing" from entry-point.
flake8                    MainProcess   3700 INFO     flake8-black: No black configuration set
flake8.checker            MainProcess   3700 INFO     Making checkers
flake8.checker            MainProcess   3732 INFO     Checking 9 files
flake8                    ForkPoolWorker-3   3808 INFO     flake8-black: loading black settings from /mnt/shared/users/xxx/repositories/toybot-scenegraph/pyproject.toml
flake8                    ForkPoolWorker-2   3817 INFO     flake8-black: loading black settings from /mnt/shared/users/xxx/repositories/toybot-scenegraph/pyproject.toml
flake8                    ForkPoolWorker-4   3864 INFO     flake8-black: loading black settings from /mnt/shared/users/xxx/repositories/toybot-scenegraph/pyproject.toml
flake8                    ForkPoolWorker-1   3874 INFO     flake8-black: loading black settings from /mnt/shared/users/xxx/repositories/toybot-scenegraph/pyproject.toml
flake8.main.application   MainProcess   4064 INFO     Finished running
flake8.main.application   MainProcess   4065 INFO     Reporting errors
./noxfile.py:1:1: D100 Missing docstring in public module
./noxfile.py:10:1: E265 block comment should start with '# '
./noxfile.py:10:2: BLK100 Black would make changes.
...

Also tried it on Linux with lots of cores - enough for one worker per file:

$ flake8 -v
flake8.plugins.manager    MainProcess    709 INFO     Loading entry-points for "flake8.extension".
flake8.plugins.manager    MainProcess   1182 INFO     Loading entry-points for "flake8.report".
flake8.plugins.manager    MainProcess   1537 INFO     Loading plugin "B" from entry-point.
flake8.plugins.manager    MainProcess   1670 INFO     Loading plugin "S" from entry-point.
flake8.plugins.manager    MainProcess   3149 INFO     Loading plugin "C90" from entry-point.
flake8.plugins.manager    MainProcess   3161 INFO     Loading plugin "TYP" from entry-point.
flake8.plugins.manager    MainProcess   3238 INFO     Loading plugin "D" from entry-point.
flake8.plugins.manager    MainProcess   3586 INFO     Loading plugin "F" from entry-point.
flake8.plugins.manager    MainProcess   3781 INFO     Loading plugin "pycodestyle.ambiguous_identifier" from entry-point.
flake8.plugins.manager    MainProcess   3781 INFO     Loading plugin "pycodestyle.bare_except" from entry-point.
flake8.plugins.manager    MainProcess   3781 INFO     Loading plugin "pycodestyle.blank_lines" from entry-point.
flake8.plugins.manager    MainProcess   3781 INFO     Loading plugin "pycodestyle.break_after_binary_operator" from entry-point.
flake8.plugins.manager    MainProcess   3781 INFO     Loading plugin "pycodestyle.break_before_binary_operator" from entry-point.
flake8.plugins.manager    MainProcess   3781 INFO     Loading plugin "pycodestyle.comparison_negative" from entry-point.
flake8.plugins.manager    MainProcess   3781 INFO     Loading plugin "pycodestyle.comparison_to_singleton" from entry-point.
flake8.plugins.manager    MainProcess   3781 INFO     Loading plugin "pycodestyle.comparison_type" from entry-point.
flake8.plugins.manager    MainProcess   3782 INFO     Loading plugin "pycodestyle.compound_statements" from entry-point.
flake8.plugins.manager    MainProcess   3782 INFO     Loading plugin "pycodestyle.continued_indentation" from entry-point.
flake8.plugins.manager    MainProcess   3782 INFO     Loading plugin "pycodestyle.explicit_line_join" from entry-point.
flake8.plugins.manager    MainProcess   3782 INFO     Loading plugin "pycodestyle.extraneous_whitespace" from entry-point.
flake8.plugins.manager    MainProcess   3782 INFO     Loading plugin "pycodestyle.imports_on_separate_lines" from entry-point.
flake8.plugins.manager    MainProcess   3782 INFO     Loading plugin "pycodestyle.indentation" from entry-point.
flake8.plugins.manager    MainProcess   3782 INFO     Loading plugin "pycodestyle.maximum_doc_length" from entry-point.
flake8.plugins.manager    MainProcess   3782 INFO     Loading plugin "pycodestyle.maximum_line_length" from entry-point.
flake8.plugins.manager    MainProcess   3782 INFO     Loading plugin "pycodestyle.missing_whitespace" from entry-point.
flake8.plugins.manager    MainProcess   3782 INFO     Loading plugin "pycodestyle.missing_whitespace_after_import_keyword" from entry-point.
flake8.plugins.manager    MainProcess   3782 INFO     Loading plugin "pycodestyle.missing_whitespace_around_operator" from entry-point.
flake8.plugins.manager    MainProcess   3782 INFO     Loading plugin "pycodestyle.module_imports_on_top_of_file" from entry-point.
flake8.plugins.manager    MainProcess   3783 INFO     Loading plugin "pycodestyle.python_3000_async_await_keywords" from entry-point.
flake8.plugins.manager    MainProcess   3783 INFO     Loading plugin "pycodestyle.python_3000_backticks" from entry-point.
flake8.plugins.manager    MainProcess   3783 INFO     Loading plugin "pycodestyle.python_3000_has_key" from entry-point.
flake8.plugins.manager    MainProcess   3783 INFO     Loading plugin "pycodestyle.python_3000_invalid_escape_sequence" from entry-point.
flake8.plugins.manager    MainProcess   3783 INFO     Loading plugin "pycodestyle.python_3000_not_equal" from entry-point.
flake8.plugins.manager    MainProcess   3783 INFO     Loading plugin "pycodestyle.python_3000_raise_comma" from entry-point.
flake8.plugins.manager    MainProcess   3783 INFO     Loading plugin "pycodestyle.tabs_obsolete" from entry-point.
flake8.plugins.manager    MainProcess   3783 INFO     Loading plugin "pycodestyle.tabs_or_spaces" from entry-point.
flake8.plugins.manager    MainProcess   3783 INFO     Loading plugin "pycodestyle.trailing_blank_lines" from entry-point.
flake8.plugins.manager    MainProcess   3783 INFO     Loading plugin "pycodestyle.trailing_whitespace" from entry-point.
flake8.plugins.manager    MainProcess   3783 INFO     Loading plugin "pycodestyle.whitespace_around_comma" from entry-point.
flake8.plugins.manager    MainProcess   3783 INFO     Loading plugin "pycodestyle.whitespace_around_keywords" from entry-point.
flake8.plugins.manager    MainProcess   3783 INFO     Loading plugin "pycodestyle.whitespace_around_named_parameter_equals" from entry-point.
flake8.plugins.manager    MainProcess   3784 INFO     Loading plugin "pycodestyle.whitespace_around_operator" from entry-point.
flake8.plugins.manager    MainProcess   3784 INFO     Loading plugin "pycodestyle.whitespace_before_comment" from entry-point.
flake8.plugins.manager    MainProcess   3784 INFO     Loading plugin "pycodestyle.whitespace_before_parameters" from entry-point.
flake8.plugins.manager    MainProcess   3784 INFO     Loading plugin "BLK" from entry-point.
flake8.plugins.manager    MainProcess   4398 INFO     Loading plugin "I" from entry-point.
flake8.plugins.manager    MainProcess   4453 INFO     Loading plugin "default" from entry-point.
flake8.plugins.manager    MainProcess   4484 INFO     Loading plugin "pylint" from entry-point.
flake8.plugins.manager    MainProcess   4484 INFO     Loading plugin "quiet-filename" from entry-point.
flake8.plugins.manager    MainProcess   4484 INFO     Loading plugin "quiet-nothing" from entry-point.
flake8                    MainProcess   4489 INFO     flake8-black: No black configuration set
flake8.checker            MainProcess   4490 INFO     Making checkers
flake8.checker            MainProcess   4525 INFO     Checking 9 files
flake8                    ForkPoolWorker-9   4660 INFO     flake8-black: loading black settings from /mnt/shared/users/xxx/repositories/toybot-scenegraph/pyproject.toml
flake8                    ForkPoolWorker-3   4661 INFO     flake8-black: loading black settings from /mnt/shared/users/xxx/repositories/toybot-scenegraph/pyproject.toml
flake8                    ForkPoolWorker-7   4663 INFO     flake8-black: loading black settings from /mnt/shared/users/xxx/repositories/toybot-scenegraph/pyproject.toml
flake8                    ForkPoolWorker-2   4665 INFO     flake8-black: loading black settings from /mnt/shared/users/xxx/repositories/toybot-scenegraph/pyproject.toml
flake8                    ForkPoolWorker-6   4666 INFO     flake8-black: loading black settings from /mnt/shared/users/xxx/repositories/toybot-scenegraph/pyproject.toml
flake8                    ForkPoolWorker-5   4674 INFO     flake8-black: loading black settings from /mnt/shared/users/xxx/repositories/toybot-scenegraph/pyproject.toml
flake8                    ForkPoolWorker-8   4676 INFO     flake8-black: loading black settings from /mnt/shared/users/xxx/repositories/toybot-scenegraph/pyproject.toml
flake8                    ForkPoolWorker-1   4707 INFO     flake8-black: loading black settings from /mnt/shared/users/xxx/repositories/toybot-scenegraph/pyproject.toml
flake8.main.application   MainProcess   4838 INFO     Finished running
flake8.main.application   MainProcess   4838 INFO     Reporting errors
./noxfile.py:1:1: D100 Missing docstring in public module
./noxfile.py:10:1: E265 block comment should start with '# '
./noxfile.py:10:2: BLK100 Black would make changes.
...

The fact that each worker loads the black configuration does not in itself seem to be a problem - although it would be nicer to do this once if possible.

This made me notice your example looks to be on an 8 CPU machine (so would have 8 workers and load the black config 8 times), but it was checking over 4296 files(!)

There seem to be a lot of stray Python files on your system - the cumulative run time could explain flake8 apparently getting stuck at 100% CPU.

[stuaxo]

I'm not in front of that computer now, but had a quick look through the code on flake8-black on my phone, and can't see anything that could cause this.

I'd like to add some logging (or just print statements), to see where flake8-black gets to and try again, got any ideas the best places I could put that ?

Hm - I just noticed all the files, that is weird - I wonder where they could be coming from ? Could it be because I told it to check the path "." ?

4 cores / 8 threads, the modern way of counting is weird to me - amazing what you can get in a mid-range laptop now !

[peterjc]

Writing to sys.stderr does not seem to work from within a flake8 plugin, I assume that flake8 buffers that somehow. Just add more logging like this:

https://github.com/peterjc/flake8-black/blob/v0.1.1/flake8_black.py#L56

[stuaxo]

Turns out there was an old virtualenv in my root folder, along with some other stuff I wasn't excluding - hence all those files.

Thanks for your patience, and pointing me in the right direction - also have a bunch more flake8 plugins to try now too.

[peterjc]

Thanks for your patience too - I'd have been rather frustrated if we hadn't been able to get to the bottom of this in the end 👍

[avylove]

I had a similar experience with nox when changing flake8 versions without regenerating the virtual environment. Is there a version-dependent cache?

[peterjc]

When run directly, black makes a cache of the results for the input files, so repeat runs of black are very fast. It does not via the flake8 plugin, see #26. This can matter if you are repeatedly checking all your files (or a complex large file).

If you are asking about nox and if it caches the software for flake8, plugins, etc in virtual environments - I don't know.