Suggestion for handling CPU resources better with concurrent lints

d3v-null commented 7 years ago

Hey! Firstly thank you for your work on linter so far, you've done an awesome job on v2 and should be immensely proud to have created such a polished and usable open source linter. Considering how many people are getting in to programming through languages like Python and using Atom, the fact that atom linter works so well out of the box (which wasn't a thing when I learnt to code on Notepad++ 10 years ago) is probably going to have a noticeable effect on code quality for the next generation of programmers. You rock 👍

Now, on to my suggestion: Something I've noticed is that if I have to refactor a large python repository, the default settings will cause huge CPU usage, if it takes a few seconds to run pylint on the file, even if that file is the only open file. This is because linter will keep calling the lint process, even before the previous lint has had time to finish, causing lints to stack up and snowball until the editor crawls to a halt. Even when according to that swirly blue/green dot in atom the lint has finished, the CPU usage stays very high on multiple cores. my When I turn off auto linting and lint manually, my CPU usage stays reasonable and only uses a few cores. The default settings make atom unusable on my computer for large files, and considering I have a pretty decent laptop, this would be a much more noticeable issue on a low powered device. When this happens I usually give up out of frustration and go back to Sublime which seems to handle lints just fine but the linter interface makes me miss atom-linter (it uses Anaconda)

What if linter either cancelled the previous lint or waited til the lint had finished / timed out before starting a new lint?

I'd love to help implement this feature. I've got a strong background in Python, my JS is a bit rusty but I should be able to figure it out, and I don't have much on this weekend. If you think I should take a crack at it, I'd love a quick brief on where to start.

Here are some CPU screenshots with and witout autolint:

I was going to send you the file that I had trouble linting, but it turns out it's not the file itself but the giant list of imports it brings in. If you want to try and recreate my problem, download this repo and open woogenerator/generator.py https://github.com/derwentx/WooGenerator . Sorry about how bloated the repo is. It's big but I've definitely had to use atom to trawl through larger repos.

Here are my specs:

> apm --version
apm  1.15.3
npm  3.10.5
node 4.4.5 x64
python 2.7.13
git 2.10.1
> apm list
Built-in Atom Packages (89)
├── atom-dark-syntax@0.28.0
├── atom-dark-ui@0.53.0
├── atom-light-syntax@0.29.0
├── atom-light-ui@0.46.0
├── base16-tomorrow-dark-theme@1.5.0
├── base16-tomorrow-light-theme@1.5.0
├── one-dark-ui@1.9.1
├── one-light-ui@1.9.1
├── one-dark-syntax@1.7.1
├── one-light-syntax@1.7.1
├── solarized-dark-syntax@1.1.2
├── solarized-light-syntax@1.1.2
├── about@1.7.2
├── archive-view@0.62.2
├── autocomplete-atom-api@0.10.0
├── autocomplete-css@0.15.0
├── autocomplete-html@0.7.2
├── autocomplete-plus@2.34.2
├── autocomplete-snippets@1.11.0
├── autoflow@0.29.0
├── autosave@0.24.0
├── background-tips@0.26.1
├── bookmarks@0.44.1
├── bracket-matcher@0.85.2
├── command-palette@0.40.1
├── deprecation-cop@0.56.2
├── dev-live-reload@0.47.0
├── encoding-selector@0.23.1
├── exception-reporting@0.41.0
├── find-and-replace@0.206.3
├── fuzzy-finder@1.4.1
├── git-diff@1.3.1
├── go-to-line@0.32.0
├── grammar-selector@0.49.1
├── image-view@0.60.0
├── incompatible-packages@0.26.1
├── keybinding-resolver@0.36.1
├── line-ending-selector@0.6.1
├── link@0.31.2
├── markdown-preview@0.159.6
├── metrics@1.1.3
├── notifications@0.66.2
├── open-on-github@1.2.1
├── package-generator@1.1.0
├── settings-view@0.247.0
├── snippets@1.0.5
├── spell-check@0.70.2
├── status-bar@1.8.1
├── styleguide@0.49.2
├── symbols-view@0.114.0
├── tabs@0.104.1
├── timecop@0.34.0
├── tree-view@0.214.1
├── update-package-dependencies@0.10.0
├── welcome@0.36.0
├── whitespace@0.36.2
├── wrap-guide@0.39.0
├── language-c@0.56.0
├── language-clojure@0.22.2
├── language-coffee-script@0.48.5
├── language-csharp@0.14.2
├── language-css@0.42.0
├── language-gfm@0.88.1
├── language-git@0.19.0
├── language-go@0.43.1
├── language-html@0.47.2
├── language-hyperlink@0.16.1
├── language-java@0.26.0
├── language-javascript@0.126.1
├── language-json@0.18.3
├── language-less@0.30.1
├── language-make@0.22.3
├── language-mustache@0.13.1
├── language-objective-c@0.15.1
├── language-perl@0.37.0
├── language-php@0.37.4
├── language-property-list@0.9.0
├── language-python@0.45.2
├── language-ruby@0.70.5
├── language-ruby-on-rails@0.25.2
├── language-sass@0.57.1
├── language-shellscript@0.25.0
├── language-source@0.9.0
├── language-sql@0.25.3
├── language-text@0.7.1
├── language-todo@0.29.1
├── language-toml@0.18.1
├── language-xml@0.34.16
└── language-yaml@0.28.0

Community Packages (64) /Users/Derwent/.atom/packages
├── Parinfer@1.17.0
├── SFTP-deployment@1.0.2
├── Sublime-Style-Column-Selection@1.7.4
├── atom-beautify@0.29.17
├── atom-sonic@0.3.2
├── autocomplete-clang@0.10.0
├── autocomplete-haskell@0.7.2
├── autocomplete-php@0.3.7
├── autocomplete-python@1.8.60
├── autocomplete-wordpress-hooks@0.2.2
├── build@0.67.0
├── busy@0.7.0
├── busy-signal@1.3.0
├── center-line@1.2.4
├── change-case@0.6.3
├── chester-atom-syntax@0.1.1
├── dash@1.7.1
├── drunken-php@0.2.3
├── elm@0.1.0
├── file-icons@2.0.17
├── grammar-token-limit@0.1.1
├── haskell-ghc-mod@1.19.7
├── hex@0.6.2
├── highlight-selected@0.12.0
├── ide-haskell@1.9.6
├── ide-haskell-cabal@1.9.0
├── ide-haskell-repl@0.6.0
├── ink@0.6.5
├── intentions@1.1.2
├── jumpy@3.1.3
├── language-assembly@0.7.1
├── language-elm@1.5.0
├── language-haskell@1.12.1
├── language-ini@1.19.0
├── language-lisp@0.2.0
├── language-lua@0.9.11
├── language-r@0.4.1
├── language-x86asm@0.9.0
├── lines@0.13.1
├── linter@2.1.0
├── linter-gcc@0.7.1
├── linter-php@1.3.1
├── linter-pylama@0.8.14
├── linter-pylint@1.2.2
├── linter-ui-default@1.1.0
├── lisp-paredit@0.5.5
├── minimap@4.26.8
├── platformio-ide@1.7.3
├── platformio-ide-terminal@2.5.0
├── proto-repl@1.4.18
├── python-indent@1.0.1
├── python-tools@0.6.9
├── python-yapf@0.13.0
├── refactor@0.11.3
├── remote-sync@4.1.5
├── sane-indentation@0.1.2
├── script@3.14.1
├── sepia-syntax@1.1.0
├── set-syntax@0.3.2
├── sort-lines@0.14.0
├── split-diff@1.2.0
├── swackets@0.26.0
├── term3@0.22.1
└── tool-bar@1.1.0
> pylint --version
pylint 1.5.5,
astroid 1.4.5
Python 2.7.10 (default, Jul 30 2016, 19:40:32)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)]

System:

macOS Sierra 10.12.3 (16D32)
MacBook Pro (Retina, 15-inch, Late 2013)
2.6 GHz Intel Core i7
16 GB 1600 MHz DDR3
NVIDIA GeForce GT 750M 2048 MB
Intel Iris Pro 1536 MB
1TB PCIE SSD

steelbrain commented 7 years ago

I think we can allow for terminating previously spawned processes for cpu intensive linters in the helper module atom-linter. I don't think this should be the default behavior for all linters, but only for cpu intensive ones.

@Arcanemagus thoughts?

Arcanemagus commented 7 years ago

Isn't the functionality for this basically already implemented in exec with the .kill() functionality?

Something along these lines should be possible already:

Add any exec calls to a Set
Call .kill(-9) on any unresolved Promises in the Set
Start the current process
If the process gets killed, return null to Linter so it doesn't change any current results (of the new call to lint())

None of the providers currently work this way though 😛.

Arcanemagus commented 7 years ago

Turns out that there are some additional things preventing this from being done in providers just using exec(), so this is blocked on https://github.com/steelbrain/exec/issues/81 for now.

steelbrain commented 7 years ago

Closing in favor of https://github.com/steelbrain/atom-linter/issues/159

steelbrain / linter

Suggestion for handling CPU resources better with concurrent lints #1373