IDLE: Ability to run 3rd party code checkers

[22200024-de1a-4081-ad85-2ac04e6b54d2]

BPO	21880
Nosy	@terryjreedy, @taleinat
PRs	python/cpython#9802
Files	3rdpartychecker-v1.diff 3rdpartychecker-v2.diff: Refactored configuration gui and logic into a separate class. Added tests. 3rdpartychecker-v3.diff 3rdpartychecker-v4.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = 'https://github.com/terryjreedy' closed_at = None created_at = labels = ['3.8', 'expert-IDLE', 'type-feature', '3.7'] title = 'IDLE: Ability to run 3rd party code checkers' updated_at = user = 'https://bugs.python.org/SaimadhavHeblikar' ``` bugs.python.org fields: ```python activity = actor = 'terry.reedy' assignee = 'terry.reedy' closed = False closed_date = None closer = None components = ['IDLE'] creation = creator = 'Saimadhav.Heblikar' dependencies = [] files = ['35801', '35821', '35868', '36366'] hgrepos = [] issue_num = 21880 keywords = ['patch'] message_count = 15.0 messages = ['221876', '221938', '222048', '222366', '222367', '222374', '222378', '222393', '225283', '327090', '327165', '327398', '327434', '327496', '327523'] nosy_count = 5.0 nosy_names = ['terry.reedy', 'taleinat', 'SilentGhost', 'jesstess', 'Saimadhav.Heblikar'] pr_nums = ['9802'] priority = 'normal' resolution = None stage = 'patch review' status = 'open' superseder = None type = 'enhancement' url = 'https://bugs.python.org/issue21880' versions = ['Python 3.7', 'Python 3.8'] ```

[22200024-de1a-4081-ad85-2ac04e6b54d2]

(This issue is continuation of http://bugs.python.org/issue18704)

This issue is about a feature to execute any 3rd party code checker from within IDLE.

I am attaching an initial patch(so as to get reviews, is functional logic wise, but missing a lot UI/UX wise.)

It is implemented as an extension.

[terryjreedy]

Read everything, looks plausible ;-). .run_checker assumes api: \<program name> pat_to_something.py \<additional args> I will download pyflakes tomorrow and see if everything works on Windows.

If so, some immediate issues:

1. Only use tempfile if editor is 'dirty'.
2. Allow full path in config for programs not on system path.

[terryjreedy]

I want to broaden this to 'external' programs. Will discuss.

Both versions appear to be trying to access non-existent configuration parameters.

Warning: configHandler.py - IdleConf.GetOption - problem retrieving configuration option 'enabled' from section 'pyflakes'. returning default value: None

Warning: configHandler.py - IdleConf.GetOption - problem retrieving configuration option 'command' from section 'pyflakes'. returning default value: None

v1. Trial on file with no warnings - need end of report summary like find in files so actually know program ran. Trial on file with warnings hangs Idle.

v2. Trying to enable pyflakes gives this
Traceback (most recent call last):
  File "F:\Python\dev\4\py34\lib\tkinter\__init__.py", line 1487, in __call__
    return self.func(*args)
  File "F:\Python\dev\4\py34\lib\idlelib\Checker.py", line 290, in ok
    self.close()
  File "F:\Python\dev\4\py34\lib\idlelib\Checker.py", line 296, in close
    self._checker.update_listbox()
AttributeError: 'function' object has no attribute 'update_listbox'

[22200024-de1a-4081-ad85-2ac04e6b54d2]

In v3, there is no subprocess usage. It imports the checker specific module,does its job and returns the result of processing.

The checker specific files are to be installed from TestPyPI(atleast for now). It has to be installed via pip. It will be detected automatically in IDLE. There will be a feature to pass additional arguments onto the checker(though not yet implemented in this patch).

This patch also supports the feature to modify the editor buffer.

To test out this patch, kindly install two packages pip install -i https://testpypi.python.org/pypi IDLEPyflakes IDLEWhitespaceRemover

(I used the reindent.py file in Tools/scripts in IDLEWhitespaceRemover)

Again, this is more a proof of concept patch. I we are to go ahead in this direction, I will be writing it from scratch again and also with tests.

Checker, is actually a misnomer if we do support the "modify buffer" feature.

[8977102d-e623-4805-b309-cd0ad35b0d72]

This seem like a new feature for IDLE, so I'd imagine it would not be included in either 2.7 or 3.4. Correct me if I'm wrong.

[22200024-de1a-4081-ad85-2ac04e6b54d2]

This seem like a new feature for IDLE, so I'd imagine it would not be >included in either 2.7 or 3.4. Correct me if I'm wrong.

Hi, Yes, it is a new feature. I think it will be included in both 2.7 and 3.4(apart from the latest version 3.5), if my understanding of PEP-434 is correct.

From PEP-434

The PEP would apply to changes in existing features and addition of >small features, such as would require a new menu entry, but not >necessarily to possible major re-writes such as switching to themed >widgets or tabbed windows

Though, I cant say for sure into what category this feature would fall into, i.e. whether it is a "small feature" or not.

[terryjreedy]

Small feature requiring a new menu entry.

[terryjreedy]

I was going to work on this 'today' (well, Saturday), but I injured my eye a bit and cannot see well enough to work on code. I am hoping things will be better after sleeping for a night.

[22200024-de1a-4081-ad85-2ac04e6b54d2]

Attached is a patch which adds capability to work with external programs which can modify the source file(Like whitespace remover tool). It works with all 4 boolean combinations for {show result, reload source}.

The test coverage will be increased, depending on what feature we choose to keep. The GUI is tested thoroughly. The runner code is tested at a basic level.

Please let me know your comments on this.

[taleinat]

It's unfortunate that this has gone dormant for so long. Is anyone interested in picking this up? I'd be happy to provide guidance and feedback.

[terryjreedy]

This issue is specifically based on msg195711 of bpo-18704. Anyone working on this should read it.

Saimadhav's work was part of his Google Summer of Code (GSOC) project, which ended soon after V4 was submitted. I recorded reviews of V1 and V2 above. I don't remember which tests and reviews, if any, I did with V3 and V4.

Some needed changes starting with v4: Checker.py should be checker.py. Implement it as a feature, not an extension. Access the in-memory config object for .idlrc/checker.cfg directly rather than through idleConf. idleConf accesses the fixed defaults and mutable user overrides as if they are one config. I am a bit surprised that idleConf worked without an empty idlelib/config-checker.def.

The main blocker was and is keeping the GUI responsive while the 3rd party program is executing. V1 & V2 used subprocess through a pipe. V3 did not use subprocess. V4 uses subprocess without a pipe. It has this blocking polling loop: while process.poll() is None: continue

If a 3rd party program is expected to revise a file, the corresponding editor should be read-only for the duration.

I intended that any issue like this should start with a coherent specification separate from the code. A doc patch is needed and that might be enough.

Since this issue was opened, it has been more firmly stated that the stdlib should not have any hard-coded dependencies on 3rd party code. In April 2016, the proposal for a GSOC project to add a GUI front end for pip got no opposition and 2 overt approvals on pydev. In August 2016, the result was rejected by the release manager and one of the additional approvers because it necessarily used the (public) pip (command-line) interface.

Not withstanding that, there could be a separate idle-checker repository containing a checker.cfg with entries for multiple checkers. Such a file would be needed to do manual tests with multiple checkers. This could include a typing annotation checker, like mypy, which is a new type of code checker added since this issue was created.

Tal, what do you think is the easiest way to turn a .diff on the tracker into a git master branch?

[taleinat]

I'll get this up on a git branch so that we can continue hacking on it.

[terryjreedy]

Idlelib modules OutputWindow and configHandler are now outwin and config.

I discovered a non-checker .py file consumer. Pygame Zero enables use of pygame to create games without boilerplate. https://pygame-zero.readthedocs.io/en/stable/ide-mode.html One can run a mygame.py file containing, for instance,

WIDTH = 800
HEIGHT = 600

def draw():
    screen.clear()
    screen.draw.circle((400, 300), 30, 'white')

from a command line with 'pgzrun mygame.py'.

In this case, one can instead run from an IDE by adding boilerplate lines 'import pgzrun' and 'pgzrun.go()' at top and bottom. But there might be people (or instructors) who prefer, if possible, to enable pgzrun as a 3rd party .py file consumer with the 'checker' option.

One issue with configparser is that it does not read comments in a .cfg file and therefore overwrites them when overwriting a file. We might add a multiline 'comment' option to each application section.

[taleinat]

Update: I've nearly got an updated version ready and working based on the current master branch. I hope to have a PR up by tomorrow.

[taleinat]

See PR9802.