test_pyrepl: test_cursor_back_write() blocks on input_hook() when tests are run sequentially

[vstinner]

Command:

$ ./python -m test -u all,-gui 
(...)
0:24:10 load avg: 0.92 [326/478] test_pyclbr
0:24:13 load avg: 0.92 [327/478] test_pyexpat
0:24:13 load avg: 0.92 [328/478] test_pyrepl

gdb:

(gdb) where
#0  0x00007f3b0325ec37 in select () from /lib64/libc.so.6
#1  0x00007f3af40f4497 in Sleep (milli=<optimized out>) at ./Modules/_tkinter.c:371
#2  0x00007f3af40f4faa in EventHook () at ./Modules/_tkinter.c:3350
#3  0x0000000000666d86 in os__inputhook_impl (module=<optimized out>) at ./Modules/posixmodule.c:16795
#4  0x0000000000666dad in os__inputhook (module=<optimized out>, _unused_ignored=_unused_ignored@entry=0x0)
    at ./Modules/clinic/posixmodule.c.h:12134
(...)

(gdb) py-bt
Traceback (most recent call first):
  <built-in method _inputhook of module object at remote 0x7f3af53d9850>
  File "/home/vstinner/python/main/Lib/_pyrepl/reader.py", line 719, in handle1
    input_hook()
  File "/home/vstinner/python/main/Lib/test/test_pyrepl/support.py", line 75, in handle_all_events
    reader.handle1()
  File "/home/vstinner/python/main/Lib/test/test_pyrepl/test_unix_console.py", line 197, in test_cursor_back_write
    _, con = handle_events_unix_console(events)
  File "/home/vstinner/python/main/Lib/unittest/mock.py", line 1423, in patched
    return func(*newargs, **newkeywargs)
  File "/home/vstinner/python/main/Lib/unittest/case.py", line 606, in _callTestMethod
    result = method()
  File "/home/vstinner/python/main/Lib/unittest/case.py", line 660, in run
    self._callTestMethod(testMethod)
(...)

Linked PRs

• gh-121046
• gh-121049

[vstinner]

Reproducer with less tests:

./python -m test -u all,-gui test_idle test_pyrepl

test_idle sets PyOS_InputHook in:

(gdb) where
#0  0x00007fffe96b927a in EnableEventHook () at ./Modules/_tkinter.c:3380
#1  0x00007fffe96be707 in Tkapp_New (screenName=screenName@entry=0x0, className=className@entry=0x7fffe9cbf588 "Tk", 
    interactive=interactive@entry=0, wantobjects=wantobjects@entry=1, wantTk=wantTk@entry=0, sync=sync@entry=0, use=0x0)
    at ./Modules/_tkinter.c:736
#2  0x00007fffe96be8f5 in _tkinter_create_impl (module=module@entry=<module at remote 0x7fffe96f5a90>, screenName=screenName@entry=0x0, 
    baseName=baseName@entry=0x7fffe8389328 "__main__", className=className@entry=0x7fffe9cbf588 "Tk", interactive=interactive@entry=0, 
    wantobjects=wantobjects@entry=1, wantTk=0, sync=0, use=0x0) at ./Modules/_tkinter.c:3176
#3  0x00007fffe96bed69 in _tkinter_create (module=<module at remote 0x7fffe96f5a90>, args=args@entry=0x7ffff7fba430, nargs=nargs@entry=8)
    at ./Modules/clinic/_tkinter.c.h:820

(gdb) py-bt
Traceback (most recent call first):
  <built-in method create of module object at remote 0x7fffe96f5a90>
  File "/home/vstinner/python/main/Lib/tkinter/__init__.py", line 2459, in __init__
    self.tk = _tkinter.create(screenName, baseName, className, interactive, wantobjects, useTk, sync, use)
  File "/home/vstinner/python/main/Lib/tkinter/__init__.py", line 2572, in Tcl
    return Tk(screenName, baseName, className, useTk)
  File "/home/vstinner/python/main/Lib/idlelib/run.py", line 94, in <module>
    tcl = tkinter.Tcl()
  <built-in method exec of module object at remote 0x7ffff7c94770>
(...)

[vstinner]

cc @pablogsal @ambv @Yhg1s

[terryjreedy]

I cannot reproduce on Windows. After some time, the test call above hung until I hit return, then it was a success.

The apparently offending chain of events is that test_idle runs (idlelib.idle_test.)test_run which imports idlelib.run which calls tkinter.Tcl which at top level calls into _tkinter which sets tcl to get tk events. Idlelib.run is the only place IDLE calls tkinter.Tcl, but I suspect that calls to _tkinter from tkinter.tk do the same. However, IDLE test that call tk() do so explicitly and later call root.destroy(). Since Tcl.destroy exists, a fix might be to add the following top level cleanup function near the top of test_run.

def tearDownModule():
    run.tcl.destroy()

I'm off to bed.

[vstinner]

If I just add tearDownModule() to Lib/idlelib/idle_test/test_run.py, I get:

ERROR: tearDownModule (idlelib.idle_test.test_run)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/python/main/Lib/idlelib/idle_test/test_run.py", line 433, in tearDownModule
    run.tcl.destroy()
    ~~~~~~~~~~~~~~~^^
  File "/home/vstinner/python/main/Lib/tkinter/__init__.py", line 2503, in destroy
    self.tk.call('destroy', self._w)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
_tkinter.TclError: invalid command name "destroy"

[terryjreedy]

Tcl() returns a Tk object created with useTk False instead of True. So the tkinter destroy method is the one called normally.

def Tcl(screenName=None, baseName=None, className='Tk', useTk=False):
    return Tk(screenName, baseName, className, useTk)

But this and perhaps other methods are not valid for such object. This is a tkinter bug.

Alternate try: in idlelib.run, replace lines 94-100

tcl = tkinter.Tcl()

def handle_tk_events(tcl=tcl):
    """Process any tk events that are ready to be dispatched if tkinter
    has been imported, a tcl interpreter has been created and tk has been
    loaded."""
    tcl.eval("update")

with

if idlelib.testing:
    def handle_tk_events(): pass
else:
    tcl = tkinter.Tcl()

    def handle_tk_events(tcl=tcl):
        """Process any tk events that are ready to be dispatched if tkinter
        has been imported, a tcl interpreter has been created and tk has been
        loaded."""
        tcl.eval("update")

This might not be a permanent solution but should work. I cannot test for several hours.

[vstinner]

I wrote a fix based on @terryjreedy's recipe and it works for me!

[vstinner]

Fixed by https://github.com/python/cpython/commit/44eafd66882589d4f4eb569d70c49724da3e9291.

[vstinner]

Thank you @terryjreedy for the fix!

[terryjreedy]

If I ever want to unittest run's handling of interactive tkinter calls, I will ask Serhiy to add a DisableEventHook to _tkinter, if not there already, and make it accessible from Python code, perhaps by making Tcl a subclass that defines a destroy method that works.