with statements are not ensuring that __exit__ is called if __enter__ succeeds

[njsmith]

BPO	29988
Nosy	@rhettinger, @gpshead, @ncoghlan, @nirs, @njsmith, @markshannon, @embray, @serhiy-storchaka, @jdemeyer, @1st1, @ryanhiebert, @xgid, @jack1142
PRs	python/cpython#1799 python/cpython#18334
Dependencies	bpo-31344: f_trace_opcodes frame attribute to switch to per-opcode tracing

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = created_at = labels = ['interpreter-core', 'type-bug', '3.9', '3.10'] title = 'with statements are not ensuring that __exit__ is called if __enter__ succeeds' updated_at = user = 'https://github.com/njsmith' ``` bugs.python.org fields: ```python activity = actor = 'Mark.Shannon' assignee = 'none' closed = True closed_date = closer = 'Mark.Shannon' components = ['Interpreter Core'] creation = creator = 'njs' dependencies = ['31344'] files = [] hgrepos = [] issue_num = 29988 keywords = ['patch'] message_count = 60.0 messages = ['291148', '291157', '294272', '294296', '294400', '294402', '294404', '294434', '297177', '297179', '297180', '297182', '297185', '297186', '297189', '297190', '301235', '301248', '301274', '301277', '301278', '301285', '301289', '301360', '301429', '301553', '301557', '301566', '301573', '301601', '301609', '301610', '301611', '301620', '301630', '301660', '301667', '301668', '301672', '301675', '301682', '301698', '301717', '301721', '301771', '301869', '312874', '321225', '321318', '350123', '350134', '352026', '352039', '352199', '352212', '352457', '353415', '372499', '386711', '389922'] nosy_count = 15.0 nosy_names = ['rhettinger', 'gregory.p.smith', 'ncoghlan', 'nirs', 'njs', 'Mark.Shannon', 'erik.bray', 'serhiy.storchaka', 'jdemeyer', 'yselivanov', 'deleted0524', 'ryanhiebert', 'xgdomingo', 'jack1142', 'Ami Fischman'] pr_nums = ['1799', '18334'] priority = 'normal' resolution = 'fixed' stage = 'resolved' status = 'closed' superseder = None type = 'behavior' url = 'https://bugs.python.org/issue29988' versions = ['Python 3.9', 'Python 3.10'] ```

[gpshead]

The signal handler in this case is CPython's internal signal handling system thus any such onus falls on us...

The great signal handling hack of "set a flag that the interpreter loop checks on occasion" trick lasted a long time, but our VM has since evolved to be much more complicated than every bytecode being a safe interrupt point.

FYI - Most of the in-room conversation we tried to capture on this issue thread was at the core dev sprint a couple years ago (Sept 2017). It'll similarly take me a while to swap this context back in.

[ncoghlan]

There is also the draft test case at https://github.com/ncoghlan/cpython/pull/2

That needs updating to account for the eval loop changes since I last worked on it, though. (I think that conflict may also be keeping the bots from running, even after I updated the issue title to use the modern convention)

[rhettinger]

Is there a chance this will get resolved for 3.8? This is an important guarantee.

[gpshead]

While there is technically still time, it'd take people spending dedicated time on it before the 3.8 release. I won't be.

Important? Sure. But not urgent; this bug is not new. We've lived with it for many releases already. I believe it has been around since 2.5 when with statements were introduced.

[markshannon]

I expect to fix this is as part of the general improvements I am making to the bytecode, interpreter and compiler. So it should be fixed for 3.9 but not for 3.8

[ncoghlan]

It's also not unique to with statements - it applies to all finally clauses. The longstanding workaround when deterministic cleanup is absolutely critical has been to run the "real" application in a subthread, and devote the main thread to gracefully terminating the subthread when requested.

When cleanup is critical, but doing it in a deterministic order is less so, __del__ methods are often used to fill the gap (although they too can be interrupted by a subsequent Ctrl-C).

I also realized that allowing infinite loops in cleanup code to ignore Ctrl-C may actually be a tolerable outcome: in the worst case, users can still escalate to Ctrl-Break/kill -9/Force stop/etc and pull the entire OS process out from under the interpreter. It's not good, but may be worth it in order to better handle users pressing Ctrl-C multiple times.

[gpshead]

As a note on the general pattern, a user at work diagnosed a ^C problem in their code when running on 2.7 to be due to Queue.get's

acquire()
try:
  ...
finally:
  release()

Pattern, with the KeyboardInterrupt triggering after acquire() but before the try is entered. so release() is never called.

A try finally pattern that probably alleviates this by entering the try block first might look like:

try: acquire() ... finally: try: release() except ThreadError: pass # interrupted before acquire() succeeded.

It'd be a shame if any with statements lock acquisition context managers need to be turned into that, but I _believe_ it'd be a viable workaround for the time being if this race is found to be biting anyone.

[ncoghlan]

Belatedly clearing the issue assignment here - while I do still sometimes ponder this problem, I haven't been actively working on it since the 2017 core sprint where Greg & I made our last serious attempt at trying to improve the situation.

Mark's PR at https://github.com/python/cpython/pull/18334 looks very promising to me, though - my main request was just to bring over the tests I wrote at the 2017 core dev sprints, and confirm that the revised eval breaker logic solves the issue.

[287fad31-6852-4cd7-9684-fe7b51e6f7cb]

Does https://github.com/python/cpython/pull/1799 solve this issue for synchronous with?

    with closing(this), closing(that):

If it does, can we backport this fix to python 3.6?

3.6 is used as system python for RHEL/Centos 8, will be used for at least 5 years or so.

[markshannon]

The bytecode instruction set has changed a lot since 3.6, so I think a backport would be impractical.

3.6 is in security fix only mode, so you'd need to take this up with Red Hat.