Open 1st1 opened 4 years ago
Whiteboard image from when Yury and I talked back in September; sketch of PEP outline on the right:
I did an substantial editing pass over Yury's first comment, check it out
Looking at my notes from September, I made a list of some obscure corners that we'll need to talk about sometime (not necessarily first draft... maybe first draft should at least have a list like this though, to encourage folks to point out any other obscure corners we need to think about?):
asyncio.Future
to support representing "cancelled and also has an exception" – I think it would work to represent cancelled as "has an exception and that exception contains at least one CancelledError", except that we need to tweak the .exception() method in a backwards-incompatible way, because it has totally disjoint behavior for regular exceptions versus CancelledError
@ambv NB this thread is where a lot of the design issues got hashed out originally, so probably worth reading, and will also probably want to cite in a references section: https://github.com/python-trio/trio/issues/611
Someone just asked about this in the Trio chat, so I guess I'll check if there are any updates?
I was out on vacation this week but I'll have the first draft up next.
-- Best regards, Łukasz Langa
On 1 Feb 2020, at 02:48, Nathaniel J. Smith notifications@github.com wrote:
Someone just asked about this in the Trio chat, so I guess I'll check if there are any updates?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.
@njsmith @ambv feel free to edit this message.
Introduction
This should be a relatively high-level and fun to read section.
[ ] Talk about exceptions in an abstract way. I.e. why it is important to have them in Python, why one of the key Python design principle is that errors should never happen silently.
[ ] Why isn't one exception enough? Sometimes you have multiple errors that are all equally important:
Main reason: Concurrency (async/await, threads, multiprocessing, ...): If you're doing multiple things at once, then multiple things can fail at once. Only options are to catch them, throw some away, or have some way to propagate multiple exceptions at once. We can't force users to catch them, so right now we have to throw them away, which breaks "Errors should never pass silently".
Hypothesis runs a single test lots of times with random input. Sometimes it detects multiple distinct failures, on different inputs. Currently it has no good way to report that.
A unit test: the test failed and also failed its tear down code.
[ ] Why aren't
__context__
and__cause__
good enough? (They give more details about a single exception, but in these cases there are multiple independent errors, that need to be handled or propagated independently. Maybe a concrete example, e.g., you have both a network error + an assertion error, some routine retry code handles the network error, that shouldn't cause the assertion error to be lost?)Motivation
Here we should talk about exception groups in more detail explaining "why" and "why now".
[ ] Why we are looking at this problem now? The answer is async/await. Here we need to talk about
asyncio.gather()
and Trio/nurseries.asyncio.gather(t1, t2, t3)
is the primitive one uses in asyncio to wait until multiple concurrent tasks are finished. It is flawed. Ift1
is failed then we'd propagate its error and try to shutdown botht2
andt3
, but shutting them down can also result in errors. Those errors are currently lost.There's a
return_exceptions=True
flag forasyncio.gather
that changes its behavior entirely. Exceptions are returned along with the results. This is a slightly better approach, but: it's an opt-in; handling exceptions this way is cumbersome -- you suddenly can't usetry..except
block.Controlling how things run concurrently is the key thing in asyncio. Current APIs are bad and we need a replacement ASAP.
Trio has a concept called nurseries and it's great. This is what users want in asyncio—literally the most requested feature. Nurseries are context managers that let you structure your async code in a very visual and obvious way.
The problem is that a nursery
with
block needs to propagate more than one exception. Boom, we need exception groups (EG) to implement them for asyncio.[ ] To sum up: there were always use cases that required a concept like EGs. It's just that they were not as pronounced as they are now with async/await.
[ ] Why does it need to be in the language/stdlib? Can it be a third party library?
What does an EG look like?
ExceptionGroup
that inherits fromBaseException
, and holds a list of exceptions + for each one an string tag giving some human-readable info about where this exception came from to enter this group. Show examples. Show nested examples.Then walk through the rationale for these choices:
[ ] The EG type must be an exception itself. It cannot be a tuple or list. Why: we want
try..finally
to work correctly when an exception group is propagated. Also, makingsys.exc_info()
/PyThreadState->exc_info
start holding non-exception objects would be super invasive and probably break lots of things.[ ] The EG type must be a
BaseException
as it can potentially contain multipleBaseExceptions
.[ ] To give useful tracebacks, we want to preserve the path the exception took, so these need to be nested (show an example of a Trio nested traceback to illustrate)
[ ] We want to attach tags / titles to exceptions within EGs. Tasks in asyncio/trio and threads in Python have names -- we want to attach a bit of information to every exception within a EG saying what's its origin.
[ ] But semantically, they represent an unstructured set of exceptions; we just use the tree structure to hold traceback info and to get a single object representing the set
[ ] Discuss why we allow single-element
ExceptionGroup
s[ ] Gives opportunity to attach tags to show which tasks an exception passed through as it propagated
[ ] In current prototypes, catching exceptions inside an
ExceptionGroup
requires special ceremony. If this ceremony is needed sometimes, then we want to make it needed always, so that users don't accidentally use regulartry
/except
and have it seem to work until they get unlucky and multiple exceptions happen at the same time. Therefore, exceptions that pass through a Trio nursery/asyncioTaskGroup
should be unconditionally wrapped in anExceptionGroup
. But, this rationale may or may not apply to a "native" version ofExceptionGroup
s, depending on what design we end up with for catching them.Working with exception groups
[ ] Core primitives:
split
andleaves
[ ] Semantics
[ ] Rationale
Catching exceptions in
ExceptionGroups
[ ] Explain basic desired semantics: can have multiple handler "arms", multiple arms can match the same group, they're executed in sequence, then any unhandled exceptions + new exceptions are bundled up into a new
ExceptionGroup
[ ] How should this be spelled? We're not sure. Trade-offs are extremely messy; we're not even going to try doing a full discussion in this first draft. Some options we see:
Modifying the behavior of
try..except
to have these semantics. (Downside: major change to the language!)Leave
try
/except
alone, add new syntax (grouptry
/groupexcept
or whatever). (Downside: you always wantgrouptry
/groupexcept
, nevertry
/except
!)No new syntax, use awkward circumlocutions instead of
try
/except
. (Downside: they're extremely awkward!)Since the design space and trade-offs are super complex, we're leaving a full discussion for a later draft / follow-up PEP.