curio.timeout_at scope - Githubissues

brennane commented 7 years ago

Howdy,

This is a question about the interplay between curio.timeout_at and (for example) curio.gather. The default mechanism is likely to give someone a confusing ride.

If you run the first example tm0.py the third "curio.gather" task swallows the timeout that should be used to control things. See the reference gist below and its function timeout_gather.

Linuss-MacBook-Pro:app breevans$ ../run tm0.py
task kid took 0.10558457000297494
task kid took 0.6065383800014388
task kid took 1.1037434339814354
task kid took 1.6023559189925436
task kid took 2.103851986990776
task kid got some error You found me! 5
task kid took 2.6063826289901044
task kid took 3.1054359920090064
task kid got some error You found me! 7
task kid took 3.6047377610229887
task kid took 4.103714413999114
task kid took 4.603543856996112
done:
  0. 0
  1. 1
  2. ('traps', '_join_task', '105'): TaskTimeout, '220025.839626368', [task|gather|178] [task|join|84] [traps|_join_task|105]
  3. 3
  4. 4
  5. ('task', 'join', '86'): TaskError, 'Task crash', [task|gather|178] [task|join|86] |CAUSE| ('tm0', 'kid', '57'): ValueError, 'You found me! 5', [kernel|_run_coro|972] [tm0|F|17] [tm0|kid|57]
  6. 6
  7. ('task', 'join', '86'): TaskError, 'Task crash', [task|gather|178] [task|join|86] |CAUSE| ('tm0', 'kid', '57'): ValueError, 'You found me! 7', [kernel|_run_coro|972] [tm0|F|17] [tm0|kid|57]
  8. 8
  9. 9

In the second example tm1.py the exception is broadcast to the inner tasks. It seems this would be the intended behavior.

Linuss-MacBook-Pro:app breevans$ ../run tm1.py
task kid took 0.1012277519912459
task kid took 0.6016836360213347
bombs away!
task _bombadier took 1.0041643189906608
task kid was bombed to smithereens
task kid took 1.0046917950094212
task kid was bombed to smithereens
task kid took 1.0048421169922221
task kid was bombed to smithereens
task kid took 1.0049122350174002
task kid was bombed to smithereens
task kid took 1.0049572810239624
task kid was bombed to smithereens
task kid took 1.005063254997367
task kid was bombed to smithereens
task kid took 1.0051939010154456
task kid was bombed to smithereens
task kid took 1.0053089649882168
task kid was bombed to smithereens
task kid took 1.0053616020013578
bomb went off first
RESULT mission :: [0, 1, TaskError('Task crash',), TaskError('Task crash',), TaskError('Task crash',), TaskError('Task crash',), TaskError('Task crash',), TaskError('Task crash',), TaskError('Task crash',), TaskError('Task crash',)]
done:
  0. 0
  1. 1
  2. ('task', 'join', '86'): TaskError, 'Task crash', [task|gather|178] [task|join|86] |CAUSE| ('traps', '_sleep', '81'): TimeBomb, 'TimeBomb', [kernel|_run_coro|972] [tm1|F|24] [tm1|kid|83] [task|sleep|138] [traps|_sleep|81]
  3. ('task', 'join', '86'): TaskError, 'Task crash', [task|gather|178] [task|join|86] |CAUSE| ('traps', '_sleep', '81'): TimeBomb, 'TimeBomb', [kernel|_run_coro|972] [tm1|F|24] [tm1|kid|83] [task|sleep|138] [traps|_sleep|81]
  4. ('task', 'join', '86'): TaskError, 'Task crash', [task|gather|178] [task|join|86] |CAUSE| ('traps', '_sleep', '81'): TimeBomb, 'TimeBomb', [kernel|_run_coro|972] [tm1|F|24] [tm1|kid|83] [task|sleep|138] [traps|_sleep|81]
  5. ('task', 'join', '86'): TaskError, 'Task crash', [task|gather|178] [task|join|86] |CAUSE| ('traps', '_sleep', '81'): TimeBomb, 'TimeBomb', [kernel|_run_coro|972] [tm1|F|24] [tm1|kid|83] [task|sleep|138] [traps|_sleep|81]
  6. ('task', 'join', '86'): TaskError, 'Task crash', [task|gather|178] [task|join|86] |CAUSE| ('traps', '_sleep', '81'): TimeBomb, 'TimeBomb', [kernel|_run_coro|972] [tm1|F|24] [tm1|kid|83] [task|sleep|138] [traps|_sleep|81]
  7. ('task', 'join', '86'): TaskError, 'Task crash', [task|gather|178] [task|join|86] |CAUSE| ('traps', '_sleep', '81'): TimeBomb, 'TimeBomb', [kernel|_run_coro|972] [tm1|F|24] [tm1|kid|83] [task|sleep|138] [traps|_sleep|81]
  8. ('task', 'join', '86'): TaskError, 'Task crash', [task|gather|178] [task|join|86] |CAUSE| ('traps', '_sleep', '81'): TimeBomb, 'TimeBomb', [kernel|_run_coro|972] [tm1|F|24] [tm1|kid|83] [task|sleep|138] [traps|_sleep|81]
  9. ('task', 'join', '86'): TaskError, 'Task crash', [task|gather|178] [task|join|86] |CAUSE| ('traps', '_sleep', '81'): TimeBomb, 'TimeBomb', [kernel|_run_coro|972] [tm1|F|24] [tm1|kid|83] [task|sleep|138] [traps|_sleep|81]

Here's my ugly gist code.

tm0.py gist showing unexpected behavior tm1.py gist showing expected behavior tb.py traceback printer

dabeaz commented 7 years ago

Intriguing. Looking into it.

dabeaz commented 7 years ago

I've pushed a change for this. gather() will now cancel all tasks if it gets any kind of cancellation. The resulting cancellation exception has the results of all tasks attached to it. For example::

try:
      results = await gather(tasks)
except CancelledError as e:
      resullts = e.results       # Partially completed results (before cancellation)

Open to suggestions on further improvement

dabeaz / curio

curio.timeout_at scope #186