Supervisor API

[goodboy]

I want to start design discussion on how to approach erlang-like supervisors including restart strategies and how this will interplay with a service discovery system and possibly real-time code replacement.

Some questions/comments I have:

• what's the (semantic) difference between a nursery and a supervisor?

  • a nursery is more primitive and is what a supervisor should be built upon (eg. python-trio/trio#569)?
  • should a supervisor be specific to managing (possibly distributed) processes in which case does having a nursery mean anything other then a special type of supervisor?
  • there's been some interesting talk of renaming nurseries and how asyncio might be approaching nurseries/supervisors

• an explicit distinction should be made between a MainProcess supervisor and the arbiter actor

  • the Arbiter is per-host and is part of service discovery between hosts

• how do we best implement a distributed supervisor (one that spawns actors over multiple hosts)?

  • does it simply send the spawn request to the appropriate arbiter who will then create or use a host-local nursery?
  • what happens when all actors on the remote host go down but a remote actor is still using a remotely spawned actor in that process cluster?

• what does a distributed process supervisor API look like?

  • how does this interact with the arbiter and service discovery system?

• how does an orchestration layer build on all this?

• here's erlang's supervisor behaviors

Much more to come...

[goodboy]

Worth a read is elixir's page on supervisors and dynamic supervisors.

[goodboy]

We had a brief discussion a while back slightly in regard to #190 (this comment) related apis.

For supervisors I actually think we can make quite a nice api by just offering composed async context managers:

async with tractor.open_nursery() as n:

    async with tractor.supervise.one_for_one(n) as one_for_one_nursery:

        await one_for_one_nursery.run_in_actor(blah)

In 3.10 parentheses syntax this becomes:

async with (
    tractor.open_nursery() as n,
    tractor.supervise.one_for_one(n) as one_for_one_nursery,
):        
        await one_for_one_nursery.run_in_actor(blah)

This has the benefit of being simple to implement:

• error handling logic is completely self contained in the specialized - lower super
• restart behavior can be factored out and called from any variety of specialized super strategies
• can an exit stack be used to compose multiple strats?

Further questions:

• how will the API differ for strats pertaining to process restarts vs. remote task restarts
  • is there even a difference from a user's perspective presuming the conditional of: a secondary actor exists -> a remote user defined task is running in that actor.
• how can this interplay with resiliency designs and decentralized fail-over patterns
• does dynamic supervision really mean anything different for us?
  • If we encap logic in appropriately written cntx mngs there's no difference whether 1 vs. N actors / tasks are spawned right?

[goodboy]

tenacity is an interesting project that seems to be defacto for python restarts.

I think the most compatible (though not fully convinced yet) api that we might want to adopt is the code block retrying.

njs just proposed better native trio support as well.

[goodboy]

A nice short post from @catern affirming the supervisor at every layer design is the future of process management.