Draft 0.1

[Gozala]

📜 Preview 🕹️ Interactive View

This pull request synthesizes w3 invocation spec with #1 in an attempt to simplify it such that it would be a good fit for both web3.storage and IPVM use cases. Most notable it:

• It addresses https://github.com/ucan-wg/invocation/issues/6 by removing batching semantics
• It introduces an Authorization primitive for authorizing group of tasks with a single signature
• Removes named addressing in favor of IPLD link based addressing
• Introduces effects in receipts

[Gozala]

@expede I have been thinking that right now we have Await that can describe dependencies than need to be awaited upon, but we lack primitive for describing dependencies that do not need to be awaited upon. There are some cases where you may want to start some tasks and thread through reference(s) into tasks started later in the workflow.

This also intentionally or unintentionally makes it so that Task input can only Join and Task output (meaning effect) can only fork. I think this is the limitation I was struggling with which makes me wonder if we could address this limitation and consequently improve effects. Here is what I have in mind:

We could replace Await in this spec with more genera Effect below

type Effect<In> struct {
  # Join instructs scheduler to execute a task and substitute placeholder with
  # a result
  &Promise<In>      "await/*"       # Resolves to the task result 
  &Promise<In>      "await/ok"      # Resolves to unwrapped ok branch
  &Promise<In>      "await/error"   # Resolves to unwrapped error branch

  # Fork instructs scheduler to execute a task and substitute a placeholder with
  # an invocation link
  &Task<In>         "task/spawn"    # Instructs to invoke with current auth
  &Invocation<In>   "task/fork"     # Instructs to invoke
}

Idea is the same as with Await it's a placeholder that gets substituted by the executor. New task/spawn and task/fork variants would simple not await on the completion instead they'd just be substituted with &Invocation.

With this such change we'll get ability to fork in the input which means you could have task take / start invocation and pass it to subtasks which can choose to than await on the result.

If we had forking here I think it would also make single fx in the receipt less annoying as well, we'd still need some kind of "identity" Task to piece all the pieces together but other than that everything would fit really well:

{
   input: {
       sendMessage: {
            "task/spawn":  { "/": "bafy...send" }
       },
       writeFile: {
            "await/*": { "/": "bafy...read" }
       }
   }
   // ...
}

We could also make identity somewhat special by trimming obsolete fields from the Task so it no longer requires wrapping overhead, but just like other Tasks could have receipt

[expede]

@Gozala

(I wish there was a way to reply to a comment directly in GitHub)

{
  input: {
      sendMessage: {
           "task/spawn":  { "/": "bafy...send" }
      },

I'm confused: why would I have this in the inputs to a Task? Unless I'm missing something, I would model this with a separate Task since I don't need its value in this one. Would this only be in the Receipt?

I would also really like to keep effects out of the inner description of a Task, just to avoid mixing levels as much has possible. I get that it's impossible to prevent someone from doing this, but also I think there's lot of design space that doesn't depend on putting this on the happy path.

we'd still need some kind of "identity" Task

An Identity Task is pretty much what I'm trying to avoid as "pure overhead", and I think is a only a consequence of forcing a single effect in the Receipt. This part of the design space requires a lot of special Tasks and moving pieces when I'm pretty sure that we can model this stuff out of existing components (which I realize is pretty much what you were saying to me last week about abandoning first-class transactions).

I keep coming back to the fact that it feels like we're exchanging one set of complexities for another between our two versions, each more suited to the author's use case.

[expede]

@Gozala I wonder if it could be helpful for me to get an example of a typical session that you expect DAG House to run in as high resolution as possible. I think that we may be talking past each other, and I'd like to show you how I'd model your use case but don't want to misrepresent it

[Gozala]

I'm confused: why would I have this in the inputs to a Task? Unless I'm missing something, I would model this with a separate Task since I don't need its value in this one. Would this only be in the Receipt?

I was proposing extending Await with two additional variants:

1. task/spawn which instructs the runtime to invoke a task with the current authorization and substitute placeholder with a handle (&Invocation).

   It is roughly equivalent of creating a promise in JS without awaiting on it. You still can pass it around and combine with other things.

   Passing just &Promise to a task isn’t same as it’s not instructing invoking it. await/* is also not the same because task won’t start until awaited task is done.

   So if I pass this into a task I create a dependency, but not the one that awaits completion but one that awaits start of execution. E.g if runtime fails or chooses not to start task/spawn dependent will not either.

2. task/fork which is essentially same as task/spawn except it wraps invocation so has different auth context.

Now whether calling extended Await an Effect makes sense or not is a different matter. Perhaps not, maybe something like Instruction or Control would be less confusing

[expede]

Sorry for all of the small comments here, but I want to capture some of this before I head off for the night:

In the above discussion, we have both async/await and fork/join. I wonder if it would be helpful to frame my suggestion as something like "would it be possible to keep things to async/await?" In this version, Tasks are implicitly async (because they're non-blocking unless you have an await). In this view, fork is just invocation, and join is await.

[expede]

It is roughly equivalent of creating a promise in JS without awaiting on it.

Right, so why not discover this from the call graph (with nothing more than await)? I'm unclear on the benefit of this additional tag.

So if I pass this into a task I create a dependency, but not the one that awaits completion but one that awaits start of execution.

Do you need that for your semantics at DAG House? I don't think that we need this for IPVM (we only need data dependencies for ordering).

[Gozala]

I would model this with a separate Task since I don't need its value in this one.

You ate right you could model this with some other task like fork which takes promise and puts it in the effect returning handle in result. However that would not guarantee that task it’s passed into won’t start even if effect will not, which is primary difference

Would this only be in the Receipt?

No idea is to make it general control structure like await, but I do expect to see it primarily in effects

[expede]

However that would not guarantee that task it’s passed into won’t start even if effect will not, which is primary difference

I'm not sure that I follow. Could you expand on this?

[expede]

No idea is to make it general control structure like await, but I do expect to see it primarily in effects

Unless there's a really critical reason, I'd ideally like to keep additional control structures out of the spec. In the same way that "lambda is the ultimate", promise pipelining is sufficient to describe any graph, and thus can describe any control flow.

That's not to say that the array in a receipt is the one true way, but the direction of introducing more control structures feels like a step backwards in added complexity.

[Gozala]

Right, so why not discover this from the call graph (with nothing more than await)? I'm unclear on the benefit of this additional tag.

Maybe we're making different assumptions an what promise as input tells the runtime. More specifically what do you expect runtime to do in the phase of such invocation:

{
  do: "some/task"
  input: {
     a: { "ucan/task": { "/": "bafy...a" },
     b: { "ucan/task": { "/": "bafy...b" },
  }
}

I was expecting that task will be just passed a and b without any substitution and than neither bafy...a or bafy...b would be invoked by runtime.

On the other hand with a following invocation:

{
  do: "some/task"
  input: {
     a: { "task/spawn": { "/": "bafy...a" },
     b: { "task/spawn": { "/": "bafy...b" },
  }
}

I would expect some/task to be only run after a and b have been spawn (concurrently) and than a and b would be substituted with invocations like [{ "/": bafy...a" }, { "/": bafy...auth }] ...

If you think that first example should behave like my second one, than I a agree task/spawn and task/fork are obsolete. However if our expectations on first one are the same we have no way of doing what I've described in second.

[expede]

If you think that first example should behave like my second one, than I a agree task/spawn and task/fork are obsolete.

Just to make sure that we're on the same page: these are not awaits, but rather some special Task type that runs multiple invocations? I would leave those semantics up to the task type. Being input, I wouldn't expect the runtime to know that there's anything special happening here unless it was some Task type defined in the spec.

[Gozala]

Just to make sure that we're on the same page: these are not awaits, but rather some special Task type that runs multiple invocations?

I'm not sure which part you're referring to by "these". Examples are regular Tasks (as per this draft) I've just omitted the with and other fields that were irrelevant here.

If you are referring to { "ucan/task": { "/": "bafy...a" } } that is how this draft represents a promise. Awaiting for the result would have looked like { 'await/*': { "ucan/task": { "/": "bafy...a" } }

[Gozala]

I had off channel conversation with @expede where she identified following problem with the task/spawn / task/fork design

• Introduces more privitives for things that we can already do (enqueuing more tasks)
• Introduces nesting of invocations (more cycles and network to discover the entire workflow)
• Need to add a version that says "but don't run it yourself" when that's not a problem that we have today
• The label "spawn" reads poorly if it's going to dedup (looks like an imperative command but isn't)

One that especially resonated that is that they imply that you always ask executor to run the task as opposed to find a prior task execution.

We also have agreed to simplify things by just making fx a Promise instead. That way design forces runtime to introduce explicit containers for forking & joining execution. I like this compromise because it forces runtime to either join or fork explicitly which would works better for us than implicit joins (that describe how multiple results are combined) in the face of multiple invocations.