Closed Gozala closed 1 year ago
@expede I have been thinking that right now we have Await
that can describe dependencies than need to be awaited upon, but we lack primitive for describing dependencies that do not need to be awaited upon. There are some cases where you may want to start some tasks and thread through reference(s) into tasks started later in the workflow.
This also intentionally or unintentionally makes it so that Task input can only Join and Task output (meaning effect) can only fork. I think this is the limitation I was struggling with which makes me wonder if we could address this limitation and consequently improve effects. Here is what I have in mind:
We could replace Await
in this spec with more genera Effect
below
type Effect<In> struct {
# Join instructs scheduler to execute a task and substitute placeholder with
# a result
&Promise<In> "await/*" # Resolves to the task result
&Promise<In> "await/ok" # Resolves to unwrapped ok branch
&Promise<In> "await/error" # Resolves to unwrapped error branch
# Fork instructs scheduler to execute a task and substitute a placeholder with
# an invocation link
&Task<In> "task/spawn" # Instructs to invoke with current auth
&Invocation<In> "task/fork" # Instructs to invoke
}
Idea is the same as with Await
it's a placeholder that gets substituted by the executor. New task/spawn
and task/fork
variants would simple not await on the completion instead they'd just be substituted with &Invocation
.
With this such change we'll get ability to fork in the input
which means you could have task take / start invocation and pass it to subtasks which can choose to than await on the result.
If we had forking here I think it would also make single fx
in the receipt less annoying as well, we'd still need some kind of "identity" Task to piece all the pieces together but other than that everything would fit really well:
{
input: {
sendMessage: {
"task/spawn": { "/": "bafy...send" }
},
writeFile: {
"await/*": { "/": "bafy...read" }
}
}
// ...
}
We could also make identity
somewhat special by trimming obsolete fields from the Task
so it no longer requires wrapping overhead, but just like other Tasks
could have receipt
@Gozala
(I wish there was a way to reply to a comment directly in GitHub)
{ input: { sendMessage: { "task/spawn": { "/": "bafy...send" } },
I'm confused: why would I have this in the inputs to a Task? Unless I'm missing something, I would model this with a separate Task since I don't need its value in this one. Would this only be in the Receipt?
I would also really like to keep effects out of the inner description of a Task, just to avoid mixing levels as much has possible. I get that it's impossible to prevent someone from doing this, but also I think there's lot of design space that doesn't depend on putting this on the happy path.
we'd still need some kind of "identity" Task
An Identity Task is pretty much what I'm trying to avoid as "pure overhead", and I think is a only a consequence of forcing a single effect in the Receipt. This part of the design space requires a lot of special Tasks and moving pieces when I'm pretty sure that we can model this stuff out of existing components (which I realize is pretty much what you were saying to me last week about abandoning first-class transactions).
I keep coming back to the fact that it feels like we're exchanging one set of complexities for another between our two versions, each more suited to the author's use case.
@Gozala I wonder if it could be helpful for me to get an example of a typical session that you expect DAG House to run in as high resolution as possible. I think that we may be talking past each other, and I'd like to show you how I'd model your use case but don't want to misrepresent it
I'm confused: why would I have this in the inputs to a Task? Unless I'm missing something, I would model this with a separate Task since I don't need its value in this one. Would this only be in the Receipt?
I was proposing extending Await
with two additional variants:
task/spawn
which instructs the runtime to invoke a task with the current authorization and substitute placeholder with a handle (&Invocation).
It is roughly equivalent of creating a promise in JS without awaiting on it. You still can pass it around and combine with other things.
Passing just &Promise to a task isn’t same as it’s not instructing invoking it. await/*
is also not the same because task won’t start until awaited task is done.
So if I pass this into a task I create a dependency, but not the one that awaits completion but one that awaits start of execution. E.g if runtime fails or chooses not to start task/spawn
dependent will not either.
task/fork
which is essentially same as task/spawn
except it wraps invocation so has different auth context.
Now whether calling extended Await
an Effect makes sense or not is a different matter. Perhaps not, maybe something like Instruction
or Control
would be less confusing
Sorry for all of the small comments here, but I want to capture some of this before I head off for the night:
In the above discussion, we have both async/await and fork/join. I wonder if it would be helpful to frame my suggestion as something like "would it be possible to keep things to async/await?" In this version, Tasks are implicitly async
(because they're non-blocking unless you have an await
). In this view, fork
is just invocation, and join
is await
.
It is roughly equivalent of creating a promise in JS without awaiting on it.
Right, so why not discover this from the call graph (with nothing more than await
)? I'm unclear on the benefit of this additional tag.
So if I pass this into a task I create a dependency, but not the one that awaits completion but one that awaits start of execution.
Do you need that for your semantics at DAG House? I don't think that we need this for IPVM (we only need data dependencies for ordering).
I would model this with a separate Task since I don't need its value in this one.
You ate right you could model this with some other task like fork
which takes promise and puts it in the effect returning handle in result. However that would not guarantee that task it’s passed into won’t start even if effect will not, which is primary difference
Would this only be in the Receipt?
No idea is to make it general control structure like await, but I do expect to see it primarily in effects
However that would not guarantee that task it’s passed into won’t start even if effect will not, which is primary difference
I'm not sure that I follow. Could you expand on this?
No idea is to make it general control structure like await, but I do expect to see it primarily in effects
Unless there's a really critical reason, I'd ideally like to keep additional control structures out of the spec. In the same way that "lambda is the ultimate", promise pipelining is sufficient to describe any graph, and thus can describe any control flow.
That's not to say that the array in a receipt is the one true way, but the direction of introducing more control structures feels like a step backwards in added complexity.
Right, so why not discover this from the call graph (with nothing more than
await
)? I'm unclear on the benefit of this additional tag.
Maybe we're making different assumptions an what promise as input tells the runtime. More specifically what do you expect runtime to do in the phase of such invocation:
{
do: "some/task"
input: {
a: { "ucan/task": { "/": "bafy...a" },
b: { "ucan/task": { "/": "bafy...b" },
}
}
I was expecting that task will be just passed a
and b
without any substitution and than neither bafy...a
or bafy...b
would be invoked by runtime.
On the other hand with a following invocation:
{
do: "some/task"
input: {
a: { "task/spawn": { "/": "bafy...a" },
b: { "task/spawn": { "/": "bafy...b" },
}
}
I would expect some/task
to be only run after a
and b
have been spawn (concurrently) and than a
and b
would be substituted with invocations like [{ "/": bafy...a" }, { "/": bafy...auth }]
...
If you think that first example should behave like my second one, than I a agree task/spawn
and task/fork
are obsolete. However if our expectations on first one are the same we have no way of doing what I've described in second.
If you think that first example should behave like my second one, than I a agree task/spawn and task/fork are obsolete.
Just to make sure that we're on the same page: these are not await
s, but rather some special Task type that runs multiple invocations? I would leave those semantics up to the task type. Being input
, I wouldn't expect the runtime to know that there's anything special happening here unless it was some Task type defined in the spec.
Just to make sure that we're on the same page: these are not
await
s, but rather some special Task type that runs multiple invocations?
I'm not sure which part you're referring to by "these". Examples are regular Task
s (as per this draft) I've just omitted the with
and other fields that were irrelevant here.
If you are referring to { "ucan/task": { "/": "bafy...a" } }
that is how this draft represents a promise. Awaiting for the result would have looked like { 'await/*': { "ucan/task": { "/": "bafy...a" } }
I had off channel conversation with @expede where she identified following problem with the task/spawn
/ task/fork
design
One that especially resonated that is that they imply that you always ask executor to run the task as opposed to find a prior task execution.
We also have agreed to simplify things by just making fx
a Promise
instead. That way design forces runtime to introduce explicit containers for forking & joining execution. I like this compromise because it forces runtime to either join or fork explicitly which would works better for us than implicit joins (that describe how multiple results are combined) in the face of multiple invocations.
📜 Preview 🕹️ Interactive View
This pull request synthesizes w3 invocation spec with #1 in an attempt to simplify it such that it would be a good fit for both web3.storage and IPVM use cases. Most notable it:
Authorization
primitive for authorizing group of tasks with a single signature