Open ch1bo opened 2 years ago
In a discussion I had with @mchakravarty we touched on the fact that it would also be possible to directly open
a head in a single transaction and do all commits incrementally after. That might lead to congestion on the state output (it's like sequential commits), but simplifies some other use cases?
Incremental decommits may be very interesting to oracle use cases, where the resulting data is getting decomitted upon consumption.
The use case of Incremental De/Commit really adds so much added value to many of the Use Cases that Hydra seeks solve for end-users.
From an end-users perspective, with all of the potential Heads that will be running, they will be locking funds in various places. A stake pool is well understood way of locking in funds, but with Heads, and the varied use cases, they may be more hesitant or unable (because they have commit to previous heads), to participate.
By having Incremental De/Commit users are free to allocate their funds as necessary with their goals, and feel more confident with the developers integrating features utilizing Hydra Heads.
So I see this as a very important feature for optics to the greater cardano community, and Hydra Pay would definitely be able to make good use of it.
@Sbcdn mentioned they would love to have this feature as well for their use case so +1
We discussed incremental commits/decommits (aka increments/decrements) today with the original authors of the Hydra Head paper:
Basically this is about adding two more transactions increment and decrement to the L1 protocol
Decrement requires certificate and all participants need to agree on taking something out of the head state
Sandro: This is a rough sketch/doesn't include anything related to L2 code
We can start from the paper and build intuition further using knowledge from research team (pick their brain not do any additional work)
Multiple ideas to go about the increment step (original paper was not clear how $\eta$ turns into $\eta'$)
Discussion pros and cons of these two approaches
What does this mean - request saying some participant is adding utxos?
Matthias: We would extend snapshots to include new information.
Currently, utxos are combined and hashed together for the on-chain code
Why MPT's? We need it in order for validator to be able to check that new state includes new utxos. We don't want to store utxo's on-chain. Could utilize inclusion proofs for the on-chain code.
Why would we want to use the L2 for incrementing the UTxO as well? Wouldn't everyone always agree?
It is likely cheaper / less work on L1
Requires more off-chain coordination (usual trade-off)
For L2 signing: we would need to sign the $\eta$, the $\eta'$ (of the latest snapshot which we expand) as well as the UTxO added/removed
to avoid commits you want to include may not exist anymore
to ensure sequencing of multiple increments
Isn't this interleaving state updates on the L2 with adding/removing funds from the L1?
Can't we do better than this combine everything approach?
After all, UTxO guarantee that only present outputs can be spent
Conclusion:
Some use cases that we believe could benefit from this:
Notes from @mchakravarty on incremental commits and more general configuration changes (from back in the days):
Hydra Head with Incremental Commits
Requirements Head state snapshot
Configuration change transactions
Adding and removing participants
Committing and decommitting UTxOs
Synchronisation
Multisigs
Besides the basic mermaid diagram above, here is another drawing of the potential life-cycle with incremental commits/decommits from our Miro board:
Grooming discussion:
increment
part and realized we need a two-step process to avoid the problem of a commit not "being picked up". This follows the design of @mchakravarty (above) and @georgeFlerovsky (Hydrozoa).Next steps:
We continued work on this after also starting #1057. We had implemented the off-chain workflow to the point of this diagram shows:
sequenceDiagram
Alice->>+API: POST /commit (UTxO)
API->>HeadLogic: Commit UTxO
par broadcast
HeadLogic->>HeadLogic: ReqInc incUTxO
and
HeadLogic->>Node B: ReqInc incUTxO
end
HeadLogic -->> Alice: WS CommitRequested
par Alice isLeader
HeadLogic->>HeadLogic: ReqSn incUTxO
and
HeadLogic->>Node B: ReqSn incUTxO
end
Note over HeadLogic,Chain: PROBLEM: Need to verify incUTxO on L1 as we authorize the TxIns to use (because of on-chain scripts).
HeadLogic->>HeadLogic: sig = sign snapshot incl. inputs(incUTxO)
par broadcast
HeadLogic->>HeadLogic: AckSn sig
and
HeadLogic->>Node B: AckSn sig
end
Node B->>HeadLogic: AckSn sig
HeadLogic -->> Alice: WS SnapshotConfirmed
HeadLogic -->> Alice: WS CommitApproved
HeadLogic -->> API: SnapshotConfirmed
API->>API: draftIncrementTx vk snapshot sig >>= finalizeTx >>= signTx sk
API-->>-Alice: IncrementTx
Alice->>Alice: sign IncrementTx
Alice->>Chain: submit IncrementTx
Chain->>HeadLogic: OnIncrementTx
HeadLogic-->>Alice: CommitFinalized
However, when working on the specification and trying to realize the recommendation of researchers, we hit the problem as indicated in the picture. Namely, that in one of the designs the assumption was made that Hydra participants would sign off on the transaction output references (TxOutRef
) of the to-be-committed UTxO
. While this would make the on-chain part fairly simply - we only need to show the out refs in the redeemer and on-chain reproduce the signed data by serializing a list of [TxOutRef]
, it makes the overall protocol very interactive and would require a new interaction between the protocol logic in the hydra-node
and the L1 chain (query the output references before signing).
We need to discuss this and the alternative of using a Merkle-Tree based $\eta$-construction with researchers. We now switch focus on the off-chain part on this item (as to the user, any variant would be identical) and the on-chain part of #1057 (which works on the "naiive"-$\eta$ the same as for the MT-based one) for the short term.
We should also revise the API for requesting a commit. Having a mere UTxOWithWitnesses
is not enough as indicated by this feature request: https://github.com/input-output-hk/hydra/discussions/1337
Notes from a discussion yesterday:
Do we need to keep ηα separate from η in the new open state?
No, as long as we ensure that Uα is differentiable while snapshots evolve off-chain
Is this a property?
For some digest function and combinator <>, given a snapshot U and a disjoint UTxO set Uα, there does not exist a U2 which results in the same digest:
∄ U2 : digest(U <> Uα) = digest(U2 <> Uα)
or:
∀ U, Uα, U2 : digest(U <> Uα) ≠ digest(U2 <> Uα)
incrementTx
could stall the head, when participants require to observe the L1 transaction before signing snapshots referring the added UTxO.incrementTx
to all participants off chain and each one needs to sign it with their Cardano key.
incrementTx
is signed by all participants (consistent with requiredSignatories
).I have revisited the incremental commit (and also decommit) over the weekend from a "transaction trace" stand point, where I wrote out the individual paths taken on the on-chain state machine and identify what information is required in which transition (= transaction / validator). First, let me include the two hand-written graphs created, describe what is shown and followed by a discussion:
All arrows are transition between states, which also corresponds to transactions. Nodes in between are states represented by a sum-type datum (Upper case name, e.g. Open
or Closed
), while labels of transitions are individual constructors of a sum-type redeemer (lower case, e.g. close
).
Both diagrams start in the $(Open, 0, \eta_0, \bot)$ state. The scenarios A and B match the basic protocol for closing with the initial snapshot $\eta_0$ or a multi-signed snapshot $\xi_2$ about some UTxO $U_2$. Scenario C is when we get off-chain approval for increment/decrement, but we instead choose to close the head instead using that same snapshot $\xi3$. The alternative is nominally a increment/decrement transaction where UTxO is added $U\alpha$ or removed $U_\omega$ yielding a new Open state, before it is closed with either the same snapshot as it was incremented/decremented (scenario D) or later closed with any other multi-signed snapshot $\xi_5$ (nominal case E). Not shown is that $\xi5$ could still be a snapshot which has $U\alpha$ or $U_\omega$ pending.
Due to space issues, the individual components of the state variables are not labeled, but it turns out that the $Open$ state holds: a snapshot number, a current head state digest and a previous head state digest ($\bot$ in the first open state - could also be $\eta0$ in the beginning). The $Closed$ state is depicted with the same, plus an optional additional digest of to be committed UTxO to fan out ($\bot$ or $\eta\alpha$) and an optional additional digest of to be decommitted UTxO to fanout ($\bot$ or $\eta_\omega$). TBD: Can we merge them?
The approach seems to work out in a very similar fashion as these slides shared by researchers:
In summary, this seems workable, but we would better verify by also testing this state-space; ideally in a model-based fashion.
I am slightly confused by how would the on-chain checks look like with these new snapshot digests in place? How would we prove that new snapshot is a direct successor to the previous one (if we ignore snapshot numbers)?
I am slightly confused by how would the on-chain checks look like with these new snapshot digests in place? How would we prove that new snapshot is a direct successor to the previous one (if we ignore snapshot numbers)?
Both, Open
and Closed
states will contain the $\eta$ digests of the current and previous off-chain state. Hence we can check equality with a snapshots reference $\eta{\mathsf{ref}}$ with either the current $\eta$ or the previous $\eta\mathsf{prev}$, which will allow us to distinguish also the individual scenarios (i.e. handling already committed or not yet decommitted UTxOs on close
)
Here is a sparse merkle tree implementation in Aiken with a offchain component that needs just a bit more work in rust https://github.com/aiken-lang/sparse-merkle-tree Here is an example of using it in on chain code. https://github.com/MicroProofs/inverse-whirlpool
The idea behind the implementation is to have an ordered list of hashes of members and then new insertions simply need a proof of the left and right hashes to verify both the old and new root.
Latest update of example transaction traces using the above mechanism of signatures including $\eta_\alpha$:
Open question: How to detect that commit was never observed and have mechanics to remove it from the state so that the following commits are enabled again? If we don't have this then anybody could prevent other parties from committing more funds just by asking for a commit inclusion and just never follow through with posting the commit transaction on-chain. Maybe not relevant for the current iteration?
Open question: How to detect that commit was never observed and have mechanics to remove it from the state so that the following commits are enabled again? If we don't have this then anybody could prevent other parties from committing more funds just by asking for a commit inclusion and just never follow through with posting the commit transaction on-chain. Maybe not relevant for the current iteration?
IMO the root of the problem is that if the user's utxo is spent directly into a single increment tx, then you have no choice but to wait until after L2 snapshot confirmation to draft the increment tx, obtain the user's signature, and submit to L1. Thus, you are completely at the user's mercy to sign and submit the increment tx: if the user reneges, then you have to figure out a way to revoke the L2 peers' confirmation of the L2 snapshot, which is impossible if the user is an L2 peer.
The increment tx comes after the L2 snapshot confirmation because you want to support commits from a pubkey address, which can only sign a complete tx and whose signature is invalidated if the tx is modified/evolved. The only options that I see around this are:
Hydrozoa uses option 3.
Open question: How to detect that commit was never observed and have mechanics to remove it from the state so that the following commits are enabled again? If we don't have this then anybody could prevent other parties from committing more funds just by asking for a commit inclusion and just never follow through with posting the commit transaction on-chain. Maybe not relevant for the current iteration?
Indeed, this would speak for a deposit-then-L2 workflow (we had considered that at some point, but wanted to avoid spending two transactions on L1).
However, is this a realistic case? If a participant requests to add something to the head, but then does not follow-through with it. How is that lack of cooperation different than not signing a snapshot? IMO both are a similar loss of consensus and warrant head closing.
Open question: How to detect that commit was never observed and have mechanics to remove it from the state so that the following commits are enabled again? If we don't have this then anybody could prevent other parties from committing more funds just by asking for a commit inclusion and just never follow through with posting the commit transaction on-chain. Maybe not relevant for the current iteration?
Indeed, this would speak for a deposit-then-L2 workflow (we had considered that at some point, but wanted to avoid spending two transactions on L1).
However, is this a realistic case? If a participant requests to add something to the head, but then does not follow-through with it. How is that lack of cooperation different than not signing a snapshot? IMO both are a similar loss of consensus and warrant head closing.
Right, but now you have an invalid multi-signed L2 snapshot and an invalid L2 ledger state. Arguably, that's worse than just a non-responsive L2 peer, because you have to clean it up.
How does the invalid multi-signed L2 snapshot interact with the contestation mechanism if the head is closed? Is there a clear way for the contestation validator to ignore the invalid snapshot and all its descendants?
Furthermore, what if this is an external commit from a user who is not an L2 peer (e.g. under delegated head architecture)? You wouldn't want to close the head just because the external user didn't submit an incrementTx...
@ch1bo If you do decide to use a deposit workflow, I think option 2 from my post above would be the simplest to implement:
- Deposit the user's utxo into an escrow that allows spending by signed user intent
This is something that the user can do on her own, in advance, without any involvement from the hydra parties. You can think of it as a "Hydra-ready" smart wallet utxo.
Smart wallet utxos are going to be widely promoted by other Cardano dapps, which also greatly benefit from intent-based user entrypoints. For example, an AMM DEX batcher can collect user intents offchain and then post just a single batch transaction to fulfill them on L1.
(See for example @MicroProofs's thread: https://x.com/MicroProofs/status/1808586118537097724)
Hydra's API could have a utility endpoint to help a user set up a smart wallet utxo. However, the /POST Commit
would assume that the utxo to be absorbed is already in a smart wallet utxo with sufficient finality, which would be verified by HeadLogic before sending ReqInc to L2 peers.
This would allow ReqInc to include the user's signed intent for her utxo to be absorbed. This means that, as soon as the L2 snapshot is confirmed, every L2 peer will have everything needed to draft+submit the incrementTx if needed.
However, the disadvantage of this approach is that it relies on users holding funds in smart wallet addresses, which have only been proposed so far and are not well supported in wallet UIs. Then again, Hydra isn't, either. 😂
Got triggered by the thought that if we want to be safe properly secure Hydra Heads against rollbacks of incrementally committed funds, having full sequencing requirements + potentially long lockup times is going to be really bad. Also the UX workflow is an odd (see https://github.com/cardano-scaling/hydra/issues/199#issuecomment-1980456949 and https://github.com/cardano-scaling/hydra/blob/f24022532cb169eb3d404e8a70362bde37a21712/docs/docs/dev/protocol.md) mix between synchronous API and asynchronous interactive rounds on the Head.
In presence of rollbacks/pessimistic settings, this is creating a Head that is not "very live" as it would need to be forced to close in case a requested to commit UTxO is spent otherwise before the incrementTx
hits the chain.
A deposit based scheme (as @GeorgeFlerovsky and others have been exploring too) using a synchrony assumption where funds are locked "to get picked up" for longer than typical rollbacks occur, is much preferable in these points. While it will require two on-chain transactions to add funds to a head, we can be sure after the first deposit that any spend into the head (before a reclaim deadline) is going to still apply if rolled back. So we can employ varying strategy on timeouts between deposit (wait long) and increment (no need to wait).
Here is a drawing of this scenario:
We discussed the above in grooming and realised that another validator will be needed for the deposit workflow; a minting policy, because we need to ensure that things are correctly recorded into the deposit; we will need a token on the deposit, and these can be used to discover deposits.
We discussed the above in grooming and realised that another validator will be needed for the deposit workflow; a minting policy, because we need to ensure that things are correctly recorded into the deposit; we will need a token on the deposit, and these can be used to discover deposits.
(1) Why do you need to enforce correct deposit datum via an on-chain minting policy?
If the datum is incorrect, then the deposit can't be collected, but the user can reclaim it after deposit timeout.
(2) Doesn't that duplicate computation that occurs when collecting the commit later?
The datum needs to be inspected when checking that the head state merkel root is evolved properly, during collection.
(1) Why do you need to enforce correct deposit datum via an on-chain minting policy?
If the datum is incorrect, then the deposit can't be collected, but the user can reclaim it after deposit timeout.
@GeorgeFlerovsky How would you know whether the datum is correct? The transaction collecting from the deposit can't inspect the output that was deposited and needs to rely that it was recorded correctly into the datum (e.g. that the right address is recorded)
Ultimately, deposit is a two step protocol and the second step needs to rely that the first step was executed correctly. Using a minting policy and a minted token for contract continuity between the two steps is the standard technique for this.
Are there other ways to ensure this?
(2) Doesn't that duplicate computation that occurs when collecting the commit later?
The datum needs to be inspected when checking that the head state merkel root is evolved properly, during collection.
In a way, yes. It requires processing the output twice, but no duplicate computation. We are not necessarily using merkle tree structures for this feature (we might switch to that for #1468), but the processing would always be two steps: serialize and digest. The plan is that the deposit
transaction does only do the serialization, which would allow for easy observation and detecting of what was committed (no input resolution needed), while the increment
transaction would need to create a digest that matches the multi-signed snapshot (this depends again how the L2 state is represented on L1 and how snapshots are structured)
I guess my question is whether your onchain code should:
(A) merely check that the serialization parses into the correct type (i.e. something like { l2Addr :: Address, l2Datum :: PlutusData, l1DepositTimeout :: Slot, l1RefundAddr :: Address }
); or
(B) also check that the serialization corresponds to the depositor's input utxo that provided the funds to create the deposit utxo.
Computation (A) is mandatory if your onchain code is responsible for the correct evolution of the head state hash during collection.
Computation (B) is a nice guardrail to prevent the depositor from shooting himself in the foot with a correct type but incorrect value in the deposit datum, but it incurs more fees and is not strictly necessary from the perspective of the hydra head.
@GeorgeFlerovsky I disagree that (B) is optional, it is crucial for the correctness of any layer 2. Otherwise anyone could claim anything on the L2. While the damage is somewhat limited on value
as the L1 ledger would ensure that there is enough spent into the L2, it's important that any datum
is retained correctly to not break scripts when they are moved into L2.
@GeorgeFlerovsky I disagree that (B) is optional, it is crucial for the correctness of any layer 2. Otherwise anyone could claim anything on the L2. While the damage is somewhat limited on
value
as the L1 ledger would ensure that there is enough spent into the L2, it's important that anydatum
is retained correctly to not break scripts when they are moved into L2.
Fair enough 👍
Although, I'm not quite sure what you mean by "anyone could claim anything on L2" and "any datum is retained correctly to not break scripts when moving to L2".
If the deposit utxo is being created from pubkey-held funds, then the deposit's L2 address and datum can be anything that the pubkey owner wants, as authorized by his signature in the tx.
If the deposit utxo is being created from script-held funds, then the deposit's L2 address and datum can be anything that the script allows in its "SendToL2" redeemer logic.
I think the difference between our views is:
You think of committing to a head as transferring a pre-existing utxo from L1 to become an identical utxo on L2.
I think of committing to a head as creating an L2 utxo from L1 funds, without necessarily requiring an identical pre-existing L1 utxo beforehand.
@GeorgeFlerovsky When typing out the specification for the on-chain checks to be done on deposit
(here) I see now more clearly what you meant:
There is no actual need for on-chain checks of what a valid deposit is (what I would have encoded in the minting policy). Anyone paying to the deposit validator (off-chain code) should ensure that what they put is a valid datum having a deadline and if a script wants to ensure continuity of its datum, it would need to green light any deposit
transaction anyways.
While the interface between downstream scripts and the deposit protocol would become slightly more involved, it comes at the benefit of greatly simplifying the protocol transactions. But..
How would you describe this interface?
I was thinking something similar to the commit
transaction's description: https://hydra.family/head-protocol/unstable/assets/files/hydra-spec-09c867d2d94685906cbc7a74873f9de5.pdf#subsection.5.2 but we would need to describe decoding of the recorded outputs in $C$ and check consistency with locked value off-chain?
@ch1bo TBH, I haven't thought too much about that interface yet.
In a hand-wavy sense, we can adapt components from what you've previously been doing for users — serializing utxos, deserializing their datum representation, comparing redeemers to hashes, etc.
We're just pushing some of those onchain/offchain mechanisms outside the hydra protocol. The onchain parts become opt-in for scripts, while pubkey users rely on the offchain checks.
Why
Hydra Heads should not need to be closed and re-opened just to add more funds. This will make Hydra Heads more flexible in enables use cases where long-living Heads are beneficial.
Furthermore, it could pave the way for getting rid of the initialization phase altogether, which would result in a much simpler protocol.
What
Implement the protocol extension for more committing additional UTXOs into a Head as already briefly described in the original Hydra Head paper.
"As user, I want to add more funds to the head, such that I can spend them later in the head"
When the head is open, a hydra client can request an incremental commit:
POST /commit
. Just like with the "normal" commit, the user needs to send either a UTxO or a "blueprint transaction".depositTx
corresponding to the requested commit. (This works just the same way as the commit endpoint works so far during initialization phase)depositTx
Submitting the
depositTx
transaction should have the requestedUTxO
eventually added to the headCommitRequested
(TBD: orDepositDetected
?) server output is sent to signal observation of the depositUTxO
and wait for aSnapshotConfirmed
with inclusion approval.incrementTx
, signs and submits that.CommitFinalized
server output is sent to the clients when theincrementTx
is observed.The node provides a list of pending commits via the API using
GET /commits
Each pending commit (deposit) has an id (i.e. the deposit outputs'
TxIn
) and a deadline attached, after which a user can request refund of the commitDELETE /commits/<id>
, which has the node construct and submit arecoverTx
for the user. TBD: okay that node pay fees here?recoverTx
Any UTxO which can be committed, can also be incrementally committed
Scenarios
All of the positive scenarios also ensure correct balance after fanout:
incrementTx
Security
Out of scope
How
Protocol design
Outline of one deposit being claimed in an increment and one deposit being recovered:
Protocol transitions:
Situation: Head is open, $U_0$ locked, off-chain busy transacting
Deposit:
depositTx
hydra-node
or through a library/tool.depositTx
ensures through minting of a deposit token (DT) that anything to be committed $\phi$ is recorded correctly into the datum (isomorphic to $U_\alpha$)depositTx
to ensure the transition to L2 is "correct"? Could mean more coupling, but simpler protocol here.Recover:
recoverTx
after the deadline has passedIncrement:
hydra-node
) observe pending deposits using the common deposit addressdepositTx
is not rolled backA node requests inclusion of a pending decommit by sending aReqSn
message with $U_\alpha$incrementTx
, which:incrementTx
on L1 with added $U_\alpha$ and make it available in their L2 stateTo be discussed
What happens if
incrementTx
is not posted after being signed on L2?fanout
for exampleDo we really need to change
η
to be a merkle-tree-like structure with inclusion proofs?Shall we drop the initialization phase?
How to deal with rollbacks/forward which result in a different $\eta$? When is it safe to integrate $U_\alpha$ into confirmed $U$?
recoverTx
should be>>
than a safe margin on observingdepositTx
(e.g. deadline = 7 days, delay on deposit observe ~ 1 day); As deposits can only be spent into the head before the deadline passed, we don't need to wait when observingincrementTx
incrementTx
(re-use contestation period?)