search

WebOfTrust / keripy

Key Event Receipt Infrastructure - the spec and implementation of the KERI protocol
https://keripy.readthedocs.io/en/latest/
Apache License 2.0
60 stars 55 forks source link

current main branch regression: KEL of a delegated `aid` lacks `source seal couples` #842

Closed VasiliyS closed 3 months ago

VasiliyS commented 3 months ago

Version

main brainch (e37f427)

Environment

linux, python 3.12.5

Expected behavior

scripts/demo/basic/delegate.sh should result in a KEL that has appropriate 'source seal couples`.

Actual behavior

/oobi/EPCqFjBoALCddSFjYKDEPXgUIPsLTVaQsTSrnMHMuxdu/witness/BBilc4-L3tFUnfM_wJr4S4OJanAv_VmF_dJNN6vkf2Ha produces a KEL where cesr attachments for dip and drt events lack source seal couples.

delegate_with_drt.e37f427.cesr.txt

Steps to reproduce

  1. setup keripy from the main branch
  2. start kli witness demo
  3. from another terminal, from keripy directory:
  4. source scripts/demo/demo-scripts.sh
  5. run scripts/demo/basic/delegate.sh
  6. run curl 'http://127.0.0.1:5642/oobi/EPCqFjBoALCddSFjYKDEPXgUIPsLTVaQsTSrnMHMuxdu/witness/BBilc4-L3tFUnfM_wJr4S4OJanAv_VmF_dJNN6vkf2Ha'
  7. check output (see attached file above)
daidoji commented 3 months ago

This part.

  -GAB
    0AAAAAAAAAAAAAAAAAAAAAAB
    EDrzTw3aqn-B6Hi9BO4KQ4gqX0eFj7FmubRoIhxxL9Ic

Maybe the regression happened as a move to CESR2.0 where -G will become -Q but it seems like this shouldn't be removed.

VasiliyS commented 3 months ago

@daidoji - yes, this part is missing. It works as intended in the v1.1.18 branch, though.

Looking at the code in the main branch, it looks like there are "todo"s to handle proper storage of the 'source seal couples' due to ongoing refactoring to CESR2.0.

I'd be great to somehow make it clear which branch is maintenance/development work on the "production code" (e.g. vLEI ecosystem) and which is the new stuff. Ideally the new stuff, with breaking changes, shouldn't be in the main, IMHO.

daidoji commented 3 months ago

@VasiliyS yeah I agree. I think the convention is that all maintenance work happens on the last tagged stable branch and that main is the development branch. Should be in one of the READMEs but I can't remember which.

pfeairheller commented 3 months ago

@daidoji - yes, this part is missing. It works as intended in the v1.1.18 branch, though.

Looking at the code in the main branch, it looks like there are "todo"s to handle proper storage of the 'source seal couples' due to ongoing refactoring to CESR2.0.

I'd be great to somehow make it clear which branch is maintenance/development work on the "production code" (e.g. vLEI ecosystem) and which is the new stuff. Ideally the new stuff, with breaking changes, shouldn't be in the main, IMHO.

This was discussed in detail at several community calls and we took community input to establish the current workflow.

pfeairheller commented 3 months ago

@VasiliyS yeah I agree. I think the convention is that all maintenance work happens on the last tagged stable branch and that main is the development branch. Should be in one of the READMEs but I can't remember which.

https://github.com/WebOfTrust/keripy/discussions/726

pfeairheller commented 3 months ago

This is a tricky issue that has nothing to do with the CESR changes.

During the work to support superseding delegated rotation (#680 ) a change was made to the delegation flow to collect witness receipts before getting delegation approval. This is a bit more secure as a delegation request would require compromise of the controller's keys as well as the witnesses before it is submitted to the delegator.

The result is that witnesses will now witness (provide receipts) for a delegated event before the delegation approval seal is created in the delegator's KEL. Which also means that the witnesses never receive the AES attachment and won't unless we implement a second pass through the witnesses to propagate the KEL with the attachment after approval is received.

This is also related to issues I have heard from others that OOBIs from delegated AIDs don't have the delegator's KEL in them and thus 3rd parties can not resolve the OOBI without first OOBI-ing with the delegator. (I think @m00sey may open an issue related to this soon).

The AES attachment is not a cryptographic requirement for the KEL to be valid (how could it be when it is not signed?). It is merely a convenience pointer to help validators find the anchoring event. It is the anchoring event that is the requirement for validation.

I'm going to add this as a discussion item for the up coming community meeting so we can talk through alternative solutions. Now that witnesses approve delegated events before the delegator, one approach that I'm pretty sure we need is to have the delegate send the delegator's KEL to their witnesses (they used to do this when the order was reversed) after approval is received and have the witnesses create the AES entry themselves.

To see proof of this, simply run a kli export... from the delegate and you'll see the -G attachment gets returned.

VasiliyS commented 3 months ago

The result is that witnesses will now witness (provide receipts) for a delegated event before the delegation approval seal is created in the delegator's KEL. Which also means that the witnesses never receive the AES attachment and won't unless we implement a second pass through the witnesses to propagate the KEL with the attachment after approval is received. ... Now that witnesses approve delegated events before the delegator, one approach that I'm pretty sure we need is to have the delegate send the delegator's KEL to their witnesses (they used to do this when the order was reversed) after approval is received and have the witnesses create the AES entry themselves.

@pfeairheller - thanks, that's clear!

Yeah, it'd be great if the approach where delegate sends the delegator's KEL was implemented. Otherwise any validator would have to do a linear search through all the ixn and rot (unless EO trait is in effect) of the delegators's KEL. And think hard what to do if the respective anchor (seal) is not found - either it is still in the"works" or the delegate is actually lying...

m00sey commented 3 months ago

In the agenda for tomorrow's dev call:

https://github.com/WebOfTrust/keri/discussions/94

SmithSamuelM commented 3 months ago

In general sending either the latest events from one's KEL or a keystatenotice as part of any interaction with another component is a good way to "percolate" key state in advance of some interaction. So for example, when a delegatee adds a new event to its KEL, it knows all the dependencies of that event already, so it can percolate them.

For thoes of you are familiar with network percolation theory, it can be shown that discovery via percolation is a very robust way to distribute global state without requiring a centralized discovery mechanism.

For example direct mode works best when key state is percolated between the parties. that is how all the direct mode demos work.

Indirect mode similarly benefits when parties percolate. On average the network bandwidth required to poll is usually more than to preemptively percolate. While sometime percolation is redundant, it saves all the back and forth of polling.

So once a delegatee gets a satisfying set of witness receipts it can go get the delegator's approval and then propagate that to its (delegatee's) witnesses. If it may be the case that the witnesses don't have the delegator's KEL, then the delegatee, who had to have it in order to confirm the delegator's approval, can percolate the latest delegator KEL events to its witnesses along with the seal of the approving (anchoring event). An intellegent component can build a dependency graph of information for any update and percolate the dependencies in stride but ahead of the update itself.

Because any discloser of some update of information i.e downstream dissemination must first verify that information prior to downstream dissemination then that discloser must have also been able to verify all the dependencies of that information. Which means the discloser has all the dependencies already. Therefore, in general, any discloser can percolate that dependencies prior to sending the update. The receipient will then also have in its possession all the dependencies prior to having to verify the update.

This is the fundamental insight of percolated discovery. One can discover on a NTK JIT basis via percolation without any centralized discovery mechanisms by simply understanding the dependency graph of any update. NTK (need to know given by the dependency graph), JIT (just in time, dependencies streamed immediately prior to update)

SmithSamuelM commented 3 months ago

For me when writing code for communicating between components of KERI (Infrastructure) I create a dependency graph of what must be verified in order to verify the new information and if I don't have good reason to believe the recipient already has every one of the dependencies then I want to percolate any presumed missing dependencies prior to sending the update. I can do this by prepending the dependencies in the stream.

Indeed, percolated discovery is enabled by KERI/CESR's asynchronous streaming support of data. Mentally it requires some adjustment to how one thinks about the architecture but the advantage of not having to have a centralized discovery mechanism is worth it. One just has to think in terms of percolation. Polling and escrows are a fault tolerant fallback not the primary mechanism for discovery.

The KERI overview slide deck includes some slides with references and a discussion of why we use percolation with OOBIs as the primary discovery mechanism.

pfeairheller commented 3 months ago

The code for witness receipting was written to behave this way and in fact it still does behave this way. The problem is that when the controller sends her KEL to the witness, we inverted the order so the controller no longer has the delegator's anchoring event when the KEL is sent to the witnesses. So we would need to add to the "2nd" pass where we send all the receipts back to the witnesses to now include the anchoring event from the delegator's KEL.

There will be one additional fix needed. Currently the witnesses won't have any trigger to create the AES database entry and resending the delegates KEL with the attachment does not solve the problem because the witness has already seen the event and does not attempt to persisting anything when seeing it again. That is what I would like to discuss tomorrow.

SmithSamuelM commented 3 months ago

@pfeairheller

anchoring event when the KEL is sent to the witnesses. So we would need to add to the "2nd" pass where we send all the receipts back to the witnesses to now include the anchoring event from the delegator's KEL.

I understand Phil, I was just trying to explain why one can't avoid "a second pass" as its being called because the dependency graph changed.

SmithSamuelM commented 3 months ago

@pfeairheller

There will be one additional fix needed. Currently the witnesses won't have any trigger to create the AES database entry and resending the delegates KEL with the attachment does not solve the problem because the witness has already seen the event and does not attempt to persisting anything when seeing it again. That is what I would like to discuss tomorrow.

Good topic

SmithSamuelM commented 3 months ago

Is suspect there are several possible solutions. One economical approach would be to send a reciept with the seal and have the receipt processing persist it. Since receipts already do that for signatures. I.e. event has already been seen but the receipt indicates that some attachment associated with that event needs to be processed after the fact of first seeing the event. Alternatively or in addition, when receiving an event that is already first seen, there is no guarantee that all the attachments of the current encounter are the same as the first seen encounter. Therefore the attachments of any encounter should always be processed. So maybe just a logic change in the process events.