search

cerc-io / watcher-ts

GNU Affero General Public License v3.0
12 stars 3 forks source link

Store watcher chains in FileCoin #111

Open ashwinphatak opened 2 years ago

AFDudley commented 2 years ago

This will require creating an IPLD Codec for Watchers. We may also want to evaluate using some of the CosmosSDK merklization tooling to save ourselves some trouble.

AFDudley commented 2 years ago

Since we want all the state and intermediate nodes available, we should avoid looking at zk-baded aggregation/compression.

Hashlimks should provide sufficient compression, at some point we will want to evaluate existing hashing algorithms in this context.

i-norden commented 2 years ago

SSZ seems like a good candidate as well

i-norden commented 2 years ago

Well, I think it merits some discussion anyways. SSZ would be good if we envision it being desirable to provide concise proofs for individual fields in the watcher chain "blocks". But it also has significant overhead vs simply hashing the entire "block".

E.g. If we used SSZ for each block we could provide concise proofs for e.g. the transformation code only- since references to it would be a field in the block- which is neat but still need to figure out what all the practical implications/tradeoffs are.

i-norden commented 2 years ago

Going hand-in-hand with this is the IPLDization of the watcher (code) itself.

i-norden commented 2 years ago

Raw chain input =A> Block0(data fields, references to raw chain input + code A) =B> Block1(data fields, references to Block0 + code B) =C> Block2(data fields, references to Block1 + code C)

For Block0 the references are to the raw chain input + the watcher code used to perform transformation A. For Block1 the references are to Block0 + the watcher code used to perform transformation B. For Block2 the references are to Block1 + the watcher code used to perform transformation C.

If the chain uses only one type of transformation then we can save space by only having the reference to that transformation code at the beginning of the chain e.g. Block0 (although the tradeoff would be that when verifying a later block model, e.g. Block2, you would need to work your way all the way back to the beginning of the chain to figure out the transformation you want to reapply in the verification process).

We can introduce similar compression in a chain of variable transformations whereby we only include a reference to the transformation code at the blocks in the chain where a transformation occurs that is different than at the previous block. Every transformation-homogenous segment contains only one code reference at the beginning of that chain segment.

E.g.

Raw chain input =A> Block0(data fields, references to raw chain input + code A) =A> Block1(data fields, references to Block0) =A> Block2(data fields, references to Block1) =B> Block3(data fields, references to Block2 + code B) =B> Block4(data fields, references to Block3) =A> Block5(data fields, references to Block4 + code A) =A> Block6(data fields, references to Block5)

i-norden commented 2 years ago

The above is poorly formatted, sorry.

i-norden commented 2 years ago

We should also consider using protobuf encoding for the individual fields in the consensus encoding of these block objects (ofc in the relational DB the fields would no be protobuf serialized values, but when we merkleize the chain we could consider them being so). I think SSZ for encoding and merkleization of the object as a whole + protobuf encoding for the individual fields in that object would be really powerful. But I'm also quite possibly getting carried away with the "cool" theoretical factor of this while neglecting practical ramifications.