Currently we store metadata part of the main DAG. This has a few issues:
The metadata is not strictly validated data so may be different between two different solana versions even processing the same slots (in theory?)
If there is corruption of metadat such as is the case #129 then we would need to re-generate the whole tree with al lthe block data.
The impact of storing the metadata in the DAG could potentially be an issue for a future proof-of-storage setup?
Proposal is:
Separate out the metadata from the current DAG. See #143 for an example idea of how this could be done.
We would move the non-validated metadata to a separate struct. In the change structure in #143.
This would mean that you would need to fetch at least two CIDs (transactionmeta and transaction) to fetch the full transaction.
tasks
[ ] implement the metadata dag generator in radiance
[ ] create a metadata generator that generates the metadata dag in faithful: this would take an existing car file, bigtable credentials and it would generate the new metadata dag from an existing car file and filling in with bigtable data where metadata is missing in the car file
impacts
extend the indexes slot-to-cid and sig-to-cid so that they have two CIDs, one is the CID of the transaction object and the other is the CID of the transactionMeta object. if the transactionMeta object CID is present, then fetch both the transactionmeta and the transaction object.
when fetching a transaction, fetch both the transaction and the transactionMeta object
when fetching a block, fetch both the block and and blockMeta object
we would need a new subset object
fixing missing meta for old epochs
for older epochs we could generate just the transactionmeta data objects we need. alternatively we could generate
backwards compatibility
we could opt to just keep the same objects we have now, but just when generating we would simple leave the rewards/metadata fields empty. if the field is empty then we know that the metadata exists in a separate tree.
when reading we could just ignore the rewards/metadata fields of existing objects and instead just read the metadata from the sepcial metadata CID.
this woudl allow us to be completely bakcwards compatible. we could then proceed to generate metadata trees for all past epochs and upload these separately.
Another option would be to create a new Block type which excludes the rewards field and a new Transaction type which excludes the metadata field. This would be a bit neater because we wouldn't have the old type around. However, the tooling woudl then need to support the two different "kind"s of Transactions and Blocks and be able to just parse it based on the kind field which one to use.
unresolved
how to use this with the standard file coin tooling to fetch a transaction with its transaction meta?
we could use just the transaction meta cid and then inside the Transactionmeta object either have a link (is this even allowed?) to the transaction object OR have a text string of the CID.
in this way someone could fetch the TransactionMeta object and then continue to fetch the transaction data too.
subset object being meta dependent?
if we want to create some kind of PoS setup, then would it be better if the Subset object was replicable?
I guess maybe it doesn't matter. since we would probably anyway need something merklized to be able to quickly verify if a specific TX was stored or not.
linuskendall
commented
1 month ago
Currently we store metadata part of the main DAG. This has a few issues:
The impact of storing the metadata in the DAG could potentially be an issue for a future proof-of-storage setup?
Proposal is:
Separate out the metadata from the current DAG. See #143 for an example idea of how this could be done.
We would move the non-validated metadata to a separate struct. In the change structure in #143.
This would mean that you would need to fetch at least two CIDs (transactionmeta and transaction) to fetch the full transaction.
tasks
impacts
fixing missing meta for old epochs
for older epochs we could generate just the transactionmeta data objects we need. alternatively we could generate
backwards compatibility
we could opt to just keep the same objects we have now, but just when generating we would simple leave the rewards/metadata fields empty. if the field is empty then we know that the metadata exists in a separate tree.
when reading we could just ignore the rewards/metadata fields of existing objects and instead just read the metadata from the sepcial metadata CID.
this woudl allow us to be completely bakcwards compatible. we could then proceed to generate metadata trees for all past epochs and upload these separately.
Another option would be to create a new Block type which excludes the rewards field and a new Transaction type which excludes the metadata field. This would be a bit neater because we wouldn't have the old type around. However, the tooling woudl then need to support the two different "kind"s of Transactions and Blocks and be able to just parse it based on the kind field which one to use.
unresolved
how to use this with the standard file coin tooling to fetch a transaction with its transaction meta?
we could use just the transaction meta cid and then inside the Transactionmeta object either have a link (is this even allowed?) to the transaction object OR have a text string of the CID.
in this way someone could fetch the TransactionMeta object and then continue to fetch the transaction data too.
subset object being meta dependent?
if we want to create some kind of PoS setup, then would it be better if the Subset object was replicable?
I guess maybe it doesn't matter. since we would probably anyway need something merklized to be able to quickly verify if a specific TX was stored or not.