search

rpcpool / yellowstone-faithful

Project Yellowstone Old Faithful is the project to make all of Solana's history accessible, content addressable and available via a variety of means.
https://old-faithful.net/
GNU Affero General Public License v3.0
72 stars 16 forks source link

Create a small tool to iterate a car file and detect missing metadata #158

Open linuskendall opened 3 weeks ago

linuskendall commented 3 weeks ago

We need a tool that iterates a car file and identifies which transactions are missing metadata.

linuskendall commented 3 weeks ago

Replaces #154

AzothZephyr commented 2 weeks ago

gm, so im interested in taking this and #154 on early next week if there's limited capacity available. im coming in mildly half cocked but from what @notwedtm has expressed some early epochs contain transactions which are missing the metadata field.

what are your thoughts around a --verify flag which takes a directory then iterates over each .car file within that directory to execute verify logic on each tx within it to close this issue? that same verification logic could then be applied at the end of a car files generation to verify the underlying data as close to source as possible then potentially expanded out to cover other cases elsewhere in the dag if need be, which should close #154 as well.

im still going through the codebase but i think this is a good approach as it gives the ability for operators who are already using / hosting car files the ability verify their store without having to blindly pull after a fix is produced and insures we don't see this issue without some form of visibility moving forward.

will move forward next week unless @gagliardetto already has a solution cooking

az

Lusitaniae commented 2 weeks ago

checking a single file could be best, in case people don't have all CAR files locally and it would be easier to parallelized using xargs/parallel on the invocation

AzothZephyr commented 2 weeks ago

very good point, i will add a subcommand for --file / -f and --dir / -d. doesn't add much work wise but covers for both cases.