jarry-xiao / candyland

Proof of concept program for compressing NFTs
GNU Affero General Public License v3.0
111 stars 13 forks source link

Danenbm/flatbuffers integration #49

Closed danenbm closed 2 years ago

danenbm commented 2 years ago

Updated ingest/main.rs to now work with flatbuffers serialized transaction data. Ingest still needs some cleanup based on async design ideas for messenger and better error handling.

Key files: geyser_plugin_nft.rs <-- use new methods to serialize then send. redis_messenger.rs <-- simplified sending. serializer.rs <-- does the flatbuffer serialization account_info.fbs, block_info.rbs, slot_status_info.rbs, transaction_info.rbs <-- flatbuffer schema files. ingest/main.rs <-- See comment above.

Testing

Although continuous gummyroll test is not working on mainline at the moment, if I compare the database entries after running continuous gummyroll test, the hash values stored in the database no matter if the test is run on mainline or this branch, so it seems like it did not regress from main.

Update: @ngundotra tested on his setup and it passed. We did some debugging on my setup and uncovered a race condition when serving proofs. @ngundotra is working on fixing that one on the API server side. When I modified the client side test to retry on mismatches, the test passes.

jarry-xiao commented 2 years ago

This is so cool. Almost feels like something that could go straight into geyser rather than as a module for this project in particular. Moving away from evil things like Borsh into things like flatbuffers makes a ton of sense for building sustainable codebases

danenbm commented 2 years ago

Thank you for the comments @jarry-xiao. Originally I wanted to make serialization a trait that can be implemented on arbitrary types such as ReplicaTransactionInfo, but ran into a few issues that I would need more time to solve. Overall I do agree flatbuffers is good, @austbot's idea to use them here.

ngundotra commented 2 years ago

Verified that this code compiles & works with the continuous test