search

davidgasquez / gitcoin-grants-data-portal

🌲 Open source, serverless, and local-first data hub for Gitcoin Grants data!
https://grantsdataportal.xyz/
MIT License
26 stars 3 forks source link

Pull infrequent Transactions from Covalent API #24

Closed DistributedDoge closed 7 months ago

DistributedDoge commented 8 months ago

For bulk data (e.g. all votes) it is probably best to use RPC approach outlined in #1 but for events/transactions that are less frequent, free tier of Covalent API could work as lightweight alternative.

As proof of concept I would start with Project Registry contract as it has few transactions so free covalent API can handle fetching them in reasonable time, across all important chains and without consuming too many free monthly credits.

Implementation outline would be:

End goal is knowing how much gas was spent on creating and updating project profiles on different chains + reconciling whatever Indexer is telling us with another source.

davidgasquez commented 8 months ago

Agree! Love this idea and it actually moves "trust" in the right direction.

The ideal goal would be to get all the data from your own nodes, but that's only an ideal. The solution you propose moves us into that direction so let's go for it!

davidgasquez commented 8 months ago

As a side note, would love to check how long does it takes cryo to get the same data from the interesting networks.

Is the Project Registry contract you're thinking about 0x4AAcca72145e1dF2aeC137E1f3C5E3D75DB8b5f3?

DistributedDoge commented 8 months ago

I am thinking about Allo v1 where contracts aren't as nicely lined up across chains, each having different address. Best source I found so far is deployment list in documentation -> chains.md.

I made a .parquet file with all contracts addresses + chain names and pinned it to IPFS here, data was extracted like so.

ProjectRegistry (proxy) for mainnet is here, 0x03506eD3f57892C85DB20C36846e9c808aFe9ef4

davidgasquez commented 8 months ago

You know what is the status of Allo v1 vs v2?

I made a .parquet file with all contracts addresses + chain names and pinned it to IPFS here, data was extracted like so.

Yessss! Should we put the extraction logic into it's own Dagster asset?