[Indexer] Implement ETL and a Scalable Database Platform - Githubissues

hack-a-chain-software / indexer-kadena

0 stars 2 forks source link

[Indexer] Implement ETL and a Scalable Database Platform #9

Closed veigajoao closed 6 months ago

veigajoao commented 8 months ago

Implement an ETL framework to work with data saved in S3
Implement ETL flows described in the contract/milestone

Load entities from s3 and save it into database

[x] [Blocks] Create entity
[x] [Blocks] Map to table
[x] [Blocks] Handle processing
[x] [Events] Create entity
[x] [Events] Map to table
[x] [Events] Handle processing
[x] [Transactions] Create entity
[x] [Transactions] Map to table
[x] [Transactions] Handle processing
[x] Add resume from the last sync
[x] [Transfers] Create entity
[x] [Transfers] Map to table
[x] [Transfers] Handle processing
[x] [Balances] Create entity
[x] [Balances] Map to table
[x] [Balances] Handle amount
[x] Add metrics
[x] Add cron for retry
[x] Add cron for missing blocks
[ ] Improve performance
[ ] Improve documentation
[ ] Refactory sync service
[ ] Add support for running processes in parallel
[ ] Run full sync (s3)
[ ] Run full sync (database)

veigajoao commented 8 months ago

https://github.com/near/near-lake-framework-rs

p-destri commented 6 months ago

milestone was approved by Kadena's team, therefore I am marking this task as finished