Public Lakehouse: fix mainnet_explorer_public.aggregated__circulating_supply

[abhishekanirudhan]

The table mainnet_explorer_public.aggregated__circulating_supply has aggregate supply statistics. This table stopped updating on Mar 21, 2024.

We use this table to understand total supply, including circulating and locked, and total staked and unstaked by day.

[eduohe]

Implementing Aggregated Circulating Supply Pipeline

• https://4221960800361869.9.gcp.databricks.com/?o=4221960800361869#notebook/730949852826779/command/2072158440953824

based on these references:

• https://github.com/near/near-indexer-for-explorer/blob/master/circulating-supply/src/main.rs#L160
• https://github.com/near/core-contracts/blob/master/lockup/src/getters.rs#L65

[eduohe]

@abhishekanirudhan there are lockup contracts that contains a bug in the lockup amount calculation method, for this reason we can't rely entirely on RPC function calls. Indexer for explorer handled this issue with a custom logic. I created a RUST API that implement the Indexer for explorer logic and deployed in GCP Cloud Run to use in this pipeline.

• Code: https://github.com/near/near-public-lakehouse/tree/main/rust-extract-apis/lockups
• RUST API endpoint: https://near-public-lakehouse-24ktefolwq-uc.a.run.app/lockup_amount

The API returns the lockup amount per account that will give us better data granularity.

New databricks tables:

• mainnet.silver_aggregated_lockups
• mainnet.silver_lockup_amount_account_daily
• mainnet.silver_lockup_amount_nf_daily and
• mainnet.gold_aggregated_circulating_supply (to replace previous mainnet_explorer_public.aggregated__circulating_supply )

Currently running a backfill for the dates > 2024-04-22

[eduohe]

backfill finished