MetricsDAO / harmony_dbt

DBT Project for the Harmony Blockchain
12 stars 8 forks source link

Implement 'ingested_at' for incremental tables #102

Closed antonyip closed 2 years ago

antonyip commented 2 years ago

Description

Please include a summary of changes and related issue (if any). Continuation of https://github.com/MetricsDAO/harmony_dbt/issues/98

Tests

05:06:58 | Concurrency: 4 threads (target='default') 05:06:58 | 05:06:58 | 1 of 60 START table model DEV.ant_hero_rentals....................... [RUN] 05:06:58 | 2 of 60 START table model DEV.ant_labels............................. [RUN] 05:06:58 | 3 of 60 START table model DEV.dfk_last_30_days....................... [RUN] 05:06:58 | 4 of 60 START table model DEV.dfk_price_attempt_above_gen1........... [RUN] 05:07:01 | 2 of 60 OK created table model DEV.ant_labels........................ [SUCCESS 1 in 2.91s] 05:07:01 | 5 of 60 START incremental model DEV.stg_ant_ingest................... [RUN] 05:07:02 | 4 of 60 OK created table model DEV.dfk_price_attempt_above_gen1...... [SUCCESS 1 in 3.77s] 05:07:02 | 6 of 60 START incremental model DEV.stg_blocks....................... [RUN] 05:08:58 | 1 of 60 OK created table model DEV.ant_hero_rentals.................. [SUCCESS 1 in 119.87s] 05:08:58 | 7 of 60 START incremental model DEV.stg_tranquil_tvl................. [RUN] 05:09:22 | 3 of 60 OK created table model DEV.dfk_last_30_days.................. [SUCCESS 1 in 143.87s] 05:09:22 | 8 of 60 START incremental model DEV.stg_txs.......................... [RUN] 05:09:34 | 6 of 60 OK created incremental model DEV.stg_blocks.................. [SUCCESS 1 in 152.64s] 05:09:34 | 9 of 60 START incremental model DEV.tokenprice_usd................... [RUN] 05:09:39 | 9 of 60 OK created incremental model DEV.tokenprice_usd.............. [SUCCESS 1 in 4.88s] 05:09:39 | 10 of 60 START table model DEV.tokens................................ [RUN] 05:11:52 | 7 of 60 OK created incremental model DEV.stg_tranquil_tvl............ [SUCCESS 1 in 173.72s] 05:11:52 | 11 of 60 START incremental model DEV.blocks.......................... [RUN] 05:11:58 | 10 of 60 OK created table model DEV.tokens........................... [SUCCESS 1 in 138.86s] 05:12:17 | 11 of 60 OK created incremental model DEV.blocks..................... [SUCCESS 1 in 25.39s] 05:12:17 | 12 of 60 START table model DEV.metric_gas_used_count................. [RUN] 05:12:17 | 13 of 60 START incremental model DEV.metric_miner_count.............. [RUN] 05:12:23 | 12 of 60 OK created table model DEV.metric_gas_used_count............ [SUCCESS 1 in 5.63s] 05:12:26 | 13 of 60 OK created incremental model DEV.metric_miner_count......... [SUCCESS 1 in 9.17s] 05:14:38 | 5 of 60 OK created incremental model DEV.stg_ant_ingest.............. [SUCCESS 1 in 457.25s] 05:14:38 | 14 of 60 START incremental model DEV.stg_delegators.................. [RUN] 05:14:38 | 15 of 60 START incremental model DEV.stg_multichain.................. [RUN] 05:14:38 | 16 of 60 START incremental model DEV.stg_staked_one.................. [RUN] 05:15:08 | 16 of 60 OK created incremental model DEV.stg_staked_one............. [SUCCESS 1 in 30.08s] 05:15:08 | 17 of 60 START incremental model DEV.stg_total_supply................ [RUN] 05:15:23 | 15 of 60 OK created incremental model DEV.stg_multichain............. [SUCCESS 1 in 45.05s] 05:15:23 | 18 of 60 START incremental model DEV.stg_validators.................. [RUN] 05:15:49 | 18 of 60 OK created incremental model DEV.stg_validators............. [SUCCESS 1 in 26.03s] 05:16:31 | 17 of 60 OK created incremental model DEV.stg_total_supply........... [SUCCESS 1 in 82.98s] 05:18:30 | 14 of 60 OK created incremental model DEV.stg_delegators............. [SUCCESS 1 in 232.32s] 05:18:30 | 19 of 60 START incremental model DEV.stg_delegators_counts........... [RUN] 05:18:30 | 20 of 60 START incremental model DEV.validators...................... [RUN] 05:19:43 | 20 of 60 OK created incremental model DEV.validators................. [SUCCESS 1 in 72.69s] 06:08:05 | 19 of 60 OK created incremental model DEV.stg_delegators_counts...... [SUCCESS 1 in 2974.65s] 10:35:40 | 8 of 60 OK created incremental model DEV.stg_txs..................... [SUCCESS 1 in 19577.87s] 10:35:40 | 21 of 60 START incremental model DEV.logs............................ [RUN] 10:35:40 | 22 of 60 START incremental model DEV.txs............................. [RUN] 11:23:58 | 22 of 60 OK created incremental model DEV.txs........................ [SUCCESS 1 in 2898.54s] 11:23:58 | 23 of 60 START incremental model DEV.dfk_potions_crafted............. [RUN] 11:23:58 | 24 of 60 START view model DEV.metric_transactions_count.............. [RUN] 11:23:58 | 25 of 60 START table model DEV.metric_unique_users_count............. [RUN] 11:23:59 | 24 of 60 OK created view model DEV.metric_transactions_count......... [SUCCESS 1 in 1.24s] 11:23:59 | 26 of 60 START table model DEV.metric_wallet_count................... [RUN] 11:26:25 | 25 of 60 OK created table model DEV.metric_unique_users_count........ [SUCCESS 1 in 146.48s] 11:26:25 | 27 of 60 START incremental model DEV.stg_daily_addresses............. [RUN] 11:28:24 | 26 of 60 OK created table model DEV.metric_wallet_count.............. [SUCCESS 1 in 264.52s] 11:28:24 | 28 of 60 START table model DEV.stg_new_addresses_daily............... [RUN] 11:29:01 | 23 of 60 OK created incremental model DEV.dfk_potions_crafted........ [SUCCESS 1 in 302.85s] 11:29:47 | 27 of 60 OK created incremental model DEV.stg_daily_addresses........ [SUCCESS 1 in 201.87s] 11:29:50 | 28 of 60 OK created table model DEV.stg_new_addresses_daily.......... [SUCCESS 1 in 85.87s] 12:11:55 | 21 of 60 OK created incremental model DEV.logs....................... [SUCCESS 1 in 5775.01s] 12:11:55 | 29 of 60 START incremental model DEV.ant_logs........................ [RUN] 12:11:55 | 30 of 60 START incremental model DEV.dfk_bank_transactions........... [RUN] 12:11:55 | 31 of 60 START incremental model DEV.dfk_hero_levelup................ [RUN] 12:11:55 | 32 of 60 START incremental model DEV.dfk_hero_summons................ [RUN] 12:13:11 | 32 of 60 OK created incremental model DEV.dfk_hero_summons........... [SUCCESS 1 in 75.90s] 12:13:11 | 33 of 60 START incremental model DEV.dfk_hero_updates................ [RUN] 12:14:35 | 31 of 60 OK created incremental model DEV.dfk_hero_levelup........... [SUCCESS 1 in 160.54s] 12:14:35 | 34 of 60 START incremental model DEV.dfk_seed_harvests............... [RUN] 12:15:28 | 30 of 60 OK created incremental model DEV.dfk_bank_transactions...... [SUCCESS 1 in 213.54s] 12:15:28 | 35 of 60 START table model DEV.fct_dfk_latest_sync_event............. [RUN] 13:23:34 | 29 of 60 OK created incremental model DEV.ant_logs................... [SUCCESS 1 in 4299.27s] 13:23:34 | 36 of 60 START table model DEV.liquidity_pools....................... [RUN] 13:24:24 | 34 of 60 OK created incremental model DEV.dfk_seed_harvests.......... [SUCCESS 1 in 4188.75s] 13:24:24 | 37 of 60 START incremental model DEV.tokenprice_jewel................ [RUN] 13:25:22 | 35 of 60 OK created table model DEV.fct_dfk_latest_sync_event........ [SUCCESS 1 in 4193.31s] 13:25:22 | 38 of 60 START incremental model DEV.transfers....................... [RUN] 13:26:01 | 36 of 60 OK created table model DEV.liquidity_pools.................. [SUCCESS 1 in 146.89s] 13:26:01 | 39 of 60 START incremental model DEV.stg_dfk_latest_sync_event....... [RUN] 13:26:04 | 39 of 60 OK created incremental model DEV.stg_dfk_latest_sync_event.. [SUCCESS 1 in 3.16s] 13:26:04 | 40 of 60 START incremental model DEV.dfk_seed_liquidity_actions...... [RUN] 13:26:58 | 37 of 60 OK created incremental model DEV.tokenprice_jewel........... [SUCCESS 1 in 154.31s] 13:26:58 | 41 of 60 START incremental model DEV.swaps........................... [RUN] 13:29:06 | 40 of 60 OK created incremental model DEV.dfk_seed_liquidity_actions. [SUCCESS 1 in 182.38s] 13:29:06 | 42 of 60 START incremental model DEV.dfk_hero_rental................. [RUN] 13:31:49 | 41 of 60 OK created incremental model DEV.swaps...................... [SUCCESS 1 in 290.64s] 13:31:49 | 43 of 60 START incremental model DEV.dfk_hero_sales.................. [RUN] 13:33:56 | 42 of 60 OK created incremental model DEV.dfk_hero_rental............ [SUCCESS 1 in 289.69s] 13:33:56 | 44 of 60 START incremental model DEV.tokenprice_btc.................. [RUN] 13:34:04 | 38 of 60 OK created incremental model DEV.transfers.................. [SUCCESS 1 in 522.10s] 13:34:04 | 45 of 60 START incremental model DEV.tokenprice_dfkgold.............. [RUN] 13:34:58 | 44 of 60 OK created incremental model DEV.tokenprice_btc............. [SUCCESS 1 in 62.14s] 13:34:58 | 46 of 60 START incremental model DEV.tokenprice_eth.................. [RUN] 13:35:44 | 43 of 60 OK created incremental model DEV.dfk_hero_sales............. [SUCCESS 1 in 235.03s] 13:35:44 | 47 of 60 START incremental model DEV.tokenprice_gaiatear............. [RUN] 13:36:09 | 45 of 60 OK created incremental model DEV.tokenprice_dfkgold......... [SUCCESS 1 in 125.56s] 13:36:09 | 48 of 60 START incremental model DEV.tokenprice_one.................. [RUN] 13:36:58 | 46 of 60 OK created incremental model DEV.tokenprice_eth............. [SUCCESS 1 in 119.60s] 13:36:58 | 49 of 60 START incremental model DEV.tokenprice_shvasrune............ [RUN] 13:37:01 | 47 of 60 OK created incremental model DEV.tokenprice_gaiatear........ [SUCCESS 1 in 77.49s] 13:37:01 | 50 of 60 START incremental model DEV.stg_tokenprice_from_swaps....... [RUN] 13:37:52 | 48 of 60 OK created incremental model DEV.tokenprice_one............. [SUCCESS 1 in 102.71s] 13:37:52 | 51 of 60 START incremental model DEV.btc_bridge...................... [RUN] 13:37:54 | 49 of 60 OK created incremental model DEV.tokenprice_shvasrune....... [SUCCESS 1 in 55.89s] 13:37:54 | 52 of 60 START table model DEV.metric_erc20_transfers_count.......... [RUN] 13:38:03 | 52 of 60 OK created table model DEV.metric_erc20_transfers_count..... [SUCCESS 1 in 9.16s] 13:38:03 | 53 of 60 START incremental model DEV.dfk_crystal_summons............. [RUN] 13:38:12 | 51 of 60 OK created incremental model DEV.btc_bridge................. [SUCCESS 1 in 20.09s] 13:38:12 | 54 of 60 START incremental model DEV.tokenprice_stone................ [RUN] 13:39:21 | 54 of 60 OK created incremental model DEV.tokenprice_stone........... [SUCCESS 1 in 69.38s] 13:39:21 | 55 of 60 START incremental model DEV.dfk_quest_rewards............... [RUN] 13:47:58 | 53 of 60 OK created incremental model DEV.dfk_crystal_summons........ [SUCCESS 1 in 594.82s] 13:47:58 | 56 of 60 START table model DEV.tranquil_markets_tokenprice........... [RUN] 13:48:00 | 56 of 60 OK created table model DEV.tranquil_markets_tokenprice...... [SUCCESS 1 in 1.92s] 13:48:00 | 57 of 60 START table model DEV.tranquil_txs.......................... [RUN] 13:50:09 | 57 of 60 OK created table model DEV.tranquil_txs..................... [SUCCESS 1 in 129.19s] 13:50:09 | 58 of 60 START table model DEV.tranquil_daily_supply_borrow.......... [RUN] 13:51:44 | 55 of 60 OK created incremental model DEV.dfk_quest_rewards.......... [SUCCESS 1 in 742.45s] 13:52:03 | 58 of 60 OK created table model DEV.tranquil_daily_supply_borrow..... [SUCCESS 1 in 114.19s] 13:53:17 | 50 of 60 OK created incremental model DEV.stg_tokenprice_from_swaps.. [SUCCESS 1 in 975.66s] 13:53:17 | 59 of 60 START incremental model DEV.tokenprices..................... [RUN] 13:53:21 | 59 of 60 OK created incremental model DEV.tokenprices................ [SUCCESS 1 in 3.65s] 13:53:21 | 60 of 60 START incremental model DEV.sushiswap_lp_actions............ [RUN] 13:54:13 | 60 of 60 OK created incremental model DEV.sushiswap_lp_actions....... [SUCCESS 1 in 52.69s] 14:35:16 | 33 of 60 OK created incremental model DEV.dfk_hero_updates........... [SUCCESS 1 in 8525.64s] 14:35:16 | 14:35:16 | Finished running 15 table models, 44 incremental models, 1 view model in 34099.35s.

Completed successfully

Done. PASS=60 WARN=0 ERROR=0 SKIP=0 TOTAL=60


- [x] Any comparison between `prod` and `dev` for any schema change
-- most tables have an additional column now `ingested_at`

# Checklist
- [x] Follow [dbt style guide](https://github.com/dbt-labs/corp/blob/main/dbt_style_guide.md)
- [x] Tag the person(s) responsible for reviewing proposed changes
-- @chuxinh 
- [x] Notes to deployment, if a `full-refresh` is needed for any table
-- full-refresh is required -- Takes about 9 hours to deploy