colorfulnotion / polkaholic

Polkaholic - Substrate Chain Block Explorer for Polkadot/Kusama Chains and Parachains
https://polkaholic.io
GNU General Public License v3.0
31 stars 7 forks source link

Evm-Etl: processChainSteps refactoring #342

Closed mkchungs closed 1 year ago

mkchungs commented 1 year ago

Consistent naming convention across: local/gs.

const step0_createRec = 0
    function: cpblk(dt, chainID)

const step1_cpblk = 1
    function: cpblk(dt, chainID)
    Source: crypto_ethereum..
    Output: gs://evm_etl/YYYY/MM/DD/chainID/

const step2_backfill = 2
    function: backfill(dt, chainID)
    Source: gs://evm_etl/YYYY/MM/DD/chainID/
    LocalTmp: /tmp/evm_etl/YYYY/MM/DD/chainID/
    Output: cbt evmchain${chainID}

const step3_indexEvmChain = 3
    function: index_evmchain(chainID, dt)
    Source: cbt evmchain${chainID}
    LocalTmp: /tmp/evm_decoded/YYYY/MM/DD/chainID/tableId*

const STEP4_cpEvmDecodedToGS = 4
    function: cpEvmDecodedToGS(dt, chainID, dryrun)
    Input: /tmp/evm_decoded/YYYY/MM/DD/chainID/tableId..
    Output: gs://evm_decoded/YYYY/MM/DD/tableId../chainID.json

const STEP5_loadGSEvmDecoded = 5
    function: loadGSEvmDecoded(dt, chainID)
    Input:gs://evm_decoded/YYYY/MM/DD/
    localInput: /disk1/evmschema

NOTE: local indexing must use /YYYY/MM/DD/chainID/ to prevent concurrent indexing issues