Pipeline for Octan Network

The pipeline collects data from multiple blockchains, processes them, then provides as data-inputs for reputation ranking system and warehouse to build an onchain analytics platform.

Support cloud native deployment (kubernetes) with autoscaler Fork from blockchain-etl

Features

• evm indexer: ./indexer
  • export file format:
    • [X] csv
    • [X] json new line
    • [X] parquet
    • [ ] iceberg (depend on issue)
  • export sink:
    • [X] s3
    • [X] local
    • [X] other filesystem follow fsspec (not tested, but highly compatible)
    • [X] clickhouse
    • [ ] starrocks
  • streaming sink:
    • [X] kafka
    • [X] kinesis
    • [X] postgres
    • [ ] clickhouse
  • trace specs:
    • [X] parity traces
    • [X] geth traces (only export)
• scheduler/monitor: airflow
• labeling engine: graphql
• graph processing
  • [X] graphframes (deprecated)
  • [X] graphscope
• resilient in memory distributed storage
  • [ ] vineyard v6d
• algorithm:
  • [X] contract classification
  • [X] in degree
  • [X] pagerank
  • [ ] fraud detection
  • [ ] bot detection

Structure

• docker: build of docker image
• k8s: yaml resources for k8s components
• docs