search

blockchain-etl / blockchain-etl-streaming

Streaming Ethereum and Bitcoin blockchain data to Google Pub/Sub or Postgres in Kubernetes
https://medium.com/google-cloud/live-ethereum-and-bitcoin-data-in-google-bigquery-and-pub-sub-765b71cd57b5
MIT License
77 stars 21 forks source link

Implement Kubernetes manifests for ethereum-etl-streaming #3

Closed medvedev1088 closed 4 years ago

medvedev1088 commented 6 years ago

A single Pod runs:

  1. download-last_synced_block - Job that downloads last_synced_block.txt from GCS bucket on node start up. See also init containers.
  2. streaming_service.py - streams blockchain data to PubSub
  3. health_checker.py - listens on HTTP port for health check requests
  4. gcs-syncd - uploads last_synced_block.txt periodically to GCS bucket and has a preStop hook that uploads the file to GCS bucket before Pod is terminated.
  5. fluentd daemon - uploads logs to GCS bucket

====================

Some code can be found here https://github.com/airswap/ethereum-etl/commit/cc7f86e139d317452c9cf0c975b99feb6ed8d7a0#diff-aa0ae5de65a67b108e5a26ddde9d3adf