search

blockchain-etl / blockchain-etl-streaming

Streaming Ethereum and Bitcoin blockchain data to Google Pub/Sub or Postgres in Kubernetes
https://medium.com/google-cloud/live-ethereum-and-bitcoin-data-in-google-bigquery-and-pub-sub-765b71cd57b5
MIT License
77 stars 21 forks source link

Implement streaming_service.py #1

Closed medvedev1088 closed 4 years ago

medvedev1088 commented 6 years ago

Streaming service

  1. Continuously polls for new blocks using Ethereum JSON RPC API (e.g. every 10 seconds).
  2. Outputs data to GooglePubSubItemExporter, which publishes data to PubSub.
  3. Saves the last synchronized block to last_sync_block.txt file on every period.

This script can be used as a basis https://github.com/blockchain-etl/ethereum-etl/blob/feature/streaming/stream.py

================================

PubSub publisher guide https://cloud.google.com/pubsub/docs/publisher

There is some code in this fork https://github.com/airswap/ethereum-etl/commit/cc7f86e139d317452c9cf0c975b99feb6ed8d7a0#diff-a8e6a25bc32464a59378bac1befac9eb

TODO: Consider using geth subscriptions to handle chain reorganizations https://github.com/ethereum/go-ethereum/wiki/RPC-PUB-SUB#newheads