Streams the following Ethereum entities to Pub/Sub or Postgres using ethereum-etl stream:
Streams blocks and transactions to Pub/Sub using bitcoin-etl stream. Supported chains:
gcloud container clusters create ethereum-etl-streaming \
--zone us-central1-a \
--num-nodes 1 \
--disk-size 10GB \
--machine-type custom-2-4096 \
--network default \
--subnetwork default \
--scopes pubsub,storage-rw,logging-write,monitoring-write,service-management,service-control,trace
kubectl
credentials:gcloud container clusters get-credentials ethereum-etl-streaming \
--zone us-central1-a
Create Pub/Sub topics (use create_pubsub_topics_ethereum.sh
). Skip this step if you need to stream to Postgres.
Create GCS bucket. Upload a text file with block number you want to start streaming from to
gs://<YOUR_BUCKET_HERE>/ethereum-etl/streaming/last_synced_block.txt
.
Create "ethereum-etl-app" service account with roles:
Download the key. Create a Kubernetes secret:
kubectl create secret generic streaming-app-key --from-file=key.json=$HOME/Downloads/key.json -n eth
brew install helm
helm init
bash patch-tiller.sh
values
dir and adjust all the files at least with your bucket and project ID.
helm install --name btc --namespace btc charts/blockchain-etl-streaming --values values/bitcoin/bitcoin/values.yaml
helm install --name bch --namespace btc charts/blockchain-etl-streaming --values values/bitcoin/bitcoin_cash/values.yaml
helm install --name dash --namespace btc charts/blockchain-etl-streaming --values values/bitcoin/dash/values.yaml
helm install --name dogecoin --namespace btc charts/blockchain-etl-streaming --values values/bitcoin/dogecoin/values.yaml
helm install --name litecoin --namespace btc charts/blockchain-etl-streaming --values values/bitcoin/litecoin/values.yaml
helm install --name zcash --namespace btc charts/blockchain-etl-streaming --values values/bitcoin/zcash/values.yaml
helm install --name eth-blocks --namespace eth charts/blockchain-etl-streaming \ --values values/ethereum/values.yaml --values values/ethereum/block_data/values.yaml helm install --name eth-traces --namespace eth charts/blockchain-etl-streaming \ --values values/ethereum/values.yaml --values values/ethereum/trace_data/values.yaml
Ethereum block and trace data streaming are decoupled for higher reliability.
To stream to Postgres:
```bash
helm install --name eth-postgres --namespace eth charts/blockchain-etl-streaming \
--values values/ethereum/values-postgres.yaml
Refer to https://github.com/blockchain-etl/ethereum-etl-postgres for table schema and initial data load.
describe
command to troubleshoot, f.e.:kubectl describe pods -n btc
kubectl describe node [NODE_NAME]
Refer to blockchain-etl-dataflow for connecting Pub/Sub to BigQuery.