Add initial Kafka piece of Data Pipeline

njfritter / poc-data-pipelines

Proof-of-Concept (POC) Data Pipelines for various use cases such as data streaming/ingestion, batch data processing, orchestration and storage. Includes technologies such as Apache Airflow, Apache Spark, Apache Kafka, AWS, Python and more

0 stars 0 forks source link

Add initial Kafka piece of Data Pipeline #2

Closed njfritter closed 7 months ago

njfritter commented 7 months ago

Add the first piece of the Kafka Spark Data Pipeline:

Gett API credentials for Coinbase
Getting a Kafka cluster installed and running locally
Getting Kafka CLI tool and Python environment setup
Querying Coinbase API and writing data to a Kafka topic
Instructions on how to get this running (on a mac for now)

njfritter commented 7 months ago

This is a retroactive issue; this was completed by this pull request.

Closing issue.