Combine All Bash Setup Scripts into One Script

njfritter / poc-data-pipelines

Proof-of-Concept (POC) Data Pipelines for various use cases such as data streaming/ingestion, batch data processing, orchestration and storage. Includes technologies such as Apache Airflow, Apache Spark, Apache Kafka, AWS, Python and more

0 stars 0 forks source link

Currently there are multiple different bash setup scripts doing different things:

docker_setup_mac.sh
kafka_python_setup_mac.sh
postgres_setup_mac.sh

These can and should be combined into one script for easier use.

Note:

Currently kafka_python_setup_mac.sh creates the Kafka topics via Kafka CLI AFTER the Kafka cluster is spun up
In order to have all of the setup done in one script, the Kafka topic creation part will need to be taken out and done separately
- Option 1: Access the Kafka container terminal (example here) and create the topics that way
- Option 2: Use the Python Kafka client (example here)

njfritter / poc-data-pipelines

Combine All Bash Setup Scripts into One Script #17