ETL(Extract, transform, and load )

wj-turner / OmegaFin

This open-source project is designed for gathering, processing, and analyzing financial data.

5 stars 3 forks source link

ETL(Extract, transform, and load ) #2

Open wj-turner opened 11 months ago

wj-turner commented 11 months ago

Apache NiFi
Apache Airflow
pandas in Python

wj-turner commented 11 months ago

Data Streaming & Processing: Kafka (a distributed event streaming platform) can handle large volumes of streaming data. It's scalable and can efficiently handle the streaming of tick data for multiple forex pairs. Apache Kafka with Kafka Streams or KSQL can be used for real-time data processing.