Port ETL pipelines to PySpark (maps onto AWS Glue)?

peacfuljoh / predictive-analytics-ytvideos

Full-stack real-time predictive anaytics for YouTube content creators

0 stars 0 forks source link

Port ETL pipelines to PySpark (maps onto AWS Glue)? #31

Closed peacfuljoh closed 9 months ago

peacfuljoh commented 9 months ago

PySpark isn't necessary given the small size of the dataset. Pipeline staging is currently implemented using generators and websockets for on-device jobs.