Design and Implement Analytics Pipeline in Amazon Glue for Python Code Execution

kaipak commented 8 months ago

Currently, the ETL pipeline is a loosely federated set of scripts, AWS Glue/Athena SQL queries, and calculated fields in QuickSight. Investigate AWS options for running arbitrary Python code. Future migrations might implement Airflow or other pipeline framework, so the solution here may be temporary or should be designed to be specific, and relatively low effort/complexity.

The solution should adhere to following

Allow execution of arbitrary Python code
Relatively simple implementation, should not be a full-blown data pipeline or ETL solution.
Modest compute requirement. AI model inference or training is not a requirement.
Easily maintainable.
Allow for Python code that can be easily migrated to Airflow or other more full-featured pipeline solution.

Sub-tasks

https://github.com/NASA-PDS/web-analytics/issues/19

jordanpadams commented 7 months ago

@kaipak how did you create this ticket? I notice there are no labels or anything associated with it, so it appears to have not been created using a template.

jordanpadams commented 7 months ago

📆 03/2024 status: In work. On schedule.

kaipak commented 7 months ago

@jordanpadams I was about to start work on this ticket this week, I can recreate it if it's causing issues with CI/CD.

jordanpadams commented 7 months ago

@kaipak nope. you are good to go.

jordanpadams commented 6 months ago

📆 04/2024 status: In work. On schedule.

jordanpadams commented 5 months ago

📆 05/2024 status: Task pack due to lack of resources and having to roll task team off the project. Hoping to restart towards end of build. Manual workaround available, so no impact on build.

jordanpadams commented 3 months ago

📆 06/2024 status: unable to bring on more staffing. deferring. Manual workaround available, so no impact on build.

NASA-PDS / web-analytics

Design and Implement Analytics Pipeline in Amazon Glue for Python Code Execution #34

Sub-tasks