Design and Implement Analytics Pipeline in Amazon Glue for Python Code Execution

[kaipak]

Currently, the ETL pipeline is a loosely federated set of scripts, AWS Glue/Athena SQL queries, and calculated fields in QuickSight. Investigate AWS options for running arbitrary Python code. Future migrations might implement Airflow or other pipeline framework, so the solution here may be temporary or should be designed to be specific, and relatively low effort/complexity.

The solution should adhere to following

• Allow execution of arbitrary Python code
• Relatively simple implementation, should not be a full-blown data pipeline or ETL solution.
• Modest compute requirement. AI model inference or training is not a requirement.
• Easily maintainable.
• Allow for Python code that can be easily migrated to Airflow or other more full-featured pipeline solution.

Sub-tasks

• https://github.com/NASA-PDS/web-analytics/issues/19

[jordanpadams]

@kaipak how did you create this ticket? I notice there are no labels or anything associated with it, so it appears to have not been created using a template.

[jordanpadams]

📆 03/2024 status: In work. On schedule.

[kaipak]

@jordanpadams I was about to start work on this ticket this week, I can recreate it if it's causing issues with CI/CD.

[jordanpadams]

@kaipak nope. you are good to go.

[jordanpadams]

📆 04/2024 status: In work. On schedule.

[jordanpadams]

📆 05/2024 status: Task pack due to lack of resources and having to roll task team off the project. Hoping to restart towards end of build. Manual workaround available, so no impact on build.