search

moj-analytical-services / etl_manager

A python package to create a database on the platform using our moj data warehousing framework
21 stars 8 forks source link

Added additional pip property to GlueJob #145

Closed gwionap closed 2 years ago

gwionap commented 2 years ago

Added pip_requirements as an additional property to GlueJob. This will pass pip requirements specified in a txt file to the job definition as outlined here. This option is not enable by default, and has to be explicitly set after initialising the GlueJob object.

Basic tests have been added to ensure this change doesn't break the normal way of using GlueJob.