moj-analytical-services / etl_manager

A python package to create a database on the platform using our moj data warehousing framework
21 stars 9 forks source link

Added GlueJob.wait_for_completion() #49

Closed xoen closed 6 years ago

xoen commented 6 years ago

GlueJob.wait_for_completion() allows the client code to poll for the Glue Job status until the job "completed" or failed.

By completed here we mean the job run was successful or it was manually stopped.

In addition to this, if a job failed or timed out the method will raise an exception (JobFailed and JobTimedOut respectively)

Also:

xoen commented 6 years ago

Speaking of style, what I've found useful in other places is to have VS code to auto format python using black so I don't have to spend too much mental energy on how to format code. An idea for the future maybe.

codecov-io commented 6 years ago

Codecov Report

Merging #49 into master will decrease coverage by 0.99%. The diff coverage is 45.83%.

Impacted file tree graph

@@           Coverage Diff           @@
##           master      #49   +/-   ##
=======================================
- Coverage   74.78%   73.78%   -1%     
=======================================
  Files           4        4           
  Lines         805      824   +19     
=======================================
+ Hits          602      608    +6     
- Misses        203      216   +13
Impacted Files Coverage Δ
etl_manager/etl.py 56.65% <45.83%> (-2.23%) :arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 7f8fa39...64ee856. Read the comment docs.