Nike-Inc / brickflow

Pythonic Programming Framework to orchestrate jobs in Databricks Workflow
https://engineering.nike.com/brickflow/
Apache License 2.0
183 stars 39 forks source link

[BUG] Change date/time parsing in AutosysSensor #89

Closed maxim-mityutko closed 7 months ago

maxim-mityutko commented 7 months ago

Closes #88 The date/time parsing in AutosysSensor is format sensitive. We can use dateutil.parser to make the process format agnostic.

Description

Using the dateutil.parser to make internal logic of AutosysSensor agnostic to the format of the execution_date provided by the Brickflow context; and using the same approach to parse lastEndUTC string to timestamp.

Related Issue

[BUG] Faulty execution_date parser in AutosysSensor

Motivation and Context

This solves the cases when/if upstream Autosys API changes and the lastEndUTC timestamp is provided in different format; It solved error when parsing the execution_date which format may be different depending on how the job is triggered.

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

Checklist: