As an airflow user
I want the validation script to spend less than 5 minutes running
So that I can get on with my day.
Value / Purpose
Currently the version of regex used for validation in Airflow is about 4 years old, and takes an extremely long time to install as part of workflows. As such, we should update to the latest version of regex (as in this PR) to save about 5 minute per run. This will save a significant amount of compute time, and as such, money for the business.
Useful Contacts
@jhpyke
User Types
No response
Hypothesis
If we update regex
Then validation will continue working the way we expect
Proposal
We only use fullmatch and compile from regex. As such, could compare the code for those functions between versions for functionality changes. Alternatively, we can try and manually violate each of the policies that rely on these functions and ensure they still work as intended.
Additional Information
Could be used as an opportunity to write unit tests for validation?
Definition of Done
[x] We are able to update the regex package version
[x] Validation script still performs as expected
[ ] Support is prepared to respond to any issues arising from the upgrade
jacobwoffenden
commented
2 months ago
User Story
As an airflow user I want the validation script to spend less than 5 minutes running So that I can get on with my day.
Value / Purpose
Currently the version of regex used for validation in Airflow is about 4 years old, and takes an extremely long time to install as part of workflows. As such, we should update to the latest version of regex (as in this PR) to save about 5 minute per run. This will save a significant amount of compute time, and as such, money for the business.
Useful Contacts
@jhpyke
User Types
No response
Hypothesis
If we update regex Then validation will continue working the way we expect
Proposal
We only use
fullmatchandcompilefromregex. As such, could compare the code for those functions between versions for functionality changes. Alternatively, we can try and manually violate each of the policies that rely on these functions and ensure they still work as intended.Additional Information
Could be used as an opportunity to write unit tests for validation?
Definition of Done