Simplifies the ML pipelines by removing the get_current_time component and replacing its logic with SQL. The component takes an input timestamp - if it is an empty string (or None), it returns the current time, otherwise it returns the input timestamp as-is. This PR:
Removes the component and unit tests
Removes references to the component in imports and E2E tests
Adds the same logic to each ingest.sql SQL script
How has this been tested?
Successful pipeline runs with both a timestamp and an empty string as input
Tests running successfully in the Datatonic repo here
Checklist
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have successfully run the E2E tests
- [ ] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes
[x] I have updated any relevant documentation to reflect my changes
Description
Simplifies the ML pipelines by removing the
get_current_time
component and replacing its logic with SQL. The component takes an input timestamp - if it is an empty string (or None), it returns the current time, otherwise it returns the input timestamp as-is. This PR:ingest.sql
SQL scriptHow has this been tested?
Successful pipeline runs with both a timestamp and an empty string as input
Tests running successfully in the Datatonic repo here
Checklist
- [ ] I have added tests that prove my fix is effective or that my feature works