This would help reduce risks of failures / task resubmit generating an extra job when running long syncs.
Details:
Currently if Kestra worker fails at fetching the status when wait=True on this current plugin resubmitted task will trigger a new job instead of using the existing one
Flow will first trigger in a dedicated task, then “wait for status” will be done in a dedicated task.
If “wait for status” task fails (worker is dead / killed / pod restarted, this task can last for 10h+ so kind of likely) => task is resubmitted with the jobId to fetch for
No need for pod restart (due to k8s node upgrades for example) supervision
Feature description
Add an dbt task to check until job completed given a
jobId.See discussion here
This would help reduce risks of failures / task resubmit generating an extra job when running long syncs.
Details:
Currently if Kestra worker fails at fetching the status when wait=True on this current plugin resubmitted task will trigger a new job instead of using the existing one
Flow will first trigger in a dedicated task, then “wait for status” will be done in a dedicated task. If “wait for status” task fails (worker is dead / killed / pod restarted, this task can last for 10h+ so kind of likely) => task is resubmitted with the jobId to fetch for
No need for pod restart (due to k8s node upgrades for example) supervision
I am willing to do a PR 👍