Support TriggerDagRunOperator

[y2k-shubham]

• looking into default plugin, it becomes apparent than TriggerDagRunOperator is presently not supported
• and it it understandable since that would require validation not just within DAG / YAML (which is presently done by networkx library), but across them .
• while ExternalTaskSensor do manage to capture dependencies across DAGs, but they fail to provide the same reactive / eager triggering as TriggerDagRunOperator does
• Moreover there are cases when a sensor simply cannot replace a triggering-operator without messing up entire schedules of DAGs .
• is there any plan to support them in future?
• is there a neat approach using which this can be implemented?

[mchalek]

hi @y2k-shubham , and thanks for the question. It is a good point about the difficulty of verifying the cross-dag dependencies that are introduced by these operators. At Etsy we have had a longstanding plan to add cross-dag dependency checking to our deployment infrastructure, but it's still not done (and we have had our fair share of errors due to incorrect ExternalTaskSensor specifications as a consequence).

However, given that we do support ExternalTaskSensor, it seems reasonable to me that we could also support TriggerDagRunOperator as well. We would only have to add a configuration for this operator here (and if you would like to do this, we would be happy for the contribution!).

Thanks for the reminder about the need for the cross-dag checking. With the holiday season coming up, we should have more down-time than usual; perhaps opportunities to add this will present themselves.