Feature Request: Multiple trigger types for jobs

[rhuanbarreto]

Is your feature request related to a problem? Please describe.

Today in the ARM template or the CLI you can only choose either "Manual", "Schedule" or "Event". Even if you have the type "Schedule" or "Event", manual triggers are possible. But if you configure as "Schedule" or "Event" only one prevail.

Describe the solution you'd like.

I wanted to be able to select multiple types of triggers. Or at least have cron and event available at the same time.

Describe alternatives you've considered.

Today I'm experimenting with the cron scaler from keda to try to have multiple event triggers. But didn't finish it yet.

[anthonychu]

@rhuanbarreto Can you please provide more details about the job that you're building? (What does it do? What event trigger?)

The guidance today is to deploy 2 jobs.

[rhuanbarreto]

We are moving pipelines from Synapse Analytics to Container Apps Jobs due to ease of deployment, simplified billing and speed (Spark clusters in Synapse are too slow).

We have python ETL jobs using dlt that can be triggered either manually (for reprocessing/backfill), in a schedule (for delta loads) or through file events (whenever a new json file comes into a blob storage folder.

At the end I discovered ACA doesn't support the cron KEDA trigger for jobs (Azure API rejects the app update). So I'm splitting the schedule and event job in 2 jobs like you said. But for the sake of concurrency, then the scheduled and the event can run at the same time, so I cannot prevent both from running concurrently.

So I hope at some point we could have only one instance with support to both trigger types (Schedule and Event, as you can manually trigger through the portal independently of trigger) so we could control better the concurrency behaviour.

[anthonychu]

You'll need to implement concurrency control yourself, perhaps the event driven job can use the ARM API to see if there's a running execution of the scheduled job. The event driven job can wait or exit if the scheduled job is running. https://learn.microsoft.com/en-us/rest/api/containerapps/jobs-executions/list?view=rest-containerapps-2024-08-02-preview&tabs=HTTP

You could also try calling the new suspend/resume API from the scheduled job to suspend the event driven job when it starts and resume it when it completes.

https://learn.microsoft.com/en-us/rest/api/containerapps/jobs/suspend?view=rest-containerapps-2024-08-02-preview&tabs=HTTP https://learn.microsoft.com/en-us/rest/api/containerapps/jobs/resume?view=rest-containerapps-2024-08-02-preview&tabs=HTTP

Another possibility is you can have an event driven job that is triggered by both blob and a queue. When it runs, if there's a message in a queue, it'll run a delta load and then deletes the message; otherwise it'll process a single file. You'll still need a second scheduled job, but its only action is to add a message to a queue.

[rhuanbarreto]

Thanks for the explanation and the new endpoints @anthonychu! All the points you mentioned are workarounds to make things work. Most probably I will add a queue event and an event sourcer to create messages at the specific times to trigger the jobs.

For now it looks like ACA won't support those scenarios ootb without those workarounds in the long run, so I'm closing this request.

Thanks a lot for the clarification! This explanation here will help others with the same request.