microsoft / azure-container-apps

Roadmap and issues for Azure Container Apps
MIT License
362 stars 29 forks source link

Container app job with event trigger connected to Blob Storage queue breaks connection if queue is cleared #1125

Closed dani2221 closed 5 months ago

dani2221 commented 6 months ago

Please provide us with the following information:

This issue is a: (mark with an x)

Issue description

On a container app job with an event trigger connected to a blob storage queue, if the queue is cleared (either from the cli or the portal), the container app job starts executing every pooling interval regardless if a message is in the queue. For an example if the polling interval is set to 5 minutes, the job would execute every 5 minutes whilst having no messages in the queue. Recreating the queue fixes the problem temporarily, but it is incredibly hard to diagnose that the problem was the queue and not the Job.

Steps to reproduce

  1. Create a container app job with blob storage queue event trigger following the tutorial
  2. Clear the queue using the portal or az storage message clear
  3. The container job starts executing on a schedule every polling interval

Expected behavior [What you expected to happen.] Clearing the queue should not brake the connection with the job

Actual behavior [What actually happened.] The container app job starts executing every polling interval on a schedule regardless of the presence of a message in the queue

Additional context

Issue occurs with both CLI and portal

anandanthony commented 6 months ago

@dani2221 we tried to reproduce the issue using the steps that you have provided, basically we tried the following:

  1. Added a message to our queue.
  2. Confirmed that in the next polling interval a job execution triggered (for the message in step# 1).
  3. Added another message in the queue.
  4. Cleared the message from the queue using az storage message clear -q <queueName> --account-name <storageAccountName> before the next polling interval.
  5. Confirmed that there was no execution triggered in the next polling interval since the queue was empty due to step# 4.

Do you see job execution getting triggered even if there are no messages in the queue (let's say there was only one message in the queue which was processed by a job execution, now when the queue is empty without even using the az cli command), does it still trigger an execution for your job at regular polling intervals. If this is happening, then maybe the minExecutions for your scale rule is greater than 0? Can you please confirm.

dani2221 commented 1 month ago

Actually found the root issue. The container job starts executing if a message is added from the azure portal. Steps to reproduce:

An ugly fix for this that worked for me was to delete and recreate the queue with the connection string