ServiceBus built-in Defer message action fails with Bad Request on ServiceBus Topic, When checking the service bus topic, the message shows the deferred state.

[lilan123]

Describe the Bug

ServiceBus built-in Defer message shows inconsistent behaviour when running to defer a message on the ServiceBus topic. Sometimes, it successfully defers the message; other times, it fails with a bad request result, as below. However, when checking the service bus topic, the message shows the deferred state.

NB: Action trigger Times

• When messages are available in a topic subscription (peek-lock) Start time 4/25/2024, 11:56:08 PM End time 4/25/2024, 11:56:08 PM
• Defer message Start time 4/26/2024, 12:01:37 AM End time 4/26/2024, 12:01:59 AM

{
    "statusCode": "BadRequest",
    "body": {
            "code": "ServiceProviderActionFailed",
            "message": "The service provider action failed with error code 'ServiceOperationContextUnavailable' and error message 'The call to retrieve the action context has timed out. This could be due to a scale-in event. Please try adjusting the auto-scale settings for this logic app to fix the number of active instances.'."
          }
} 

Plan Type

Standard

Steps to Reproduce the Bug or Issue

1. Setup workflow with built-in ServiceBus trigger "When messages are available in a topic subscription (peek-lock)"
2. Add built-in ServiceBus action "Defer message".
3. Add any actions. (eg. Initialize Variable)
4. Add built-in ServiceBus action "Get deferred message from a topic subscription"
5. Add built-in ServiceBus action "Complete the message"

Workflow JSON

{
    "definition": {
        "$schema": "https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json#",
        "actions": {
            "Catch": {
                "actions": {
                    "Archive_Blob_-_On_Exception": {
                        "inputs": {
                            "parameters": {
                                "blobName": "@{variables('BlobPathFailed')}/@{body('Parse_JSON')?['FileName']}",
                                "containerName": "@variables('ArchiveContainerName')",
                                "content": "@body('Reads_Blob_Content_from_Azure_Storage')?['content']"
                            },
                            "serviceProviderConfiguration": {
                                "connectionName": "AzureBlob",
                                "operationId": "uploadBlob",
                                "serviceProviderId": "/serviceProviders/AzureBlob"
                            }
                        },
                        "type": "ServiceProvider"
                    },
                    "Dead-letter_the_message": {
                        "inputs": {
                            "parameters": {
                                "deadLetterReason": "Error while transfroming",
                                "messageId": "@triggerBody()?['messageId']"
                            },
                            "serviceProviderConfiguration": {
                                "connectionName": "serviceBus",
                                "operationId": "deadLetterMessage",
                                "serviceProviderId": "/serviceProviders/serviceBus"
                            }
                        },
                        "runAfter": {
                            "Get_deferred_message_from_a_topic_subscription_on_exception": [
                                "Succeeded"
                            ]
                        },
                        "type": "ServiceProvider"
                    },
                    "Delete_blob": {
                        "inputs": {
                            "parameters": {
                                "blobName": "@body('Parse_JSON')?['FileName']",
                                "containerName": "@variables('ContainerName')"
                            },
                            "serviceProviderConfiguration": {
                                "connectionName": "AzureBlob",
                                "operationId": "deleteBlob",
                                "serviceProviderId": "/serviceProviders/AzureBlob"
                            }
                        },
                        "runAfter": {
                            "Archive_Blob_-_On_Exception": [
                                "Succeeded"
                            ]
                        },
                        "type": "ServiceProvider"
                    },
                    "Get_deferred_message_from_a_topic_subscription_on_exception": {
                        "inputs": {
                            "parameters": {
                                "sequenceNumber": "@triggerBody()?['sequenceNumber']",
                                "subscriptionName": "process-purchase-order",
                                "topicName": "replenishment-purchase-order"
                            },
                            "serviceProviderConfiguration": {
                                "connectionName": "serviceBus",
                                "operationId": "getDeferredMessageFromTopic",
                                "serviceProviderId": "/serviceProviders/serviceBus"
                            }
                        },
                        "runAfter": {
                            "Delete_blob": [
                                "Succeeded"
                            ]
                        },
                        "type": "ServiceProvider"
                    }
                },
                "runAfter": {
                    "Try": [
                        "SKIPPED",
                        "FAILED",
                        "TIMEDOUT"
                    ]
                },
                "type": "Scope"
            },
            "Defer_message": {
                "inputs": {
                    "parameters": {
                        "messageId": "@triggerBody()?['messageId']"
                    },
                    "serviceProviderConfiguration": {
                        "connectionName": "serviceBus",
                        "operationId": "deferMessage",
                        "serviceProviderId": "/serviceProviders/serviceBus"
                    }
                },
                "runAfter": {},
                "type": "ServiceProvider"
            },
            "Initialize_ArchiveContainerName": {
                "inputs": {
                    "variables": [
                        {
                            "name": "ArchiveContainerName",
                            "type": "string",
                            "value": "replenishment-archive-receive-purchaseorder"
                        }
                    ]
                },
                "runAfter": {
                    "Initialize_BlobPath_On_Failure": [
                        "Succeeded"
                    ]
                },
                "type": "InitializeVariable"
            },
            "Initialize_BlobPath_On_Failure": {
                "inputs": {
                    "variables": [
                        {
                            "name": "BlobPathFailed",
                            "type": "string",
                            "value": "replenishment-receive-purchaseorder-failed"
                        }
                    ]
                },
                "runAfter": {
                    "Initialize_ContainerName": [
                        "Succeeded"
                    ]
                },
                "type": "InitializeVariable"
            },
            "Initialize_Blob_Name": {
                "inputs": {
                    "variables": [
                        {
                            "name": "BlobName",
                            "type": "string"
                        }
                    ]
                },
                "runAfter": {
                    "Initialize_Error_Message": [
                        "Succeeded"
                    ]
                },
                "type": "InitializeVariable"
            },
            "Initialize_ContainerName": {
                "inputs": {
                    "variables": [
                        {
                            "name": "ContainerName",
                            "type": "string",
                            "value": "replenishment-receive-purchaseorder"
                        }
                    ]
                },
                "runAfter": {
                    "Defer_message": [
                        "Succeeded"
                    ]
                },
                "type": "InitializeVariable"
            },
            "Initialize_Counter": {
                "inputs": {
                    "variables": [
                        {
                            "name": "Counter",
                            "type": "integer",
                            "value": 2
                        }
                    ]
                },
                "runAfter": {
                    "Initialize_LineSplitPOC": [
                        "Succeeded"
                    ]
                },
                "type": "InitializeVariable"
            },
            "Initialize_Error_Message": {
                "inputs": {
                    "variables": [
                        {
                            "name": "ErrorMessage",
                            "type": "string"
                        }
                    ]
                },
                "runAfter": {
                    "Initialize_ArchiveContainerName": [
                        "Succeeded"
                    ]
                },
                "type": "InitializeVariable"
            },
            "Initialize_LineSplit1": {
                "inputs": {
                    "variables": [
                        {
                            "name": "LineSplit1",
                            "type": "array"
                        }
                    ]
                },
                "runAfter": {
                    "Initialize_Blob_Name": [
                        "Succeeded"
                    ]
                },
                "type": "InitializeVariable"
            },
            "Initialize_LineSplit2": {
                "inputs": {
                    "variables": [
                        {
                            "name": "LineSplit2",
                            "type": "array"
                        }
                    ]
                },
                "runAfter": {
                    "Initialize_LineSplit1": [
                        "Succeeded"
                    ]
                },
                "type": "InitializeVariable"
            },
            "Initialize_LineSplitPOC": {
                "inputs": {
                    "variables": [
                        {
                            "name": "LineSplitPOC",
                            "type": "array"
                        }
                    ]
                },
                "runAfter": {
                    "Initialize_LineSplit2": [
                        "Succeeded"
                    ]
                },
                "type": "InitializeVariable"
            },
            "Initialize_POCData": {
                "inputs": {
                    "variables": [
                        {
                            "name": "POCData",
                            "type": "string"
                        }
                    ]
                },
                "runAfter": {
                    "Initialize_Counter": [
                        "Succeeded"
                    ]
                },
                "type": "InitializeVariable"
            },
            "Initialize_POC_Notes": {
                "inputs": {
                    "variables": [
                        {
                            "name": "POCNotes",
                            "type": "string",
                            "value": ",\"POC NOTES\":["
                        }
                    ]
                },
                "runAfter": {
                    "Initialize_POCData": [
                        "Succeeded"
                    ]
                },
                "type": "InitializeVariable"
            },
            "Initialize_variable_-_vRef": {
                "inputs": {
                    "variables": [
                        {
                            "name": "vRef",
                            "type": "string"
                        }
                    ]
                },
                "runAfter": {
                    "Initialize_POC_Notes": [
                        "Succeeded"
                    ]
                },
                "type": "InitializeVariable"
            },
            "Push_Exception": {
                "actions": {
                    "Condition_to_check_if_exception_is_from_Service_Bus": {
                        "actions": {},
                        "else": {
                            "actions": {
                                "Send_message_to_exception_logger": {
                                    "inputs": {
                                        "parameters": {
                                            "entityName": "integration-exception-in",
                                            "message": {
                                                "contentData": {
                                                    "Area": "@{workflow()['name']}",
                                                    "DateTime": "@{utcNow()}",
                                                    "Description": "@{replace(variables('ErrorMessage'),'\"','')}",
                                                    "IsFreshServiceOn": "@{parameters('IsFreshServiceOn')}",
                                                    "LogicAppName": "@{parameters('LogicappName')}",
                                                    "ParentRunId": "@{body('Parse_JSON')?['ParentRunId']}",
                                                    "RecordRef": "@{variables('vRef')}",
                                                    "RunId": "@{workflow()['run']['name']}",
                                                    "Type": "Technical"
                                                },
                                                "contentType": "application/json"
                                            }
                                        },
                                        "serviceProviderConfiguration": {
                                            "connectionName": "serviceBus",
                                            "operationId": "sendMessage",
                                            "serviceProviderId": "/serviceProviders/serviceBus"
                                        }
                                    },
                                    "runAfter": {
                                        "Set_Error_Message": [
                                            "SUCCEEDED"
                                        ]
                                    },
                                    "type": "ServiceProvider"
                                },
                                "Set_Error_Message": {
                                    "inputs": {
                                        "name": "ErrorMessage",
                                        "value": "@{coalesce(first(body('Filter_array'))?['outputs'],first(body('Filter_array'))?['error'],first(body('Filter_array')))}"
                                    },
                                    "type": "SetVariable"
                                },
                                "Terminate": {
                                    "inputs": {
                                        "runStatus": "Failed"
                                    },
                                    "runAfter": {
                                        "Send_message_to_exception_logger": [
                                            "SUCCEEDED"
                                        ]
                                    },
                                    "type": "Terminate"
                                }
                            }
                        },
                        "expression": {
                            "and": [
                                {
                                    "equals": [
                                        "@if(equals(first(body('Filter_array'))?['inputs']?['serviceProviderConfiguration']?['operationId'],'getDeferredMessageFromTopic'),true,if(equals(first(body('Filter_array'))?['inputs']?['serviceProviderConfiguration']?['operationId'],'completeMessage'),true,false))",
                                        "@true"
                                    ]
                                }
                            ]
                        },
                        "runAfter": {
                            "Filter_array": [
                                "Succeeded"
                            ]
                        },
                        "type": "If"
                    },
                    "Filter_array": {
                        "inputs": {
                            "from": "@result('Try')",
                            "where": "@equals(item()?['status'], 'Failed')"
                        },
                        "type": "Query"
                    }
                },
                "runAfter": {
                    "Catch": [
                        "Succeeded",
                        "FAILED",
                        "TIMEDOUT"
                    ],
                    "Try": [
                        "FAILED"
                    ]
                },
                "type": "Scope"
            },
            "Try": {
                "actions": {
                    "Archive_Received_PO_File": {
                        "inputs": {
                            "parameters": {
                                "blobName": "@{variables('ContainerName')}/@{body('Parse_JSON')?['FileName']}",
                                "containerName": "@variables('ArchiveContainerName')",
                                "content": "@body('Reads_Blob_Content_from_Azure_Storage')?['content']"
                            },
                            "serviceProviderConfiguration": {
                                "connectionName": "AzureBlob",
                                "operationId": "uploadBlob",
                                "serviceProviderId": "/serviceProviders/AzureBlob"
                            }
                        },
                        "runAfter": {
                            "Send_message_to_topic": [
                                "Succeeded"
                            ]
                        },
                        "type": "ServiceProvider"
                    },
                    "Complete_the_message": {
                        "inputs": {
                            "parameters": {
                                "messageId": "@triggerBody()?['messageId']"
                            },
                            "serviceProviderConfiguration": {
                                "connectionName": "serviceBus",
                                "operationId": "completeMessage",
                                "serviceProviderId": "/serviceProviders/serviceBus"
                            }
                        },
                        "runAfter": {
                            "Get_deferred_message_from_a_topic_subscription": [
                                "Succeeded"
                            ]
                        },
                        "type": "ServiceProvider"
                    },
                    "Compose_Aggregated_Data": {
                        "inputs": "@outputs('Compose_PO_Header')",
                        "runAfter": {
                            "Set_vRef": [
                                "Succeeded"
                            ]
                        },
                        "type": "Compose"
                    },
                    "Compose_PO_Header": {
                        "inputs": "{\n\"PURCHASE ORDER HEADER\":{\n\"InvoiceVendorAccountNumber\":\"@{variables('LineSplit1')[2]}\",\n\"OrderVendorAccountNumber\":\"@{variables('LineSplit1')[2]}\"\n},\n\"HEADER NOTES\":{\n\"Notes\":\"@{replace(variables('LineSplit2')[43],'^',' ')}\",\n \"DocumentAttachmentTypeCode\":\"Note\",\n \"AccessRestriction\":\"External\"\n},\n\"POC\":",
                        "runAfter": {
                            "Set_LineSplit2": [
                                "Succeeded"
                            ]
                        },
                        "type": "Compose"
                    },
                    "Deletes_a_Blob_from_Azure_Storage": {
                        "inputs": {
                            "parameters": {
                                "blobName": "@body('Parse_JSON')?['FileName']",
                                "containerName": "@variables('ContainerName')"
                            },
                            "serviceProviderConfiguration": {
                                "connectionName": "AzureBlob",
                                "operationId": "deleteBlob",
                                "serviceProviderId": "/serviceProviders/AzureBlob"
                            }
                        },
                        "runAfter": {
                            "Archive_Received_PO_File": [
                                "Succeeded"
                            ]
                        },
                        "type": "ServiceProvider"
                    },
                    "Filter_Purchase_Order_Data": {
                        "inputs": {
                            "from": "@split(replace(outputs('Reads_Blob_Content_from_Azure_Storage')?['body']?['content'],'\u000b','|'),'\n')",
                            "where": "@not(equals(item(), ''))"
                        },
                        "runAfter": {
                            "Reads_Blob_Content_from_Azure_Storage": [
                                "Succeeded"
                            ]
                        },
                        "type": "Query"
                    },
                    "Get_deferred_message_from_a_topic_subscription": {
                        "inputs": {
                            "parameters": {
                                "sequenceNumber": "@triggerBody()?['sequenceNumber']",
                                "subscriptionName": "process-purchase-order",
                                "topicName": "replenishment-purchase-order"
                            },
                            "serviceProviderConfiguration": {
                                "connectionName": "serviceBus",
                                "operationId": "getDeferredMessageFromTopic",
                                "serviceProviderId": "/serviceProviders/serviceBus"
                            }
                        },
                        "runAfter": {
                            "Deletes_a_Blob_from_Azure_Storage": [
                                "Succeeded"
                            ]
                        },
                        "type": "ServiceProvider"
                    },
                    "Parse_JSON": {
                        "inputs": {
                            "content": "@triggerBody()?['contentData']",
                            "schema": {
                                "properties": {
                                    "FileName": {
                                        "type": "string"
                                    },
                                    "ParentRunId": {
                                        "type": "string"
                                    },
                                    "SolarPurchId": {
                                        "type": "string"
                                    }
                                },
                                "type": "object"
                            }
                        },
                        "type": "ParseJson"
                    },
                    "Reads_Blob_Content_from_Azure_Storage": {
                        "inputs": {
                            "parameters": {
                                "blobName": "@body('Parse_JSON')?['FileName']",
                                "containerName": "@variables('ContainerName')"
                            },
                            "serviceProviderConfiguration": {
                                "connectionName": "AzureBlob",
                                "operationId": "readBlob",
                                "serviceProviderId": "/serviceProviders/AzureBlob"
                            }
                        },
                        "runAfter": {
                            "Set_variable_-_vRef": [
                                "SUCCEEDED"
                            ]
                        },
                        "type": "ServiceProvider"
                    },
                    "Send_message_to_topic": {
                        "inputs": {
                            "parameters": {
                                "entityName": "replenishment-purchase-order",
                                "message": {
                                    "contentData": {
                                        "FileName": "@{variables('BlobName')}",
                                        "ParentRunId": "@{body('Parse_JSON')?['ParentRunId']}",
                                        "SolarPurchId": "@{variables('LineSplit2')[1]}"
                                    },
                                    "contentType": "application/json",
                                    "label": "PurchaseOrderSendData"
                                }
                            },
                            "serviceProviderConfiguration": {
                                "connectionName": "serviceBus",
                                "operationId": "sendMessage",
                                "serviceProviderId": "/serviceProviders/serviceBus"
                            }
                        },
                        "runAfter": {
                            "Set_variable_content_-_vRef": [
                                "SUCCEEDED"
                            ]
                        },
                        "type": "ServiceProvider"
                    },
                    "Set_BlobName": {
                        "inputs": {
                            "name": "BlobName",
                            "value": "@{body('Parse_JSON')?['SolarPurchId']}-Processed-@{utcNow()}.json"
                        },
                        "runAfter": {
                            "Compose_Aggregated_Data": [
                                "SUCCEEDED"
                            ]
                        },
                        "type": "SetVariable"
                    },
                    "Set_LineSplit1": {
                        "inputs": {
                            "name": "LineSplit1",
                            "value": "@split(body('Filter_Purchase_Order_Data')[0],'|')"
                        },
                        "runAfter": {
                            "Filter_Purchase_Order_Data": [
                                "Succeeded"
                            ]
                        },
                        "type": "SetVariable"
                    },
                    "Set_LineSplit2": {
                        "inputs": {
                            "name": "LineSplit2",
                            "value": "@split(body('Filter_Purchase_Order_Data')[1],'|')"
                        },
                        "runAfter": {
                            "Set_LineSplit1": [
                                "Succeeded"
                            ]
                        },
                        "type": "SetVariable"
                    },
                    "Set_vRef": {
                        "inputs": {
                            "name": "vRef",
                            "value": "Exception in processing purchase order with blob name:@{body('Parse_JSON')?['FileName']} and purchase order id:@{variables('LineSplit2')[1]} from Solar"
                        },
                        "runAfter": {
                            "Compose_PO_Header": [
                                "Succeeded"
                            ]
                        },
                        "type": "SetVariable"
                    },
                    "Set_variable_-_vRef": {
                        "inputs": {
                            "name": "vRef",
                            "value": "Exception in processing purchase order with blob name:@{body('Parse_JSON')?['FileName']} from Solar"
                        },
                        "runAfter": {
                            "Parse_JSON": [
                                "Succeeded"
                            ]
                        },
                        "type": "SetVariable"
                    },
                    "Set_variable_content_-_vRef": {
                        "inputs": {
                            "name": "vRef",
                            "value": "Exception in processing purchase order with blob name:@{variables('BlobName')} and purchase order id:@{variables('LineSplit2')[1]} from Solar"
                        },
                        "runAfter": {
                            "Uploads_a_Blob_to_Azure_Storage": [
                                "Succeeded"
                            ]
                        },
                        "type": "SetVariable"
                    },
                    "Uploads_a_Blob_to_Azure_Storage": {
                        "inputs": {
                            "parameters": {
                                "blobName": "@variables('BlobName')",
                                "containerName": "@variables('ContainerName')",
                                "content": "@outputs('Compose_Aggregated_Data')"
                            },
                            "serviceProviderConfiguration": {
                                "connectionName": "AzureBlob",
                                "operationId": "uploadBlob",
                                "serviceProviderId": "/serviceProviders/AzureBlob"
                            }
                        },
                        "runAfter": {
                            "Set_BlobName": [
                                "Succeeded"
                            ]
                        },
                        "type": "ServiceProvider"
                    }
                },
                "runAfter": {
                    "Initialize_variable_-_vRef": [
                        "SUCCEEDED"
                    ]
                },
                "type": "Scope"
            }
        },
        "contentVersion": "1.0.0.0",
        "outputs": {},
        "triggers": {
            "When_messages_are_available_in_a_topic_subscription_(peek-lock)": {
                "inputs": {
                    "parameters": {
                        "isSessionsEnabled": false,
                        "subscriptionName": "process-purchase-order",
                        "topicName": "replenishment-purchase-order"
                    },
                    "serviceProviderConfiguration": {
                        "connectionName": "serviceBus",
                        "operationId": "peekLockTopicMessages",
                        "serviceProviderId": "/serviceProviders/serviceBus"
                    }
                },
                "splitOn": "@triggerOutputs()?['body']",
                "type": "ServiceProvider"
            }
        }
    },
    "kind": "Stateful"
}

Screenshots or Videos

Additional context

No response

### Tasks

[RichaKapoor7]

did you get any workaround for this issue?

[lilan123]

@RichaKapoor7 As of now, the workaround that we implemented is to change the run after setting as below.

[RichaKapoor7]

Thanks for your reply. But message still remains in queue as deferred. Why this is happening at first place?

On Thu, 30 May, 2024, 07:57 Lilan Sameera, @.***> wrote:

@RichaKapoor7 https://github.com/RichaKapoor7 As of now, the workaround that we implemented is to change the run after setting as below. image.png (view on web) https://github.com/Azure/logicapps/assets/7977013/a18ffad0-b920-4512-ac0f-7e9441285315

— Reply to this email directly, view it on GitHub https://github.com/Azure/logicapps/issues/1049#issuecomment-2138812476, or unsubscribe https://github.com/notifications/unsubscribe-auth/BGBJH2RENQOZWHH2ONOFFHTZE3EW7AVCNFSM6AAAAABG5LZGDKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZYHAYTENBXGY . You are receiving this because you were mentioned.Message ID: @.***>

[HLChucky]

We have the same issue when completing/dead-lettering a message. Granted, it happens whilst our app service plan is scaling outwards, so the error message is correct. However - why does this happen? Why is Service Bus the only connector I've encountered that does not work well when scaling outwards?

[github-actions[bot]]

This issue is stale because it has been open for 45 days with no activity.

[github-actions[bot]]

This issue was closed because it has been inactive for 14 days since being marked as stale.