search

Azure / logicapps

Azure Logic Apps labs, samples, and tools
MIT License
361 stars 301 forks source link

LogicApps v2 crash under load #278

Closed Mortana89 closed 3 years ago

Mortana89 commented 3 years ago

V2, S1 plan that scales to 5 instances; Every night we receive quite some requests and the LA are triggered. We are talking about roughly 1000 requests in a small timeframe (like 5 minutes). We get a lot of failed LA runs, making it very unreliable currently. The logging does not say much besides InternalServerError. Encountered internal server error. The tracking Id is '32809a24-1fe6-47d8-bd6a-07e359c3746b'.

It's really difficult to triage, but the workflow runs that are failed are 'orchestrator' runs. This is a workflow that, based on an incoming parameter, decide which child workflow to call (async). It's the latter step that has a failed status. It's not possible for us to see if the child workflow actually got executed though (yes, both are stateful but there's no easy correlation as far we could Tell)

We are really stuck here, an attempt we took, to throttle, was using concurrency limits but this is broken in V2 as well.

shailesh-agre commented 3 years ago

During preview, there are resource limitations in place at various levels. So this is not going to be actual case. Very soon we will publish documents related to performance. We request you to test the scenario post the product goes to live.

Mortana89 commented 3 years ago

Thanks. We investigated further and root caused the issue. We use schema validation on our requests, which uses Newtonsoft under the bonnet. But apparently the LA Runtime is not using any commercial license and as such the LA engines crash when exceeding 1000 schema validations/hour.

It's logged currently, FYI

shailesh-agre commented 3 years ago

Thank you for confirming - as this will be be taken care in GA by default, closing this issue.