search

microsoft / coe-starter-kit

Other
739 stars 216 forks source link

[CoE Starter Kit - BUG] SYNC HELPER - Cloud Flows is taking hours to complete the each run #8820

Closed devipriyanka414 closed 2 weeks ago

devipriyanka414 commented 4 weeks ago

Does this bug already exist in our backlog?

Describe the issue

Most of the runs from SYNC HELPER - Cloud Flows are taking so many hours to complete.

Sync Helper - Cloud Flows Sync Helper - Cloud Flows 2

Expected Behavior

No response

What solution are you experiencing the issue with?

Core

What solution version are you using?

4.33

What app or flow are you having the issue with?

SYNC HELPER - Cloud Flows

What method are you using to get inventory and telemetry?

Cloud flows

Steps To Reproduce

No response

Anything else?

No response

Jenefer-Monroe commented 4 weeks ago

This implies that your user is being throttled. Might I ask about how many environments are in your tenant? And about how many flows? That will help me advice what to do next.

devipriyanka414 commented 4 weeks ago

We have 1500+ environments and 27k flows exist in our tenant

Jenefer-Monroe commented 4 weeks ago

Thank you, and is this a new install? Or has it been around for a while?

devipriyanka414 commented 3 weeks ago

It's been more than 2 years and we've been encountering this issue for several months. It only affects this particular flow.

Jenefer-Monroe commented 3 weeks ago

Can you please check the value of these environment variables?

You can do this in the CoE Admin Command Center as shown here image

devipriyanka414 commented 3 weeks ago

Below are the values of the environment variables Full Inventory - Yes is All Environments Inventory - Yes InventoryFilter_DaysToLookBack - 7

Jenefer-Monroe commented 3 weeks ago

Oh wonderful we found the issue.

Full inventory = yes means that you are inventorying every object every time, instead of only looking for changed ones, you are therefore being throttled. You want that env var to be no and that will mean only flows that are new or recently modified will go through this flow.

  1. If possible, cancel all the currently running instances of SYNC HELPER - Cloud Flows

  2. Then change that value Full Inventory to No.

  3. Restart all the flows that reference this environment variable. You can find them by going to the default solution > environment variables > FullInventory > Show Dependenies image image

That should clean it up for you.

devipriyanka414 commented 3 weeks ago

We'll update the environment variable and restart the flows to check if the issue is resolved. Thanks! :)

Jenefer-Monroe commented 2 weeks ago

closing out as no further action for starter kit team