microsoft / coe-starter-kit

Other
743 stars 219 forks source link

Admin | Sync Template v4 (Flows) Succeeds for over 95% of almost 10k flows, but hangs "running" for about 4% #7732

Closed RickKush closed 6 months ago

RickKush commented 7 months ago

Does this bug already exist in our backlog?

Describe the issue

Admin | Sync Template v4 (Flows) runs successfully for over 9,000 flows, but hangs in the running state for a little over 400 flows. There are no errors, warning or other clues.

Expected Behavior

Run to completion for all flows: either success, failure, or cancelled.

What solution are you experiencing the issue with?

Core

What solution version are you using?

4.23

What app or flow are you having the issue with?

Admin | Sync Template v4 (Flows)

What method are you using to get inventory and telemetry?

Cloud flows

Steps To Reproduce

Run Admin | Sync Template v3 (Driver) and then look at the run history for Admin | Sync Template v4 (Flows). Filter for "running" and see each flow ID repeated and still running once for each day.

Anything else?

No response

AB#2386

RickKush commented 7 months ago

How can I get the flow run history into a csv or xls? When I do "Get .csv file" in the flow run history, it only exports the most recent 100 runs no matter how patiently I have paged thru hundreds and hundreds of runs that are all still running.

RickKush commented 7 months ago

image

RickKush commented 7 months ago

image

RickKush commented 7 months ago

image

RickKush commented 7 months ago

image

RickKush commented 7 months ago

image

Jenefer-Monroe commented 7 months ago

You are likely being throttled by the product for API calls. I believe you are setting up new yes? If so, you can likely expect your volume to work going forward as we do not touch every object and only those that have changed goning forward. In the meantime you can turn the Driver flow off so that new runs dont get triggered until these complete.

Long term the solution here will be the BYODL architecture which, as you've found is not ready yet to come out of Preview. Or you can purchase Power Automate Process licenses and apply them to long running flows in your kit.

RickKush commented 7 months ago

Sorry, this runs per environment, not per flow, and it is hanging for about 400 environments out of 1,200 total environments.

RickKush commented 7 months ago

That sounds great, but there is one problem: Over 700 of these flow runs succeeded and completed yesterday, and not one of them ran for more than two minutes. The ones that are running hours and days are not completing. Is there a flag somewhere that tells the driver to run for all environments instead of just the ones that have actually changed?

Jenefer-Monroe commented 7 months ago

Every environment gets updated every day which in turns runs the app, ai model, bpf, etc flow for every environment.

That's how we then see if there are any new / updatd flows etc. By running the associated inventory flow for each environment. That's not configurable its the basic architecture of the sync flow system.

The reason the other ones are running and not moving along is that you are API limit throttled. You only get a certain number of API calls per day from flow for a license.

So your options are

  1. switch to BYODL (per other conversations not an option today due to blocking bugs and the preview nature of the area)
  2. buy additional Power Automate Process licenses to put on these long running flows
  3. (my recommendation) turn off the Driver flow and wait until all your flows complete, they will eventually seems like it might take a couple days. Then when they do, turn the driver flow back on and see if you still hit throttles after this initial starter inventory.
RickKush commented 7 months ago

I have stopped the driver, and I have 1,156 flow instances running. I have been watching this for a couple hours, and not one of them has completed, but the durations do keep increasing. It would be nice to get some sort of export to use for tracking, but saving to csv only saves the first 100 and powershell doesn't get the environment IDs or the duration, or even the status, like running.

Jenefer-Monroe commented 7 months ago

Yes unfortunately the export features are limited, that's product feature stuff though and unfortunately we are not staffed to assist with product support. I dont know of a better way to do that. The API count refreshes every 24 hours. So I suggest you just wait a few days.

RickKush commented 7 months ago

Booz Allen Hamilton Internal

The user already has Power Automate Premium, per user:

Enables licensed users to automate modern applications via API-based digital process automation (cloud flows) and legacy applications via UI-based robotic process automation (desktop flows) in attended mode.

From: Jenefer Monroe @.> Sent: Tuesday, February 27, 2024 2:17 PM To: microsoft/coe-starter-kit @.> Cc: Kushner, Rick [USA] @.>; Author @.> Subject: [External] Re: [microsoft/coe-starter-kit] Admin | Sync Template v4 (Flows) Succeeds for over 95% of almost 10k flows, but hangs "running" for about 4% (Issue #7732)

Every environment gets updated every day which in turns runs the app, ai model, bpf, etc flow for every environment.

That's how we then see if there are any new / updatd flows etc. By running the associated inventory flow for each environment. That's not configurable its the basic architecture of the sync flow system.

The reason the other ones are running and not moving along is that you are API limit throttled. You only get a certain number of API calls per day from flow for a license.

So your options are

  1. switch to BYODL (per other conversations not an option today due to blocking bugs and the preview nature of the area)
  2. buy additional Power Automate Processhttps://urldefense.com/v3/__https:/powerautomate.microsoft.com/en-us/pricing/__;!!May37g!NehFC1XLInPwPD85w_GWNI4HF4JSnAApahN1VjwPnf8geVDf8WjVwmIi9Df9wCNBlUtavH5hVF4EEppzf58QIgvH6Q$ licenses to put on these long running flows
  3. (my recommendation) turn off the Driver flow and wait until all your flows complete, they will eventually seems like it might take a couple days. Then when they do, turn the driver flow back on and see if you still hit throttles after this initial starter inventory.

- Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/microsoft/coe-starter-kit/issues/7732*issuecomment-1967430264__;Iw!!May37g!NehFC1XLInPwPD85w_GWNI4HF4JSnAApahN1VjwPnf8geVDf8WjVwmIi9Df9wCNBlUtavH5hVF4EEppzf5_jHkXIxA$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/A32AAGS5JRRH5UZ6T7K2PL3YVYWMHAVCNFSM6AAAAABD3BQKUCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRXGQZTAMRWGQ__;!!May37g!NehFC1XLInPwPD85w_GWNI4HF4JSnAApahN1VjwPnf8geVDf8WjVwmIi9Df9wCNBlUtavH5hVF4EEppzf5-xnsf2JA$. You are receiving this because you authored the thread.Message ID: @.**@.>>

Jenefer-Monroe commented 7 months ago

I'm not sure where you got that from but I presume that's about the number of flows, not the number of api requests There are a ton of different limitation types on all the license types. You can learn about them here: Limits of automated, scheduled, and instant flows

Jenefer-Monroe commented 6 months ago

closing out as no further action for starter kit team