[CoE Starter Kit - Feature]: Improve Performance of Audit Log import and record update

[alanmWDC]

Is your feature request related to a problem? Please describe.

The Audit log sync Flow can take 20 hours to run in large organizations as there can be 80 or more content files to download and process, plus making a query to the Power Apps table to do an record check and update.

Describe the solution you'd like

Improve the speed by dumping the content initially into the Audit Logs table as is and then use Power Automate to update the Power Apps record and relate or do this on a schedule.

Describe alternatives you've considered

Alternatively use a DataFlow to do an upsert of the last run matching on the AppID and DataFlow to dump all records into the Audit Log table.

Additional context?

Would the new solution using the Audit log dumps out of Power Platform improve this?

AB#1709

[Jenefer-Monroe]

Thanks for posting this idea. Putting on backlog to explore.

[Jenefer-Monroe]

With the work for https://github.com/microsoft/coe-starter-kit/issues/7167, this is an easy add. Targeting for next release as a result.

[Jenefer-Monroe]

This now happens in this flow: Admin | Audit Logs | Update Data (V2)

Removed from: Admin | Audit Logs | Sync Audit Logs (V2)

Only did it for launch event, we can see if its necessary to do more but hopefully not.

[alanmWDC]

Hi,

In practice I encountered some other challenges with large environments just pulling the audit log so this solution might be suited to medium but not large organisations.

My new approach is a Runbook that uses the Search-UnifiedAuditLog PowerShell command to just get the operation I need, and then dump these somewhere to import with a DataFlow. I suspect the Microsoft Power Platform self-service analytics export Power Platform inventory and usage data (preview) - Power Platform | Microsoft Learnhttps://learn.microsoft.com/en-us/power-platform/admin/self-service-analytics will be the best data source to use but need to assess the storage costs of having the Azure Data Lake.

Just saw your other update and then Admin | Audit Logs | Update Data (V2) looks like a good option post import.

———

Alan Marshall

Power Platform Consultant (External)

@.***

Auckland, New Zealand

Email: @.***

---

From: Jenefer Monroe @.> Sent: 19 December 2023 09:41 To: microsoft/coe-starter-kit @.> Cc: Alan Marshall (CW) @.>; Author @.> Subject: Re: [microsoft/coe-starter-kit] [CoE Starter Kit - Feature]: Improve Performance of Audit Log import and record update (Issue #7240)

CAUTION: This email originated from outside of Western Digital. Do not click on links or open attachments unless you recognize the sender and know that the content is safe.

With the work for #7167https://github.com/microsoft/coe-starter-kit/issues/7167, this is an easy add. Targeting for next release as a result.

— Reply to this email directly, view it on GitHubhttps://github.com/microsoft/coe-starter-kit/issues/7240#issuecomment-1861588330, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BCQKDIC5X44H3WET6MPJUETYKCTBFAVCNFSM6AAAAABAKEHZZWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNRRGU4DQMZTGA. You are receiving this because you authored the thread.Message ID: @.***>

[Jenefer-Monroe]

We have an early release of this giant upcoming version of the kit. Please test for us! See this bug for build and how tos: https://github.com/microsoft/coe-starter-kit/issues/7322

[Jenefer-Monroe]

Makes sense on your specific solution. Thanks for sharing.

[CoEStarterKitBot]

@alanmWDC This has been fixed in the latest release. Please install the latest version of the toolkit following the instructions for installing updates. Note that if you do not remove the unmanaged layers as described there you will not receive updates from us.

[ramontebar]

Hey @Jenefer-Monroe, should this documentation now be updated? The new flow "Admin | Audit Logs | Update Data (V2)" is not mentioned in the setup audit log yet.