[CoE Starter Kit - BUG] Audit Logs - ContentID returned not being recognized

[NicoleSSmith]

Does this bug already exist in our backlog?

• [X] I have checked and confirm this is a new bug.

Describe the issue

Admin | Audit Logs | Sync Audit Logs (V2) is failing. When I navigate to the Flow Instance URL, getting the follow error:

Expected Behavior

No response

What solution are you experiencing the issue with?

Core

What solution version are you using?

4.11

What app or flow are you having the issue with?

Admin | Audit Logs | Sync Audit Logs (V2

What method are you using to get inventory and telemetry?

None

Steps To Reproduce

No response

Anything else?

No response

AB#558

[Jenefer-Monroe]

Can you please share a screenshot of your App Registrations API Permissions?

[NicoleSSmith]

See API Permissions here:

[Jenefer-Monroe]

Can you delete that Delegated permission? I think that may conflict.

[NicoleSSmith]

I will remove the delegated permission and will re-establish.

Has there been a change to the licensing and role requirements for the HTTP call? Reding through Collect audit logs using an HTTP action - Power Platform | Microsoft Learnhttps://learn.microsoft.com/en-us/power-platform/guidance/coe/setup-auditlog-http#update-environment-variables it has the following steps: Before you set up the audit log flows

1. Microsoft 365 audit log search must be turned on for the audit log connector to work. More information: Turn audit log search on or offhttps://learn.microsoft.com/en-us/microsoft-365/compliance/turn-audit-log-search-on-or-off?preserve-view=true&view=o365-worldwide
2. The user identity running the flow must have permission to the audit logs. Minimum permissions for this is described in Before you search the audit logshttps://learn.microsoft.com/en-us/microsoft-365/compliance/audit-log-search?preserve-view=true&view=o365-worldwide#before-you-search-the-audit-log.
3. Your tenant must have a subscription that supports unified audit logging. More information: Security & Compliance Center availability for business and enterprise planshttps://learn.microsoft.com/en-us/office365/servicedescriptions/office-365-platform-service-description/office-365-securitycompliance-center
4. A global admin is required to configure the Azure AD app registration.

The Office 365 Management APIs use Azure Active Directory (Azure AD) to provide authentication services that you can use to grant rights for your application to access them.

When you get into the sep 3 link, instructions state that you need one of the following licenses:

Enterprise Mobility + Security E5/A5, Microsoft 365 E5/A5, Microsoft 365 E5/A5/F5 Security and F5 Security & Compliance, and Microsoft Entra ID Premium Plan 2 provide the rights for a user to benefit from Microsoft Entra ID Governance.

I just want to make sure that nothing is needed before I proceed to re-establish the API permissions.

---

From: Jenefer Monroe @.> Sent: Tuesday, August 8, 2023 7:37 AM To: microsoft/coe-starter-kit @.> Cc: Nicole Smith @.>; Author @.> Subject: [EXTERNAL] Re: [microsoft/coe-starter-kit] [CoE Starter Kit - BUG] Action 'Get_Azure_Secret' failed (Issue #6248)

Can you delete that Delegated permission? I think that may conflict.

— Reply to this email directly, view it on GitHubhttps://github.com/microsoft/coe-starter-kit/issues/6248#issuecomment-1669750445, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A5DWRO6365GM75Z655Y27FLXUJFMHANCNFSM6AAAAAA3EYOE2I. You are receiving this because you authored the thread.Message ID: @.***>

[Jenefer-Monroe]

No there is no license change associated with the change for http

[NicoleSSmith]

I completely deleted the API permissions and set up a new permission with the type being application:

I wasn't sure if I needed to, but I created a new client secret under the Microsft 365 Management Application registration. I placed the client secret value into the enviromental variables in CoE Admin Command center App.

I canceled all running instances of Admin | Audit Logs | Sync Audit Logs (V2) and turned it off, then back on. I reran the flow, which took several hours.

The flow has again, failed. See the error message below.

Please advise.

[Jenefer-Monroe]

Open the last attempted run for Admin | Audit Logs | Office 365 Management API Subscription Is the app id correctly ingested here? Is the secret correctly ingested here? Does this call fail hence correctly marking the variable as false?

[NicoleSSmith]

These are screenshots from my most recently attempted, but failed run. Shot one matches:

Second part successfully ran:

And the third also looks the same:

[Jenefer-Monroe]

Are you a commercial tenant?

[NicoleSSmith]

No. We are an enterprise tenant.

[Jenefer-Monroe]

Sorry that meant commercial vs a government tenant (like GCC / DOD / etc) Sounds like you are.

I'm a little confused on which Audit Log system you are using from looking at the screenshots above.

There are two solutions at play, the old deprecated one with a custom connector and a new one that uses HTTP. Please take a step back and look at these two, and decide which path you will follow. It's highly recommended that you use HTTP in qhich case you can delete the audit logs solution and no longer have the flow you originally posted about

Collect audit logs using an HTTP action Collect audit logs using a custom connector (deprecated)

[NicoleSSmith]

We were using a custom connector, but I previously followed the https://learn.microsoft.com/en-us/power-platform/guidance/coe/setup-auditlog-http instruction set to set up the HTTP path. In this step: https://learn.microsoft.com/en-us/power-platform/guidance/coe/setup-auditlog-http You turn on the Admin | Audit Logs | Office 365 Management API Subscription flow on and run. I have opened the flow and verified that the action to start the subscription has passed. I see a value of 400, which means the subscription has been successfully enabled in the past. The step after that is to turn on flows. This is the flow that is failing every hour and this is the last step in the instructions.

I no longer have the audit log solution installed: I am accessing the flow in the Center of Excellence - Core Components.

[Jenefer-Monroe]

oh i see. ok first thing, please run the following flow. If you use the setup wizard for your upgrades this will be done for you: Admin | Sync Template v3 (Call Updates)

Once that is done, return and we can see what happened further above as well.

[NicoleSSmith]

That flow has successfully ran:

[Jenefer-Monroe]

wonderful now reurn the failure from the audit logs. if it fails it should be in a different place and if not we will investigate

[NicoleSSmith]

I reran the failed flow. It gave me the same error in the same spot. We are using the Client secret and not the Client Azure Secret

[Jenefer-Monroe]

OK we had a bug there. If you paste the following in the place shown @{first(outputs('Get_ID_Fail_2')?['body/value'])?['admin_coesolutionmetadataid']}

Then it will fail here instead which is the expected behavior and then we will need to explore above what caused the failure.

[Jenefer-Monroe]

Once you local fix the above See how many loops you go through there

Walk through the loops, you'll be looking for one that goes down the true path here to increment this variable.

[NicoleSSmith]

I edited the flow in the Update Last Run Fail 2 Then I ran the flow and got the matching error as seen here: The BuildVontentSlotArray step only went through one loop and I get the matching increment variable condition is not satisfied message. How should I proceed?

[NicoleSSmith]

And editing the flow, created an unmanaged layer. Will this be a problem on the next release?

[NicoleSSmith]

Any updates on next steps?

[Jenefer-Monroe]

How about this loop? Do you increment it in these?

[NicoleSSmith]

Yes. This is what that step looks like

[Jenefer-Monroe]

Thank you, adding the author of the flow here, hopefully they can help us understand if there is a change we can make.

[pete-msft]

@NicoleSSmith Could you please share the output of the GetContentDetails step. It seems that all HTTP calls of the first content slot (1/44) fails, so would like to see what is the error in that step. Can you also check the status of few other GetAndProcessEvents loop steps to see are there more these same errors in that loop

[NicoleSSmith]

@pete-msft - This is the output of the GetContentDetails step { "error": { "code": "AF20052", "message": "Blob key 20230821140744842016388$20230821140833780017133$audit_general$Audit_General$na0020 in the url is invalid" } It appears as if all of the the GetContentDetails events failed. I scrolled through the first 20 loops and they all have the same error with code AF20052, but with different combinations of Blob keys ending with "$audit_general$Audit_General$na0020 in the url is invalid"

[pete-msft]

@NicoleSSmith @Jenefer-Monroe Blob key (contentId) seems to be in correct format at least. So, is this GCC/DoD tenant? If yes then please double check that "Audit Logs - Audience" and "Audit Logs - Authority" URLs are correct https://learn.microsoft.com/en-us/power-platform/guidance/coe/setup-auditlog-http#update-environment-variables

Just tested in my tenant and here is one contentId which worked compared to one you sent 20230828171604712068871$20230828173205648032977$audit_general$Audit_General$emea9037 20230821140744842016388$20230821140833780017133$audit_general$Audit_General$na0020

Here is similar problem in GCC thought root cause of that error is not solved https://github.com/microsoft/coe-starter-kit/issues/4890#issuecomment-1468092899

[NicoleSSmith]

We are not a GCC/DoD tenant. We are an enterprise tenant. The URLs match the "commercial" tenant URLs.

[pete-msft]

@NicoleSSmith Ok so it is Commercial tenant. Can you still check few contentIds returned by the ListAuditLogContent step are they same as in ParseContentIDs step

-

[NicoleSSmith]

The first few seem to match

[NicoleSSmith]

I mixed my screenshots up, but confirming the second one shows the matching IDs

[pete-msft]

@NicoleSSmith Thanks! Did you compare the whole string that it is exactly the same like the below one? There are not spaces in the end or any other differences?

20230821140744842016388$20230821140833780017133$audit_general$Audit_General$na0020

[NicoleSSmith]

@pete-msft They look exactly the same. Here are the first three complete data fields from the ListAuditLogContent body:

[ { "contentUri": "https://manage.office.com/api/v1.0/bbfb3f2b-353c-4e04-863a-76285f90f906/activity/feed/audit/20230821140744842016388$20230821140833780017133$audit_general$Audit_General$na0020", "contentId": "20230821140744842016388$20230821140833780017133$audit_general$Audit_General$na0020", "contentType": "Audit.General", "contentCreated": "2023-08-21T14:08:33.780Z", "contentExpiration": "2023-08-28T14:07:44.842Z" }, { "contentUri": "https://manage.office.com/api/v1.0/bbfb3f2b-353c-4e04-863a-76285f90f906/activity/feed/audit/20230821140833864077953$20230821140950387057192$audit_general$Audit_General$na0020", "contentId": "20230821140833864077953$20230821140950387057192$audit_general$Audit_General$na0020", "contentType": "Audit.General", "contentCreated": "2023-08-21T14:09:50.387Z", "contentExpiration": "2023-08-28T14:07:44.842Z" }, { "contentUri": "https://manage.office.com/api/v1.0/bbfb3f2b-353c-4e04-863a-76285f90f906/activity/feed/audit/20230821140950559057198$20230821141123536076966$audit_general$Audit_General$na0020", "contentId": "20230821140950559057198$20230821141123536076966$audit_general$Audit_General$na0020", "contentType": "Audit.General", "contentCreated": "2023-08-21T14:11:23.536Z", "contentExpiration": "2023-08-28T14:07:44.842Z" },

This is the first three strings from the ParseContentIDs. All strings match: [ { "ContentID": "20230821140744842016388$20230821140833780017133$audit_general$Audit_General$na0020" }, { "ContentID": "20230821140833864077953$20230821140950387057192$audit_general$Audit_General$na0020" }, { "ContentID": "20230821140950559057198$20230821141123536076966$audit_general$Audit_General$na0020" },

[pete-msft]

@NicoleSSmith Thanks again for providing more info! Just realized thought that re-running over 7 days old instance should not even work anymore as Office 365 Management API has a limitation to return only last 7 days back of events. I would still run the flow manually to get latest events. If flow is not enabled currently then just turn on the flow and run ones manually.

[NicoleSSmith]

@pete-msft My flow has been continually failing for over a month. I have a failed run from about 90 minutes ago as well that I can pull information from. In the most recent failure, the content IDs match as well. I get the same error for the failure The GetContentDetails has the same blob key error { "error": { "code": "AF20052", "message": "Blob key 20230829140953101088348$20230829141138392012843$audit_general$Audit_General$na0020 in the url is invalid" } } I have tried turning the flow off and back on. I have also tried canceling the running flow and turning off and back on. I have resubmitted failed runs. None of those steps have worked. It stills try to pick up the azure secret, which we do not use.

[pete-msft]

@NicoleSSmith That "error" in the run details can be ignored as it is showing first exception which happens in the flow run and that is handled exception. If Azure secret is not defined the flow will then continue because of "configure run after" configuration. But this AF20052 "Blob key..." error is strange if the contentId is correct which I haven't seen before.

@Jenefer-Monroe what do you think we should do next with this? Could we involve PG owning the Office 365 Management API?

[Jenefer-Monroe]

I think we would be better off creating a repro outside the context of the kit and having OP create a support ticket. I'm not sure the easiest way to create such a repro but basically I think it would just be like get all the ContentIDs for some time period and then loop them to get each. I suppose the bug woudl repro there if I understand it.

[pete-msft]

Ok, we can probably create a PowerShell script to do the same. If you think this would be ok way to test outside of the kit then I can create a script tomorrow

[Jenefer-Monroe]

Thanks Pete! I'll go ahead and clean up the above so we can keep this as a reference in case others see this issue.

[NicoleSSmith]

Do you need any additional information from me at this time?

---

From: Jenefer Monroe @.> Sent: Wednesday, August 30, 2023 5:45 AM To: microsoft/coe-starter-kit @.> Cc: Nicole Smith @.>; Mention @.> Subject: [EXTERNAL] Re: [microsoft/coe-starter-kit] [CoE Starter Kit - BUG] Action 'Get_Azure_Secret' failed (Issue #6248)

Thanks Pete! I'll go ahead and clean up the above so we can keep this as a reference in case others see this issue.

— Reply to this email directly, view it on GitHubhttps://github.com/microsoft/coe-starter-kit/issues/6248#issuecomment-1699102316, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A5DWRO22ETH5JYXAYVJ64ETXX4YXBANCNFSM6AAAAAA3EYOE2I. You are receiving this because you were mentioned.Message ID: @.***>

[pete-msft]

@NicoleSSmith If you want to test and verify does the same happen outside of CoE then here is PowerShell script attached which does exactly similar calls to O365 Management API Test-AuditLogAPI.ps1.txt Just remove '.txt' and probably you need to unblock the file

Then define $TenantID, $ClientID and $ClientSecret values of your service principal in the beginning of the script. It will take 1h hour back of Audit.General event and outputs the count of events found if the HTTP calls works :)

[NicoleSSmith]

@pete-msft When this PowerShell script was ran, we had 7483 events. It showed Planner, Teams, Yammer, CRM, etc events.

Audit.General events found - 7483

[petepuu]

@NicoleSSmith Sorry about the delay in my response. Audit.General content type contains events of other services as well and sync flow just filters the events which are used in CoE Kit. As the similar HTTP calls works in PowerShell there must be now something wrong in the audit log sync flow which breaks the contentId value in your case. I have checked the values you have sent many times but do not see any issues with those. I will do some testing in my environment but probably next we need to take one contentID (blob key) not working and copy-paste it to the PowerShell script to see does that work

[NicoleSSmith]

No worries! Let me know next steps once you get your testing done. I am eager to get the flow up and running.

[petepuu]

@NicoleSSmith Here is the script to test the failed HTTP calls having that "Blob key..." error.

Copy ContentID value of ParseContentIDs action to the $contentId parameter of the script. In script the param value need to be inside single quotes. Also provide $TenantID, $clientID and $clientSecret. Would be good to test with few ContentIDs which failed. Remember that those need to be max 7 days old content slots of events

$TenantID = ''
$clientID = ''
$clientSecret = ''
$contentId = ''

$ReqTokenBody = @{
    grant_type = "client_credentials"
    client_id = $clientID
    client_secret = $clientSecret
    scope = "https://manage.office.com/.default"
    }

$authUri = "https://login.microsoftonline.com/$TenantID/oauth2/v2.0/token"

$TokenResponse = Invoke-RestMethod -Uri $authUri -ContentType "application/x-www-form-urlencoded" -Method POST -Body $ReqTokenBody

$authheader = @{
"Authorization" = "Bearer $($Tokenresponse.access_token)"
"Content-type" = "application/json"
}

$uri = "https://manage.office.com/api/v1.0/$TenantID/activity/feed/audit/$($contentId)"

Invoke-RestMethod -Uri $uri -Method Get -Headers $authheader 

Params should be like this

[petepuu]

If previous test works and returns the events, then next step would be to edit the flow and add Compose action to see the actual whole URL used in the call. Add Compose action after the ResetHttpCallFailed2 action and copy-paste the URI value of the GetContentDetails action to Compose value. Then save and test the flow and see what is the URI value of the Compose action

You can then even test that URI in the PowerShell script like below (need to be in single quotes)

[Jenefer-Monroe]

We have another repro https://github.com/microsoft/coe-starter-kit/issues/6527. Pointing them here to hopefully assist with a repro

[mrmonto1]

Hi. I updated to the Sept. release and it looks like it's still requiring the Azure Client Secret. Is there a way to bypass this?

[Jenefer-Monroe]

Please note that, per your previous bug, that failures like is a caught/expected failure. We use this call to see if you are storing the secret as plan text or azure. In your case the call would fail and set the next variable to false as a result and then continue. And I believe fail at the same spot as the user above.

[mrmonto1]

Thanks @Jenefer-Monroe! Per your advice, I dug a little deeper into the flow and found following error below. Restarting the subscription to Audit Log Content with the Admin | Audit Logs | Office 365 Management API Subscription flow and then re-running Admin | Audit Logs | Sync Audit Logs (V2) seems to have fixed it and audit data is now being populated into the Audit Log table. I know I did this when I installed the COE in July. I guess I must have missed this step after upgrading the COE. I'll remember to do this next time I upgrade. I suggest updating the COE update instructions to include this because the current instructions mention to update the deprecated custom connector for Audit Logs. https://learn.microsoft.com/en-us/power-platform/guidance/coe/after-setup#updating-the-audit-log-solution