[CoE Starter Kit - BUG] Environments with FinOps aren't included in inventory, flow Admin | Sync Template v3 (Driver) fails

jukkan commented 1 year ago

Does this bug already exist in our backlog?

[X] I have checked and confirm this is a new bug.

Describe the issue

Does the CoE Starter Kit currently support collecting inventory data where Dynamics 365 Finance & Operations / FinOps / F&O (including variations here for sake of search) is enabled? When doing a clean install on a tenant with the May 2023 Core solution, I see that the new "Admin | Sync Template v3 (Driver)" sync flow (which I guess was just a rename, based on #5585) is failing for 4 environments. These all appear to be D365 FinOps enabeld environments, but they are also used for non-FinOps apps.

The flow fails with an error message like "Entity 'admin_environment' With Id = a9391928-22b0-48c8-948c-f856905f5647 Does Not Exist" for the Get Environment action. Drilling into the flow run details, I see that another error appears in the Get Maker action under the "No" branch ofthe "Check if Legacy Environment" condition:

The flow fails to retrieve data for the MakerID "Microsoft Dynamics Deployment Service", with error status code 502 and a message like "The server did not receive a response from an upstream server. Request tracking id '08585167641064872423744561704CU71'."

So, the intial Get Environments action does manage to retrieve data for the FinOps environments, but then the individual processing of the environments fails with the above errors.

Since I've read somewhere in another issue that you may not have any FinOps available environments at your disposal to test with when developing the CoE Starter Kit, I'm pasting in the full details of one such environment that is returned by the Get Environments action (replaced customer specific name with "XXXX"):

{ "id": "/providers/Microsoft.BusinessAppPlatform/scopes/admin/environments/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX", "type": "Microsoft.BusinessAppPlatform/scopes/environments", "location": "europe", "name": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX", "properties": { "tenantId": "9e2a23af-98f4-4ad5-b266-ef9b2b0b80a5", "azureRegion": "westeurope", "displayName": "XXXX", "createdTime": "2021-03-27T02:39:45.1720997Z", "createdBy": { "id": "Microsoft Dynamics Deployment Service", "displayName": "SYSTEM", "type": "NotSpecified" }, "lastModifiedTime": "2022-02-14T08:37:45.7830204Z", "provisioningState": "Succeeded", "creationType": "Partner", "environmentSku": "Production", "isDefault": false, "capacity": [ { "capacityType": "Database", "actualConsumption": 9873.444, "ratedConsumption": 9873.444, "capacityUnit": "MB", "updatedOn": "2023-05-23T10:31:17Z" }, { "capacityType": "File", "actualConsumption": 10491.3, "ratedConsumption": 10491.3, "capacityUnit": "MB", "updatedOn": "2023-05-23T10:31:17Z" }, { "capacityType": "Log", "actualConsumption": 1585.884, "ratedConsumption": 1585.884, "capacityUnit": "MB", "updatedOn": "2023-05-23T10:31:17Z" }, { "capacityType": "FinOpsDatabase", "actualConsumption": 209829.663, "ratedConsumption": 209829.663, "capacityUnit": "MB", "updatedOn": "2023-05-23T10:31:17Z" }, { "capacityType": "FinOpsFile", "actualConsumption": 40787.05, "ratedConsumption": 40787.05, "capacityUnit": "MB", "updatedOn": "2023-05-23T10:31:17Z" } ], "addons": [], "clientUris": { "admin": "https://admin.powerplatform.microsoft.com/environments/environment/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX/hub", "maker": "https://make.powerapps.com/environments/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX/home" }, "runtimeEndpoints": { "microsoft.BusinessAppPlatform": "https://europe.api.bap.microsoft.com", "microsoft.CommonDataModel": "https://europe.api.cds.microsoft.com", "microsoft.PowerApps": "https://europe.api.powerapps.com", "microsoft.PowerAppsAdvisor": "https://europe.api.advisor.powerapps.com", "microsoft.PowerVirtualAgents": "https://powervamg.eu-il108.gateway.prod.island.powerapps.com", "microsoft.ApiManagement": "https://management.EUROPE.azure-apihub.net", "microsoft.Flow": "https://emea.api.flow.microsoft.com" }, "databaseType": "CommonDataService", "finOpsMetadata": { "type": "Linked", "id": "7035ef06-e1ed-48bc-863a-ef7ff05d513e", "url": "https://XXXX.operations.dynamics.com" }, "linkedEnvironmentMetadata": { "resourceId": "0ea7ae90-1345-474d-99ce-48e7cd14e3e6", "friendlyName": "XXXX", "uniqueName": "unq0ea7ae901345474d99ce48e7cd14e", "domainName": "XXXX", "version": "9.2.23044.00224", "instanceUrl": "https://XXXX.crm4.dynamics.com/", "instanceApiUrl": "https://XXXX.api.crm4.dynamics.com", "baseLanguage": 1033, "instanceState": "Ready", "createdTime": "2022-01-17T08:56:52.517Z", "backgroundOperationsState": "Enabled", "scaleGroup": "EURCRMLIVESG700", "platformSku": "Standard", "schemaType": "Full" }, "trialScenarioType": "None", "notificationMetadata": { "state": "NotSpecified", "branding": "NotSpecific" }, "retentionPeriod": "P28D", "states": { "management": { "id": "Ready" }, "runtime": { "runtimeReasonCode": "NotSpecified", "requestedBy": { "displayName": "SYSTEM", "type": "NotSpecified" }, "id": "Enabled" } }, "updateCadence": { "id": "Frequent" }, "retentionDetails": { "retentionPeriod": "P28D", "backupsAvailableFromDateTime": "2023-04-25T11:23:34.0085174Z" }, "protectionStatus": { "keyManagedBy": "Microsoft" }, "cluster": { "category": "Prod", "number": "108", "uriSuffix": "eu-il108.gateway.prod.island", "geoShortName": "EU", "environment": "Prod" }, "connectedGroups": [], "lifecycleOperationsEnforcement": { "allowedOperations": [ { "type": { "id": "Backup" } }, { "type": { "id": "Convert" } }, { "type": { "id": "Copy" } }, { "type": { "id": "Edit" } }, { "type": { "id": "Restore" } }, { "type": { "id": "UpdateProtectionStatus" } }, { "type": { "id": "NewIdentity" } }, { "type": { "id": "SwapIdentity" } }, { "type": { "id": "RevertIdentity" } }, { "type": { "id": "EnableGovernanceConfiguration" } } ], "disallowedOperations": [ { "type": { "id": "Provision" }, "reason": { "message": "Provision cannot be performed because there is no linked CDS instance or the CDS instance version is not supported." } }, { "type": { "id": "Unlock" }, "reason": { "message": "Unlock cannot be performed because there is no linked CDS instance or the CDS instance version is not supported." } }, { "type": { "id": "Promote" }, "reason": { "message": "Promote cannot be performed on environment of type Production." } }, { "type": { "id": "Reset" }, "reason": { "message": "Reset cannot be performed on environment of type Production." } }, { "type": { "id": "NewCustomerManagedKey" }, "reason": { "message": "NewCustomerManagedKey cannot be performed on environment of type Production." } }, { "type": { "id": "RotateCustomerManagedKey" }, "reason": { "message": "RotateCustomerManagedKey cannot be performed on environment of type Production." } }, { "type": { "id": "RevertToMicrosoftKey" }, "reason": { "message": "RevertToMicrosoftKey cannot be performed on environment of type Production." } }, { "type": { "id": "NewNetworkInjection" }, "reason": { "message": "NewNetworkInjection cannot be performed on environment of type Production." } }, { "type": { "id": "SwapNetworkInjection" }, "reason": { "message": "SwapNetworkInjection cannot be performed on environment of type Production." } }, { "type": { "id": "RevertNetworkInjection" }, "reason": { "message": "RevertNetworkInjection cannot be performed on environment of type Production." } }, { "type": { "id": "Delete" }, "reason": { "message": "Delete cannot be performed on Power Platform environments that are linked to Finance and Operations environments. Please use Lifecycle Services to perform this operation." } }, { "type": { "id": "Recover" }, "reason": { "message": "Recover cannot be performed on Power Platform environments that are linked to Finance and Operations environments. Please use Lifecycle Services to perform this operation." } }, { "type": { "id": "Enable" }, "reason": { "message": "Enable cannot be performed on Power Platform environments that are linked to Finance and Operations environments. Please use Lifecycle Services to perform this operation." } }, { "type": { "id": "Disable" }, "reason": { "message": "Disable cannot be performed on Power Platform environments that are linked to Finance and Operations environments. Please use Lifecycle Services to perform this operation." } }, { "type": { "id": "DisableGovernanceConfiguration" }, "reason": { "message": "DisableGovernanceConfiguration cannot be performed on Power Platform environment because of the governance configuration." } }, { "type": { "id": "UpdateGovernanceConfiguration" }, "reason": { "message": "UpdateGovernanceConfiguration cannot be performed on Power Platform environment because of the governance configuration." } } ] }, "governanceConfiguration": { "protectionLevel": "Basic" } } }

Expected Behavior

No response

What solution are you experiencing the issue with?

Core

What solution version are you using?

4.7

What app or flow are you having the issue with?

Admin | Sync Template v3 (Driver)

What method are you using to get inventory and telemetry?

Cloud flows

Steps To Reproduce

No response

Anything else?

No response

manuelap-msft commented 1 year ago

Hello,

thanks for the (as always!) super detailed analysis, this is so helpful for us!

At the moment, this would be expected behaviour - from the info you're sharing, it looks like the createdBy user of the environment returns Microsoft Dynamics Deployment Service as ID and then our downstream processes (the Helper - maker check flow) can't process this as that's not a valid GUID. We don't have a workaround for this implemented as of yet - we do have a workaround for SYSTEM user but I think those usually come through with an empty ID and not an invalid ID.

Here's the bit of your response value I'm referring to:

"createdBy": { "id": "Microsoft Dynamics Deployment Service", "displayName": "SYSTEM", "type": "NotSpecified" }

Although we've not tested this ourselves, I think the rest should work but as it's failing at the Get Maker step it then doesn't follow any other steps to insert the environment.

To get this working we would need to:

Update the Helper - Maker Check flow to add an exception branch for when the ID = Microsoft Dynamics Deployment Service
Generate an actual GUID for Microsoft Dynamics Deployment Service and insert them into the Maker table
Every time Helper - Maker Check gets called with ID = Microsoft Dynamics Deployment Service, check if this user already exists and make sure we don't insert it again

OR

Insert Microsoft Dynamics Deployment Service into Maker table during Setup Wizard (that's what we do for the SYSTEM user)
then the logic in the Helper - Maker Check is easier as we would just be able to expect that that user exists

I will add this on the backlog for us to investigate, but can't commit to an ETA at the moment.

Jenefer-Monroe commented 1 year ago

To investigate with the other finops bugs. Might want to just exclude them.

jukkan commented 1 year ago

These environments with FinOps are normal Dataverse environments that can include a wealth of other apps and flows. In this particular case, the customer's primary production environment happens to include FinOps. It also includes 17 apps and 371 flows. Leaving them out from the inventory will distort tenant level statistics of Power Platform resources deployed quite significantly.

Considering that on the product side of the house there's a big push towards the "One Dynamics One Platform" (ODOP) vision (see this session for a summary), customers might rightfully question how serious Microsoft is about the platform story in practice.

"One Dynamics One Platform (ODOP) focuses on the convergence of Finance and Operations apps with Microsoft Power Platform, enabling every customer to take full advantage of the Power Platform. These features improve the development, administration, user experiences, and insights by removing barriers, tightening integrations, and enhancing cross-platform capabilities."

We of course know product vision does not always turn into tech reality very rapidly. Just leaving this here as an example to potentially use in discussions around getting the necessary resources and product team support to get closer to that vision.

manuelap-msft commented 1 year ago

Appreciate the feedback. Like any other team, we have to prioritize our backlog against the resources we have available and invest in feedback items that will have the most customer impact or that we know impact a lot of customers - so far, support for FinOps environments has only come up twice (and the other person that raised it was an MS internal demo environment and not a customer case). Currently, we have higher priority items to work on but we have added this to our backlog for investigation.

If this is critical for you or customers you work with, you may have to extend the CoE kit yourself in the meantime to handle those environments - after all, what we provide with this solution is a starter kit.

jax-coe-admin commented 1 year ago

My environment is also encountering this issue with environments deployed by Microsoft Dynamics Deployment Service.

I guess that makes us the third to report it.

jukkan commented 1 year ago

When Dynamics 365 components get left out, time to resort to good ol' Dynamics 365 tools to get the job done! As a workaround, I exported the Environments list from Power Platform Admin View into Excel. Then I checked what data the successfully inventoried environments had and copied the deatils for FinOps environments from the "Get Environments" JSON back into that Excel as new rows. Remove all rows for existing inventoried environments, navigate to the classic Dynamics 365 web client UI to start the data import job, watch new environments get added to CoE Starter Kit inventory database and see the sync flows get kicked off. After a moment, it looks like the essential data is at least captured in the Power BI report.

Why not just add the rows from the modern Dataverse row editing experience in Power Apps Maker portal? Well, I used that first but then learned that sync flows didn't appear for the environments. It turns out that whereas the helper flow "SYNC HELPER - Apps" uses the environment name field, the equivalent "SUNC HELPER - Cloud Flows" flow refererenes the admin_environmentid column, i.e. record GUID. Still today it is not possible to manually enter the GUID for new records in the modern Maker GUIs (you can do it in flows, like the CoE Starer Kit does). So, I had to use the old Excel import trick to feed the identical values into bothe Environment Name and Environment (GUID, primary key) columns. See this blog post for detailed steps on the Excel process.

Will this cause issues in the future? Possibly, since updating the inventory of those environments may need to be manually triggered if the "Admin | Sync Template v3 (Driver)" flow always fails before getting to update the environment, due to its inability to handle the "Microsoft Dynamics Deployment Service" Maker ID value. Still, instead of going an manipulating the main Driver flow and missing out on all future bugfixes/enhancements, this felt like a better option to get the inventory data to be complete for analysis purposes.

Jenefer-Monroe commented 1 year ago

Thanks for writing this up for folks while we figure out what to do on our side!

annsarP commented 1 year ago

Hello! Same issue here. 2 days ago, just before the weekend our Admin | Sync Template v3 (Driver) flow has been suspended due to fails.

I'll work on custom error handling in that case. Crossing fingers for a solution from the team. Thank you for your work.

manuelap-msft commented 1 year ago

We are investigating this right now, and tentatively exploring a fix for July. We have some complexities in terms of SYSTEM users (historical tech debt) that makes this more challenging then it may seem on the surface, but we've done a bit of brainstorming and think we have a way forward.

We don't have F&O in our development tenant right now, so @jukkan if okay with you, once we've explored our fix we may ask you to test this on your side to make sure it works before we ship it as we're developing a bit blind right now.

jukkan commented 1 year ago

Sure, we'll be glad to test the fix with F&O environments!

Jenefer-Monroe commented 1 year ago

For the sync flow architecture. the fix is as shown below. Basic idea is to keep this change as small as possible. So we will not add a new system user but instead just attribute these to the same System user. Manuela is working on the DAta Export architecture change seperately. Thank you for helping us test!

Change:

In Admin | Sync Template v3 (Driver) Store environments in the CoE CDS Entity > Apply to each Environment > Check if Legacy Environment > (no) > Get Maker Replace MakerID input with @{if(equals(items('Apply_to_each_Environment')?['properties/createdBy/displayName'], 'SYSTEM'), 'SYSTEM', items('Apply_to_each_Environment')?['properties/createdBy/id'])}

Question:

What I dont know is if more objects than Environments get attributed to this user. Obviously this is only a fix for Environments. Please do let me know.

jukkan commented 1 year ago

I tested changing the MakerID field value accordingly and now the flow run went through successfully for the first time. Upon initial inspection, I didn't notice any data missing from the FinOps environments.

Jenefer-Monroe commented 1 year ago

Great news! Do you know, is it just environments that get allocated to this system user?

jax-coe-admin commented 1 year ago

I applied the change as well and will report back after the flow has completed running

jukkan commented 1 year ago

Great news! Do you know, is it just environments that get allocated to this system user?

So far I haven't seen anything else. I don't think the provisioning process for FinOps Dataverse environment creates any other components that would need to be within the CoE Starter Kit inventory scope today. It would of course be great if the folks behind FinOps + Dataverse platform convergence features could verify this...

jax-coe-admin commented 1 year ago

The flow completed successfully with the changes. To the best of my knowledge, it appears to have inventoried the D365 environments.

Jenefer-Monroe commented 1 year ago

Wonderful, thank you so much for helping to test!

annsarP commented 1 year ago

Hello! I replaced Maker ID with input provided and now it replaces previous string with 'SYSTEM ' (and it's still failing :/)

Have I done something wrong?

Jenefer-Monroe commented 1 year ago

looks like you might have a space after SYSTEM for id?

annsarP commented 1 year ago

EDIT: Ah yes, my bad... space was added after expression... Fixed. :)

Jenefer-Monroe commented 1 year ago

Thank goodness, I was worried for a minute :)

CoEStarterKitBot commented 1 year ago

@jukkan This has been fixed in the latest release. Please install the latest version of the toolkit following the instructions for installing updates. Note that if you do not remove the unmanaged layers as described there you will not receive updates from us.

CoEStarterKitBot commented 1 year ago

@jukkan This has been fixed in the latest release. Please install the latest version of the toolkit following the instructions for installing updates. Note that if you do not remove the unmanaged layers as described there you will not receive updates from us.

microsoft / coe-starter-kit