search

microsoft / coe-starter-kit

Other
751 stars 221 forks source link

Data Export : Default Environment keeps getting marked as deleted #7562

Closed FacilyDev closed 8 months ago

FacilyDev commented 9 months ago

Does this bug already exist in our backlog?

Describe the issue

For several days I'm having an issue that our Default environment is marked as deleted. I tried few times to manually uncheck [Deleted] and remove [Date of deletion] in Environments table but the next day it again gets marked as deleted with new date. As my source is data lake I checked environments.json but this environment is there and is Active. All dataflows refresh correctly.

I'm not sure if it has something to do with the fact that EnvId field doesn't have default- in the beginning (EnvName field has it) and it 's not possible to change it manually.

Expected Behavior

Default environment must be visible with all the Flows / Apps as it is the most used environment in our tenant.

What solution are you experiencing the issue with?

Core

What solution version are you using?

4.22

What app or flow are you having the issue with?

Probably it is this one: [CLEANUP - Admin | Sync Template v4 (Check Deleted)]

What method are you using to get inventory and telemetry?

Data Export

Steps To Reproduce

No response

Anything else?

No response

AB#2165

Jenefer-Monroe commented 9 months ago

I have just confirmed that deleted works correctly in that flow. We do compare "Name" which includes the default.

Please go to the most recent run of CLEANUP - Admin | Sync Template v4 (Check Deleted) Is the default envt returned in this call? image

Jenefer-Monroe commented 9 months ago

And if its not, can you please go see if the identity running the flow is shown as a sys admin for the default envt?

FacilyDev commented 9 months ago

Yes, default is returned in this step, however with capital "D" -- name starts with Default-

Jenefer-Monroe commented 9 months ago

Oh I see, thats the bug. The dataflow should not be writing it like that. Let me take a look at how easy this is to fix.

FacilyDev commented 9 months ago

Hang on! But you don't use dataflow but use action directly from the Platform (which I find strange to be honest). image

Jenefer-Monroe commented 9 months ago

Thats just for the delete case. When we insert these environments its from the datalake, which unfortunately calls these ents with an lower case name. (default-abc123...) unlike the rest of the product which returns them with upper case (Default-abc123...)

The datalake doesnt give us the needed information for checking for delete.

Jenefer-Monroe commented 9 months ago

How to local fix:

Go to your Dataflows and edit the Environment Dataflow image

Select Environments > Advanced Editor image

Replace what is there with the below:

let Source = AzureStorage.DataLake(DatalakeURL & "/powerapps/environments"), excludehiddenfiles = Table.SelectRows(Source, each [Attributes]?[Hidden]? <> true), excludezerolengthfiles = Table.SelectRows(excludehiddenfiles, each [Attributes]?[Size]? > 0),

"Get content from json files" = Table.AddColumn(excludezerolengthfiles, "Transform file", each #"Transform file (4)"([Content])),

"Remove unneeded columns" = Table.SelectColumns(#"Get content from json files", {"Name", "Transform file"}),

"Expanded columns" = Table.ExpandTableColumn(#"Remove unneeded columns", "Transform file", {"environmentId", "environmentName", "isDefault", "environmentState", "environmentUrl", "environmentType", "securityGroup", "purpose", "cdsInstanceId", "cdsInstanceUrl", "createdTime", "createdPrincipalId", "lastModifiedTime", "lastModifiedPrincipalId", "deletedTime", "environmentRegion", "tenantGuid", "governanceConfigurationType", "isBackgroundJobEnabled", "provisioningState", "retentionPeriod"}, {"environmentId", "environmentName", "isDefault", "environmentState", "environmentUrl", "environmentType", "securityGroup", "purpose", "cdsMetadataName", "cdsInstanceUrl", "createdOn", "createdBy", "lastModifiedOn", "lastModifiedBy", "deletedOn", "environmentRegion", "tenantId", "governanceConfigurationType", "isBackgroundJobEnabled", "provisioningState", "retentionPeriod"}),

"Filtered rows" = Table.SelectRows(#"Expanded columns", each not Text.StartsWith([environmentId], "legacy-") or not Text.StartsWith([environmentId], "Legacy-")),

"Replace errors" = Table.ReplaceErrorValues(#"Filtered rows", {{"environmentId", null}, {"environmentName", null}, {"isDefault", null}, {"environmentState", null}, {"environmentUrl", null}, {"environmentType", null}, {"securityGroup", null}, {"purpose", null}, {"cdsMetadataName", null}, {"cdsInstanceUrl", null}, {"createdOn", null}, {"createdBy", null}, {"lastModifiedOn", null}, {"lastModifiedBy", null}, {"deletedOn", null}, {"environmentRegion", null}, {"tenantId", null}}),

"Replaced value" = Table.ReplaceValue(#"Replace errors", "Microsoft Dynamics Deployment Service", "SYSTEM", Replacer.ReplaceValue, {"createdBy"}),

// Strip out Legacy-/Default- from Environment ID

"Add Environment GUID" = Table.TransformColumnTypes(Table.AddColumn(#"Replaced value", "environmentGUID", each if Text.Length([environmentId]) = 36 then Text.Range([environmentId], 0,36) else Text.Range(Text.AfterDelimiter([environmentId], "-", 0), 0, 36)), {{"environmentGUID", type text}}),

"Add isNotDeleted" = Table.TransformColumnTypes(Table.AddColumn(#"Add Environment GUID", "isNotDeleted", each false), {{"isNotDeleted", type logical}}),

// Add a column with timestamp

"Add QueriedOn" = Table.AddColumn(#"Add isNotDeleted", "QueriedOn", each DateTimeZone.UtcNow()),

"Merged with makers" = Table.NestedJoin(#"Add QueriedOn", {"createdBy"}, admin_makers, {"admin_makerid"}, "admin_makers", JoinKind.LeftOuter),

"Expanded maker columns" = Table.ExpandTableColumn(#"Merged with makers", "admin_makers", {"admin_displayname", "admin_userprincipalname", "admin_makerid", "admin_makerisorphaned"}, {"admin_displayname", "admin_userprincipalname", "admin_makerid", "admin_makerisorphaned"}),

"Lowercased environment id" = Table.TransformColumns(#"Expanded maker columns", {{"environmentId", each Text.Lower(_), type nullable text}}),

// true/false if the environment has CDS

"Add hasCDS column" = Table.AddColumn(#"Lowercased environment id", "hasCDS", each if [cdsInstanceUrl] = "-" then "false" else "true", type text),

// If environment created by SYSTEM

"Replaced SYSTEM maker" = Table.ReplaceValue(#"Add hasCDS column", null, "SYSTEM", Replacer.ReplaceValue, {"admin_displayname"}),

"Replace IsOrphan flag for SYSTEM environments" = Table.ReplaceValue(#"Replaced SYSTEM maker", null, false, Replacer.ReplaceValue, {"admin_makerisorphaned"}),

Custom = Table.ReplaceValue(#"Replace IsOrphan flag for SYSTEM environments", "default-", "Default-", Replacer.ReplaceText, {"environmentId"}) ,

"Inserted conditional column" = Table.AddColumn(Custom, "IsOrphan", each if [admin_makerisorphaned] = true then "Yes" else "No"),

"Add createdBy ID column" = Table.AddColumn(#"Inserted conditional column", "combinedCreatedBy", each [admin_makerid], type text),

// Get additional meta data for environments from the ListEnvironmentsAPI

"Merged with ListEnvironmentsAPI" = Table.NestedJoin(#"Add createdBy ID column", {"environmentId"}, ListEnvironmentsAPI, {"value.name"}, "ListEnvironmentsAPI", JoinKind.LeftOuter),

"Replaced SYSTEM user" = Table.ReplaceValue(#"Merged with ListEnvironmentsAPI", "SYSTEM", null, Replacer.ReplaceValue, {"createdBy"}),

"Expand ListEnvironmentsAPI columns" = Table.ExpandTableColumn(#"Replaced SYSTEM user", "ListEnvironmentsAPI", {"value.properties.governanceConfiguration.protectionLevel", "CrmInstanceMetadataName", "value.properties.linkedEnvironmentMetadata.resourceId", "IsAdminMode", "BackgroundOpsEnabled", "value.properties.states.management.id"}, {"value.properties.governanceConfiguration.protectionLevel", "CrmInstanceMetadataName", "OrganizationID", "IsAdminMode", "BackgroundOpsEnabled", "EnvironmentStateManagementID"}),

"Inserted conditional column 1" = Table.AddColumn(#"Expand ListEnvironmentsAPI columns", "markDeleted", each if [environmentId] <> null then false else true),

"Inserted conditional column 2" = Table.AddColumn(#"Inserted conditional column 1", "markDeletedDate", each if [markDeleted] = false then null else [QueriedOn]),

"Changed column type" = Table.TransformColumnTypes(#"Inserted conditional column 2", {{"Name", type text}, {"environmentId", type text}, {"environmentName", type text}, {"isDefault", type logical}, {"environmentState", type text}, {"environmentUrl", type text}, {"environmentType", type text}, {"securityGroup", type text}, {"purpose", type any}, {"cdsMetadataName", type text}, {"cdsInstanceUrl", type text}, {"createdOn", type datetime}, {"createdBy", type text}, {"lastModifiedOn", type text}, {"lastModifiedBy", type text}, {"deletedOn", type text}, {"environmentRegion", type text}, {"tenantId", type text}, {"environmentGUID", type text}, {"QueriedOn", type datetimezone}, {"admin_displayname", type text}, {"admin_userprincipalname", type text}, {"admin_makerid", type text}, {"admin_makerisorphaned", type logical}, {"hasCDS", type logical}, {"IsOrphan", type text}, {"combinedCreatedBy", type text}, {"value.properties.governanceConfiguration.protectionLevel", type text}, {"CrmInstanceMetadataName", type text}, {"isNotDeleted", type logical}, {"IsAdminMode", type logical}, {"BackgroundOpsEnabled", type logical}, {"EnvironmentStateManagementID", type text}, {"markDeleted", type logical}, {"markDeletedDate", type datetime}})

in

"Changed column type"

FacilyDev commented 9 months ago

Except the step above should I also remove [Deleted] mark in Environments table and remove DataflowRefreshHistory records as described in #7511 or just definition of Environments dataflow will rectify the situation?

FacilyDev commented 9 months ago

I can confirm it worked only after editing dataflow. Hopefully, the flow "check deleted" will not mess up tomorrow 🙂 Thanks for lightning-fast help (very unusual by MS support and without fiddler log required 😉)

Jenefer-Monroe commented 9 months ago

You were the 3rd person to raise this so you were a virutal fiddler log with your observations hitting me just right :) Happy to help, thank you so much for validating!!

niwrejim21 commented 9 months ago

@Jenefer-Monroe, after performing the above fix, our PowerBI dashboard is now showing less number of PowerApps. Admin View is showing 10k+ items compared to the PowerBI report - it is now only showing 2k+ items. Do you have any idea why this is the case?

Checking the number, it seems that it is not getting the number of PowerApps in the default environment and it is not showing in PowerBI Report.

bartsneary commented 9 months ago

@Jenefer-Monroe, when I noticed the same was happening in my environment and I replaced the query in Advanced Editor in my Environments Dataflow, I now get:

---------- Message ---------- Expression.Error: A cyclic reference was encountered during evaluation.

---------- Session ID ---------- 8b3f1c1a-4cdb-44bb-893c-91ccfbc8dbfc

---------- Request ID ---------- eb10cc1a-9cfa-4f74-ae5f-fa6528aec888

---------- Mashup script ---------- section Section1; shared #"Transform file (4)" = let Source = AzureStorage.DataLake(DatalakeURL & "/powerapps/environments"), excludehiddenfiles = Table.SelectRows(Source, each [Attributes]?[Hidden]? <> true), excludezerolengthfiles = Table.SelectRows(excludehiddenfiles, each [Attributes]?[Size]? > 0),

"Get content from json files" = Table.AddColumn(excludezerolengthfiles, "Transform file", each #"Transform file (4)"([Content])),

"Remove unneeded columns" = Table.SelectColumns(#"Get content from json files", {"Name", "Transform file"}),

"Expanded columns" = Table.ExpandTableColumn(#"Remove unneeded columns", "Transform file", {"environmentId", "environmentName", "isDefault", "environmentState", "environmentUrl", "environmentType", "securityGroup", "purpose", "cdsInstanceId", "cdsInstanceUrl", "createdTime", "createdPrincipalId", "lastModifiedTime", "lastModifiedPrincipalId", "deletedTime", "environmentRegion", "tenantGuid", "governanceConfigurationType", "isBackgroundJobEnabled", "provisioningState", "retentionPeriod"}, {"environmentId", "environmentName", "isDefault", "environmentState", "environmentUrl", "environmentType", "securityGroup", "purpose", "cdsMetadataName", "cdsInstanceUrl", "createdOn", "createdBy", "lastModifiedOn", "lastModifiedBy", "deletedOn", "environmentRegion", "tenantId", "governanceConfigurationType", "isBackgroundJobEnabled", "provisioningState", "retentionPeriod"}),

"Filtered rows" = Table.SelectRows(#"Expanded columns", each not Text.StartsWith([environmentId], "legacy-") or not Text.StartsWith([environmentId], "Legacy-")),

"Replace errors" = Table.ReplaceErrorValues(#"Filtered rows", {{"environmentId", null}, {"environmentName", null}, {"isDefault", null}, {"environmentState", null}, {"environmentUrl", null}, {"environmentType", null}, {"securityGroup", null}, {"purpose", null}, {"cdsMetadataName", null}, {"cdsInstanceUrl", null}, {"createdOn", null}, {"createdBy", null}, {"lastModifiedOn", null}, {"lastModifiedBy", null}, {"deletedOn", null}, {"environmentRegion", null}, {"tenantId", null}}),

"Replaced value" = Table.ReplaceValue(#"Replace errors", "Microsoft Dynamics Deployment Service", "SYSTEM", Replacer.ReplaceValue, {"createdBy"}),

// Strip out Legacy-/Default- from Environment ID

"Add Environment GUID" = Table.TransformColumnTypes(Table.AddColumn(#"Replaced value", "environmentGUID", each if Text.Length([environmentId]) = 36 then Text.Range([environmentId], 0,36) else Text.Range(Text.AfterDelimiter([environmentId], "-", 0), 0, 36)), {{"environmentGUID", type text}}),

"Add isNotDeleted" = Table.TransformColumnTypes(Table.AddColumn(#"Add Environment GUID", "isNotDeleted", each false), {{"isNotDeleted", type logical}}),

// Add a column with timestamp

"Add QueriedOn" = Table.AddColumn(#"Add isNotDeleted", "QueriedOn", each DateTimeZone.UtcNow()),

"Merged with makers" = Table.NestedJoin(#"Add QueriedOn", {"createdBy"}, admin_makers, {"admin_makerid"}, "admin_makers", JoinKind.LeftOuter),

"Expanded maker columns" = Table.ExpandTableColumn(#"Merged with makers", "admin_makers", {"admin_displayname", "admin_userprincipalname", "admin_makerid", "admin_makerisorphaned"}, {"admin_displayname", "admin_userprincipalname", "admin_makerid", "admin_makerisorphaned"}),

"Lowercased environment id" = Table.TransformColumns(#"Expanded maker columns", {{"environmentId", each Text.Lower(_), type nullable text}}),

// true/false if the environment has CDS

"Add hasCDS column" = Table.AddColumn(#"Lowercased environment id", "hasCDS", each if [cdsInstanceUrl] = "-" then "false" else "true", type text),

// If environment created by SYSTEM

"Replaced SYSTEM maker" = Table.ReplaceValue(#"Add hasCDS column", null, "SYSTEM", Replacer.ReplaceValue, {"admin_displayname"}),

"Replace IsOrphan flag for SYSTEM environments" = Table.ReplaceValue(#"Replaced SYSTEM maker", null, false, Replacer.ReplaceValue, {"admin_makerisorphaned"}),

Custom = Table.ReplaceValue(#"Replace IsOrphan flag for SYSTEM environments", "default-", "Default-", Replacer.ReplaceText, {"environmentId"}) ,

"Inserted conditional column" = Table.AddColumn(Custom, "IsOrphan", each if [admin_makerisorphaned] = true then "Yes" else "No"),

"Add createdBy ID column" = Table.AddColumn(#"Inserted conditional column", "combinedCreatedBy", each [admin_makerid], type text),

// Get additional meta data for environments from the ListEnvironmentsAPI

"Merged with ListEnvironmentsAPI" = Table.NestedJoin(#"Add createdBy ID column", {"environmentId"}, ListEnvironmentsAPI, {"value.name"}, "ListEnvironmentsAPI", JoinKind.LeftOuter),

"Replaced SYSTEM user" = Table.ReplaceValue(#"Merged with ListEnvironmentsAPI", "SYSTEM", null, Replacer.ReplaceValue, {"createdBy"}),

"Expand ListEnvironmentsAPI columns" = Table.ExpandTableColumn(#"Replaced SYSTEM user", "ListEnvironmentsAPI", {"value.properties.governanceConfiguration.protectionLevel", "CrmInstanceMetadataName", "value.properties.linkedEnvironmentMetadata.resourceId", "IsAdminMode", "BackgroundOpsEnabled", "value.properties.states.management.id"}, {"value.properties.governanceConfiguration.protectionLevel", "CrmInstanceMetadataName", "OrganizationID", "IsAdminMode", "BackgroundOpsEnabled", "EnvironmentStateManagementID"}),

"Inserted conditional column 1" = Table.AddColumn(#"Expand ListEnvironmentsAPI columns", "markDeleted", each if [environmentId] <> null then false else true),

"Inserted conditional column 2" = Table.AddColumn(#"Inserted conditional column 1", "markDeletedDate", each if [markDeleted] = false then null else [QueriedOn]),

"Changed column type" = Table.TransformColumnTypes(#"Inserted conditional column 2", {{"Name", type text}, {"environmentId", type text}, {"environmentName", type text}, {"isDefault", type logical}, {"environmentState", type text}, {"environmentUrl", type text}, {"environmentType", type text}, {"securityGroup", type text}, {"purpose", type any}, {"cdsMetadataName", type text}, {"cdsInstanceUrl", type text}, {"createdOn", type datetime}, {"createdBy", type text}, {"lastModifiedOn", type text}, {"lastModifiedBy", type text}, {"deletedOn", type text}, {"environmentRegion", type text}, {"tenantId", type text}, {"environmentGUID", type text}, {"QueriedOn", type datetimezone}, {"admin_displayname", type text}, {"admin_userprincipalname", type text}, {"admin_makerid", type text}, {"admin_makerisorphaned", type logical}, {"hasCDS", type logical}, {"IsOrphan", type text}, {"combinedCreatedBy", type text}, {"value.properties.governanceConfiguration.protectionLevel", type text}, {"CrmInstanceMetadataName", type text}, {"isNotDeleted", type logical}, {"IsAdminMode", type logical}, {"BackgroundOpsEnabled", type logical}, {"EnvironmentStateManagementID", type text}, {"markDeleted", type logical}, {"markDeletedDate", type datetime}})

in

"Changed column type";

shared #"Sample file (4)" = let Source = AzureStorage.DataLake(DatalakeURL & "/powerapps/environments"),

"Filtered hidden files" = Table.SelectRows(Source, each [Attributes]?[Hidden]? <> true),

Navigation = #"Filtered hidden files"{0}[Content]

in Navigation; shared #"Parameter (4)" = let Parameter = #"Sample file (4)" meta [IsParameterQuery = true, IsParameterQueryRequired = true, Type = type binary] in Parameter; [Description = "Environment Web API endpoint. Get via make.powerapps.com > Settings Cog > Developer resources > Web API Endpoint value (example https://mycoe.api.crm.dynamics.com/api/data/v9.2)"] shared EnvironmentAPI = let EnvironmentAPI = "https://pplatcoe.crm.dynamics.com/api/data/v9.2" meta [IsParameterQuery = true, IsParameterQueryRequired = true, Type = type any] in EnvironmentAPI; shared DatalakeURL = let DatalakeURL_Apps = "https://exptenantanalytics.dfs.core.windows.net/powerplatform" meta [IsParameterQuery = true, IsParameterQueryRequired = true, Type = type any] in DatalakeURL_Apps; shared admin_makers = let Source = OData.Feed(#"EnvironmentAPI", null, [Implementation = "2.0"]),

"Navigation 1" = Source{[Name = "admin_makers", Signature = "table"]}[Data],

#"Removed columns" = Table.RemoveColumns(#"Navigation 1", {"utcconversiontimezonecode", "admin_m2managerprinciplename", "modifiedon", "admin_lastpulsesurvey", "admin_preferredlanguage", "admin_numberofuiflows_state", "admin_numberofcustomconnectors", "admin_numberofsharepointapps_date", "admin_numberofpvas_date", "admin_userisserviceprinciple", "admin_numberofflows_date", "admin_country", "admin_numberofpvas_state", "importsequencenumber", "admin_numberofcustomconnectors_date", "admin_numberofuiflows", "admin_numberofuiflows_date", "timezoneruleversionnumber", "admin_photoobject", "admin_numberofflows_state", "admin_numberofenvironments_state", "admin_department", "admin_jobtitle", "admin_managerid", "_owningteam_value", "admin_numberofenvironments", "admin_city", "_owninguser_value", "admin_managerprinciplename", "_modifiedby_value", "admin_accountenabled", "admin_numberofenvironments_date", "admin_photoobject_url", "admin_numberofflows", "admin_office", "admin_numberofapps", "admin_numberofcanvasapps", "admin_numberofcanvasapps_state", "overriddencreatedon", "versionnumber", "_createdby_value", "admin_numberofsharepointapps", "_owningbusinessunit_value", "admin_numberofcanvasapps_date", "admin_photoobject_timestamp", "statecode", "admin_numberofapps_date", "admin_numberofpvas", "statuscode", "admin_photoobjectid", "_ownerid_value", "admin_photo", "admin_numberofapps_state", "admin_numberofcustomconnectors_state", "_modifiedonbehalfby_value", "admin_numberofsharepointapps_state", "_createdonbehalfby_value", "admin_numberofmodeldrivenapps_date", "createdon", "admin_company", "admin_numberofmodeldrivenapps", "admin_numberofmodeldrivenapps_state", "createdby", "createdonbehalfby", "modifiedby", "modifiedonbehalfby", "owninguser", "owningteam", "ownerid", "owningbusinessunit", "admin_maker_SyncErrors", "admin_maker_DuplicateMatchingRecord", "admin_maker_DuplicateBaseRecord", "admin_maker_AsyncOperations", "admin_maker_MailboxTrackingFolders", "admin_maker_ProcessSession", "admin_maker_BulkDeleteFailures", "admin_maker_PrincipalObjectAttributeAccesses", "admin_admin_bpf_maker", "admin_admin_solution_SolutionCreator_admin_make", "admin_App_AppOwner_admin_Maker", "admin_EnvironmentMaker", "admin_Flow_DerivedOwner_admin_Maker", "admin_Flow_FlowCreator_admin_Maker", "admin_portal_PortalOwner_admin_Maker", "admin_PVA_PVAOwner_admin_Maker", "admin_rel_ConnectorMaker", "admin_RPA_RPAOwner_admin_Maker", "coe_EnvCreationRequest_SystemAdmins", "admin_admin_aibuildermodel_AiCreator_admin_make"})

in

"Removed columns";

shared Environments = let Source = AzureStorage.DataLake(DatalakeURL & "/powerapps/environments"), excludehiddenfiles = Table.SelectRows(Source, each [Attributes]?[Hidden]? <> true), excludezerolengthfiles = Table.SelectRows(excludehiddenfiles, each [Attributes]?[Size]? > 0),

"Get content from json files" = Table.AddColumn(excludezerolengthfiles, "Transform file", each #"Transform file (4)"([Content])),

"Remove unneeded columns" = Table.SelectColumns(#"Get content from json files", {"Name", "Transform file"}),

"Expanded columns" = Table.ExpandTableColumn(#"Remove unneeded columns", "Transform file", {"environmentId", "environmentName", "isDefault", "environmentState", "environmentUrl", "environmentType", "securityGroup", "purpose", "cdsInstanceId", "cdsInstanceUrl", "createdTime", "createdPrincipalId", "lastModifiedTime", "lastModifiedPrincipalId", "deletedTime", "environmentRegion", "tenantGuid", "governanceConfigurationType", "isBackgroundJobEnabled", "provisioningState", "retentionPeriod"}, {"environmentId", "environmentName", "isDefault", "environmentState", "environmentUrl", "environmentType", "securityGroup", "purpose", "cdsMetadataName", "cdsInstanceUrl", "createdOn", "createdBy", "lastModifiedOn", "lastModifiedBy", "deletedOn", "environmentRegion", "tenantId", "governanceConfigurationType", "isBackgroundJobEnabled", "provisioningState", "retentionPeriod"}),

"Filtered rows" = Table.SelectRows(#"Expanded columns", each not Text.StartsWith([environmentId], "legacy-") or not Text.StartsWith([environmentId], "Legacy-")),

"Replace errors" = Table.ReplaceErrorValues(#"Filtered rows", {{"environmentId", null}, {"environmentName", null}, {"isDefault", null}, {"environmentState", null}, {"environmentUrl", null}, {"environmentType", null}, {"securityGroup", null}, {"purpose", null}, {"cdsMetadataName", null}, {"cdsInstanceUrl", null}, {"createdOn", null}, {"createdBy", null}, {"lastModifiedOn", null}, {"lastModifiedBy", null}, {"deletedOn", null}, {"environmentRegion", null}, {"tenantId", null}}),

"Replaced value" = Table.ReplaceValue(#"Replace errors", "Microsoft Dynamics Deployment Service", "SYSTEM", Replacer.ReplaceValue, {"createdBy"}),

// Strip out Legacy-/Default- from Environment ID

"Add Environment GUID" = Table.TransformColumnTypes(Table.AddColumn(#"Replaced value", "environmentGUID", each if Text.Length([environmentId]) = 36 then Text.Range([environmentId], 0,36) else Text.Range(Text.AfterDelimiter([environmentId], "-", 0), 0, 36)), {{"environmentGUID", type text}}),

"Add isNotDeleted" = Table.TransformColumnTypes(Table.AddColumn(#"Add Environment GUID", "isNotDeleted", each false), {{"isNotDeleted", type logical}}),

// Add a column with timestamp

"Add QueriedOn" = Table.AddColumn(#"Add isNotDeleted", "QueriedOn", each DateTimeZone.UtcNow()),

"Merged with makers" = Table.NestedJoin(#"Add QueriedOn", {"createdBy"}, admin_makers, {"admin_makerid"}, "admin_makers", JoinKind.LeftOuter),

"Expanded maker columns" = Table.ExpandTableColumn(#"Merged with makers", "admin_makers", {"admin_displayname", "admin_userprincipalname", "admin_makerid", "admin_makerisorphaned"}, {"admin_displayname", "admin_userprincipalname", "admin_makerid", "admin_makerisorphaned"}),

"Lowercased environment id" = Table.TransformColumns(#"Expanded maker columns", {{"environmentId", each Text.Lower(_), type nullable text}}),

// true/false if the environment has CDS

"Add hasCDS column" = Table.AddColumn(#"Lowercased environment id", "hasCDS", each if [cdsInstanceUrl] = "-" then "false" else "true", type text),

// If environment created by SYSTEM

"Replaced SYSTEM maker" = Table.ReplaceValue(#"Add hasCDS column", null, "SYSTEM", Replacer.ReplaceValue, {"admin_displayname"}),

"Replace IsOrphan flag for SYSTEM environments" = Table.ReplaceValue(#"Replaced SYSTEM maker", null, false, Replacer.ReplaceValue, {"admin_makerisorphaned"}),

Custom = Table.ReplaceValue(#"Replace IsOrphan flag for SYSTEM environments", "default-", "Default-", Replacer.ReplaceText, {"environmentId"}),

"Inserted conditional column" = Table.AddColumn(Custom, "IsOrphan", each if [admin_makerisorphaned] = true then "Yes" else "No"),

"Add createdBy ID column" = Table.AddColumn(#"Inserted conditional column", "combinedCreatedBy", each [admin_makerid], type text),

// Get additional meta data for environments from the ListEnvironmentsAPI

"Merged with ListEnvironmentsAPI" = Table.NestedJoin(#"Add createdBy ID column", {"environmentId"}, ListEnvironmentsAPI, {"value.name"}, "ListEnvironmentsAPI", JoinKind.LeftOuter),

"Replaced SYSTEM user" = Table.ReplaceValue(#"Merged with ListEnvironmentsAPI", "SYSTEM", null, Replacer.ReplaceValue, {"createdBy"}),

"Expand ListEnvironmentsAPI columns" = Table.ExpandTableColumn(#"Replaced SYSTEM user", "ListEnvironmentsAPI", {"value.properties.governanceConfiguration.protectionLevel", "CrmInstanceMetadataName", "value.properties.linkedEnvironmentMetadata.resourceId", "IsAdminMode", "BackgroundOpsEnabled", "value.properties.states.management.id"}, {"value.properties.governanceConfiguration.protectionLevel", "CrmInstanceMetadataName", "OrganizationID", "IsAdminMode", "BackgroundOpsEnabled", "EnvironmentStateManagementID"}),

"Inserted conditional column 1" = Table.AddColumn(#"Expand ListEnvironmentsAPI columns", "markDeleted", each if [environmentId] <> null then false else true),

"Inserted conditional column 2" = Table.AddColumn(#"Inserted conditional column 1", "markDeletedDate", each if [markDeleted] = false then null else [QueriedOn]),

"Changed column type" = Table.TransformColumnTypes(#"Inserted conditional column 2", {{"Name", type text}, {"environmentId", type text}, {"environmentName", type text}, {"isDefault", type logical}, {"environmentState", type text}, {"environmentUrl", type text}, {"environmentType", type text}, {"securityGroup", type text}, {"purpose", type any}, {"cdsMetadataName", type text}, {"cdsInstanceUrl", type text}, {"createdOn", type datetime}, {"createdBy", type text}, {"lastModifiedOn", type text}, {"lastModifiedBy", type text}, {"deletedOn", type text}, {"environmentRegion", type text}, {"tenantId", type text}, {"environmentGUID", type text}, {"QueriedOn", type datetimezone}, {"admin_displayname", type text}, {"admin_userprincipalname", type text}, {"admin_makerid", type text}, {"admin_makerisorphaned", type logical}, {"hasCDS", type logical}, {"IsOrphan", type text}, {"combinedCreatedBy", type text}, {"value.properties.governanceConfiguration.protectionLevel", type text}, {"CrmInstanceMetadataName", type text}, {"isNotDeleted", type logical}, {"IsAdminMode", type logical}, {"BackgroundOpsEnabled", type logical}, {"EnvironmentStateManagementID", type text}, {"markDeleted", type logical}, {"markDeletedDate", type datetime}})

in

"Changed column type";

shared ListEnvironmentsAPI = let Source = Json.Document(Web.Contents("https://api.bap.microsoft.com/providers/Microsoft.BusinessAppPlatform/scopes/admin/environments?api-version=2020-10-01")),

"Converted to table" = Table.FromRecords({Source}),

"Expanded value" = Table.ExpandListColumn(#"Converted to table", "value"),

"Expanded value1" = Table.ExpandRecordColumn(#"Expanded value", "value", {"id", "type", "location", "name", "properties"}, {"value.id", "value.type", "value.location", "value.name", "value.properties"}),

"Expanded value.properties" = Table.ExpandRecordColumn(#"Expanded value1", "value.properties", {"tenantId", "azureRegion", "displayName", "description", "createdTime", "createdBy", "lastModifiedTime", "usedBy", "lastModifiedBy", "provisioningState", "provisioningDetails", "creationType", "environmentSku", "isDefault", "clientUris", "runtimeEndpoints", "databaseType", "linkedEnvironmentMetadata", "trialScenarioType", "notificationMetadata", "retentionPeriod", "states", "updateCadence", "retentionDetails", "protectionStatus", "cluster", "connectedGroups", "lifecycleOperationsEnforcement", "governanceConfiguration"}, {"value.properties.tenantId", "value.properties.azureRegion", "value.properties.displayName", "value.properties.description", "value.properties.createdTime", "value.properties.createdBy", "value.properties.lastModifiedTime", "value.properties.usedBy", "value.properties.lastModifiedBy", "value.properties.provisioningState", "value.properties.provisioningDetails", "value.properties.creationType", "value.properties.environmentSku", "value.properties.isDefault", "value.properties.clientUris", "value.properties.runtimeEndpoints", "value.properties.databaseType", "value.properties.linkedEnvironmentMetadata", "value.properties.trialScenarioType", "value.properties.notificationMetadata", "value.properties.retentionPeriod", "value.properties.states", "value.properties.updateCadence", "value.properties.retentionDetails", "value.properties.protectionStatus", "value.properties.cluster", "value.properties.connectedGroups", "value.properties.lifecycleOperationsEnforcement", "value.properties.governanceConfiguration"}),

"Expanded value.properties.createdBy" = Table.ExpandRecordColumn(#"Expanded value.properties", "value.properties.createdBy", {"id", "displayName", "email", "type", "tenantId", "userPrincipalName"}, {"value.properties.createdBy.id", "value.properties.createdBy.displayName", "value.properties.createdBy.email", "value.properties.createdBy.type", "value.properties.createdBy.tenantId", "value.properties.createdBy.userPrincipalName"}),

"Expanded value.properties.usedBy" = Table.ExpandRecordColumn(#"Expanded value.properties.createdBy", "value.properties.usedBy", {"id", "type", "tenantId", "userPrincipalName"}, {"value.properties.usedBy.id", "value.properties.usedBy.type", "value.properties.usedBy.tenantId", "value.properties.usedBy.userPrincipalName"}),

"Expanded value.properties.lastModifiedBy" = Table.ExpandRecordColumn(#"Expanded value.properties.usedBy", "value.properties.lastModifiedBy", {"id", "displayName", "email", "type", "tenantId", "userPrincipalName"}, {"value.properties.lastModifiedBy.id", "value.properties.lastModifiedBy.displayName", "value.properties.lastModifiedBy.email", "value.properties.lastModifiedBy.type", "value.properties.lastModifiedBy.tenantId", "value.properties.lastModifiedBy.userPrincipalName"}),

"Expanded value.properties.provisioningDetails" = Table.ExpandRecordColumn(#"Expanded value.properties.lastModifiedBy", "value.properties.provisioningDetails", {"message", "operations"}, {"value.properties.provisioningDetails.message", "value.properties.provisioningDetails.operations"}),

"Expanded value.properties.clientUris" = Table.ExpandRecordColumn(#"Expanded value.properties.provisioningDetails", "value.properties.clientUris", {"admin", "maker"}, {"value.properties.clientUris.admin", "value.properties.clientUris.maker"}),

"Expanded value.properties.runtimeEndpoints" = Table.ExpandRecordColumn(#"Expanded value.properties.clientUris", "value.properties.runtimeEndpoints", {"microsoft.BusinessAppPlatform", "microsoft.CommonDataModel", "microsoft.PowerApps", "microsoft.PowerAppsAdvisor", "microsoft.PowerVirtualAgents", "microsoft.ApiManagement", "microsoft.Flow"}, {"value.properties.runtimeEndpoints.microsoft.BusinessAppPlatform", "value.properties.runtimeEndpoints.microsoft.CommonDataModel", "value.properties.runtimeEndpoints.microsoft.PowerApps", "value.properties.runtimeEndpoints.microsoft.PowerAppsAdvisor", "value.properties.runtimeEndpoints.microsoft.PowerVirtualAgents", "value.properties.runtimeEndpoints.microsoft.ApiManagement", "value.properties.runtimeEndpoints.microsoft.Flow"}),

"Expanded value.properties.linkedEnvironmentMetadata" = Table.ExpandRecordColumn(#"Expanded value.properties.runtimeEndpoints", "value.properties.linkedEnvironmentMetadata", {"resourceId", "friendlyName", "uniqueName", "domainName", "version", "instanceUrl", "instanceApiUrl", "baseLanguage", "instanceState", "createdTime", "securityGroupId", "backgroundOperationsState", "scaleGroup", "platformSku"}, {"value.properties.linkedEnvironmentMetadata.resourceId", "value.properties.linkedEnvironmentMetadata.friendlyName", "value.properties.linkedEnvironmentMetadata.uniqueName", "value.properties.linkedEnvironmentMetadata.domainName", "value.properties.linkedEnvironmentMetadata.version", "value.properties.linkedEnvironmentMetadata.instanceUrl", "value.properties.linkedEnvironmentMetadata.instanceApiUrl", "value.properties.linkedEnvironmentMetadata.baseLanguage", "value.properties.linkedEnvironmentMetadata.instanceState", "value.properties.linkedEnvironmentMetadata.createdTime", "value.properties.linkedEnvironmentMetadata.securityGroupId", "value.properties.linkedEnvironmentMetadata.backgroundOperationsState", "value.properties.linkedEnvironmentMetadata.scaleGroup", "value.properties.linkedEnvironmentMetadata.platformSku"}),

"Expanded value.properties.notificationMetadata" = Table.ExpandRecordColumn(#"Expanded value.properties.linkedEnvironmentMetadata", "value.properties.notificationMetadata", {"state", "branding"}, {"value.properties.notificationMetadata.state", "value.properties.notificationMetadata.branding"}),

"Expanded value.properties.states" = Table.ExpandRecordColumn(#"Expanded value.properties.notificationMetadata", "value.properties.states", {"management", "runtime"}, {"value.properties.states.management", "value.properties.states.runtime"}),

"Expanded value.properties.states.management" = Table.ExpandRecordColumn(#"Expanded value.properties.states", "value.properties.states.management", {"id"}, {"value.properties.states.management.id"}),

"Expanded value.properties.states.runtime" = Table.ExpandRecordColumn(#"Expanded value.properties.states.management", "value.properties.states.runtime", {"runtimeReason", "runtimeReasonCode", "requestedBy", "reason", "id"}, {"value.properties.states.runtime.runtimeReason", "value.properties.states.runtime.runtimeReasonCode", "value.properties.states.runtime.requestedBy", "value.properties.states.runtime.reason", "value.properties.states.runtime.id"}),

"Expanded value.properties.states.runtime.requestedBy" = Table.ExpandRecordColumn(#"Expanded value.properties.states.runtime", "value.properties.states.runtime.requestedBy", {"displayName", "type"}, {"value.properties.states.runtime.requestedBy.displayName", "value.properties.states.runtime.requestedBy.type"}),

"Expanded value.properties.updateCadence" = Table.ExpandRecordColumn(#"Expanded value.properties.states.runtime.requestedBy", "value.properties.updateCadence", {"id"}, {"value.properties.updateCadence.id"}),

"Expanded value.properties.retentionDetails" = Table.ExpandRecordColumn(#"Expanded value.properties.updateCadence", "value.properties.retentionDetails", {"retentionPeriod", "backupsAvailableFromDateTime"}, {"value.properties.retentionDetails.retentionPeriod", "value.properties.retentionDetails.backupsAvailableFromDateTime"}),

"Expanded value.properties.protectionStatus" = Table.ExpandRecordColumn(#"Expanded value.properties.retentionDetails", "value.properties.protectionStatus", {"keyManagedBy"}, {"value.properties.protectionStatus.keyManagedBy"}),

"Expanded value.properties.cluster" = Table.ExpandRecordColumn(#"Expanded value.properties.protectionStatus", "value.properties.cluster", {"category", "number", "uriSuffix", "geoShortName", "environment"}, {"value.properties.cluster.category", "value.properties.cluster.number", "value.properties.cluster.uriSuffix", "value.properties.cluster.geoShortName", "value.properties.cluster.environment"}),

"Expanded value.properties.lifecycleOperationsEnforcement" = Table.ExpandRecordColumn(#"Expanded value.properties.cluster", "value.properties.lifecycleOperationsEnforcement", {"allowedOperations", "disallowedOperations"}, {"value.properties.lifecycleOperationsEnforcement.allowedOperations", "value.properties.lifecycleOperationsEnforcement.disallowedOperations"}),

"Expanded value.properties.governanceConfiguration" = Table.ExpandRecordColumn(#"Expanded value.properties.lifecycleOperationsEnforcement", "value.properties.governanceConfiguration", {"protectionLevel"}, {"value.properties.governanceConfiguration.protectionLevel"}),

"Removed columns" = Table.RemoveColumns(#"Expanded value.properties.governanceConfiguration", {"value.id", "value.type", "value.location", "value.properties.tenantId", "value.properties.azureRegion", "value.properties.displayName", "value.properties.description", "value.properties.createdTime", "value.properties.createdBy.type", "value.properties.createdBy.tenantId", "value.properties.createdBy.userPrincipalName", "value.properties.lastModifiedTime", "value.properties.usedBy.id", "value.properties.usedBy.type", "value.properties.usedBy.tenantId", "value.properties.usedBy.userPrincipalName", "value.properties.lastModifiedBy.id", "value.properties.lastModifiedBy.displayName", "value.properties.lastModifiedBy.email", "value.properties.lastModifiedBy.type", "value.properties.lastModifiedBy.tenantId", "value.properties.lastModifiedBy.userPrincipalName", "value.properties.provisioningState", "value.properties.provisioningDetails.message", "value.properties.provisioningDetails.operations", "value.properties.creationType", "value.properties.environmentSku", "value.properties.isDefault", "value.properties.clientUris.admin", "value.properties.clientUris.maker", "value.properties.runtimeEndpoints.microsoft.BusinessAppPlatform", "value.properties.runtimeEndpoints.microsoft.CommonDataModel", "value.properties.runtimeEndpoints.microsoft.PowerApps", "value.properties.runtimeEndpoints.microsoft.PowerAppsAdvisor", "value.properties.runtimeEndpoints.microsoft.PowerVirtualAgents", "value.properties.runtimeEndpoints.microsoft.ApiManagement", "value.properties.runtimeEndpoints.microsoft.Flow", "value.properties.databaseType", "value.properties.linkedEnvironmentMetadata.friendlyName", "value.properties.linkedEnvironmentMetadata.domainName", "value.properties.linkedEnvironmentMetadata.version", "value.properties.linkedEnvironmentMetadata.instanceApiUrl", "value.properties.linkedEnvironmentMetadata.baseLanguage", "value.properties.linkedEnvironmentMetadata.instanceState", "value.properties.linkedEnvironmentMetadata.createdTime", "value.properties.linkedEnvironmentMetadata.securityGroupId", "value.properties.linkedEnvironmentMetadata.scaleGroup", "value.properties.linkedEnvironmentMetadata.platformSku", "value.properties.trialScenarioType", "value.properties.notificationMetadata.state", "value.properties.notificationMetadata.branding", "value.properties.retentionPeriod", "value.properties.states.runtime.runtimeReason", "value.properties.states.runtime.runtimeReasonCode", "value.properties.states.runtime.requestedBy.displayName", "value.properties.states.runtime.requestedBy.type", "value.properties.states.runtime.reason", "value.properties.updateCadence.id", "value.properties.retentionDetails.retentionPeriod", "value.properties.retentionDetails.backupsAvailableFromDateTime", "value.properties.protectionStatus.keyManagedBy", "value.properties.cluster.category", "value.properties.cluster.number", "value.properties.cluster.uriSuffix", "value.properties.cluster.geoShortName", "value.properties.cluster.environment", "value.properties.connectedGroups", "value.properties.lifecycleOperationsEnforcement.allowedOperations", "value.properties.lifecycleOperationsEnforcement.disallowedOperations"}),

"Inserted Text Between Delimiters" = Table.AddColumn(#"Removed columns", "Text Between Delimiters", each Text.BetweenDelimiters([value.properties.linkedEnvironmentMetadata.instanceUrl], ".", "."), type text),

"Renamed columns" = Table.RenameColumns(#"Inserted Text Between Delimiters", {{"Text Between Delimiters", "CrmText"}}),

"Inserted Merged Column" = Table.AddColumn(#"Renamed columns", "Merged", each Text.Combine({[value.properties.linkedEnvironmentMetadata.uniqueName], ".", [CrmText]}), type text),

"Renamed columns 1" = Table.RenameColumns(#"Inserted Merged Column", {{"Merged", "CrmInstanceMetadataName"}}),

"Lowercased text" = Table.TransformColumns(#"Renamed columns 1", {{"value.name", each Text.Lower(Text.From(_)), type nullable text}}),

"Inserted IsAdminMode" = Table.AddColumn(#"Lowercased text", "IsAdminMode", each if [value.properties.states.runtime.id] = "AdminMode" then true else false),

"Inserted BackgroundOpsEnabled" = Table.AddColumn(#"Inserted IsAdminMode", "BackgroundOpsEnabled", each if [value.properties.linkedEnvironmentMetadata.backgroundOperationsState] = "Disabled" then false else true)

in

"Inserted BackgroundOpsEnabled";

Jenefer-Monroe commented 9 months ago

You may need to delete run history, perhaps its matching to the old casing.

Jenefer-Monroe commented 9 months ago

@Jenefer-Monroe, after performing the above fix, our PowerBI dashboard is now showing less number of PowerApps. Admin View is showing 10k+ items compared to the PowerBI report - it is now only showing 2k+ items. Do you have any idea why this is the case?

Checking the number, it seems that it is not getting the number of PowerApps in the default environment and it is not showing in PowerBI Report.

I did not test or make this change in the PBI's dataflow, only in the dataflow which feeds the dataverse tables.

bartsneary commented 9 months ago

Thanks for replying Jenefer! I tried clearing out the Dataflow refresh history table. It didn't make any difference. In case it helps, my query seems to error on this transform action on the 4th step of the query.

[cid:79e0df9a-c59e-4795-b8df-65ed1eadbfe9]

[cid:b82a5501-08b7-4136-8261-f5e718211574]

Bart Sneary

MTC Architect | 952-837-4560 W or 507-403-3088 M

@.**@.>

3601 West 76th St, Suite 600, Edina, MN 55435 | www.microsoft.com/mtchttp://www.microsoft.com/mtc

[cid:903343cc-3dca-4369-910a-add69b8ddb1a]

From: Jenefer Monroe @.> Sent: Monday, February 5, 2024 1:51 PM To: microsoft/coe-starter-kit @.> Cc: Bart Sneary @.>; Comment @.> Subject: Re: [microsoft/coe-starter-kit] Data Export : Default Environment keeps getting marked as deleted (Issue #7562)

You may need to delete run history, perhaps its matching to the old casing.

— Reply to this email directly, view it on GitHubhttps://github.com/microsoft/coe-starter-kit/issues/7562#issuecomment-1927964301, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJ645VFQSEUZDDI67WZG4JLYSEZ3BAVCNFSM6AAAAABCRBSE3OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRXHE3DIMZQGE. You are receiving this because you commented.Message ID: @.***>

Jenefer-Monroe commented 9 months ago

@Bart are you editing the Dataflow that is used for the PBI or the Dataflow that is used for bringing the data into the envt?

This is the only Dataflow I edited image

Jenefer-Monroe commented 9 months ago

Or (your picture did not come through) are you saying that is what you editted and you failed at this step? image

bartsneary commented 9 months ago

I was trying to edit the Dataflow bringing the data from the lake into Dataverse.

[cid:1a070ffb-5cf7-4967-8115-441f8a86ebd4]

Bart Sneary

MTC Architect | 952-837-4560 W or 507-403-3088 M

@.**@.>

3601 West 76th St, Suite 600, Edina, MN 55435 | www.microsoft.com/mtchttp://www.microsoft.com/mtc

[cid:cb741157-d2f9-48ae-ba75-1fe1abc71e57]

From: Jenefer Monroe @.> Sent: Tuesday, February 6, 2024 7:38 AM To: microsoft/coe-starter-kit @.> Cc: Bart Sneary @.>; Comment @.> Subject: Re: [microsoft/coe-starter-kit] Data Export : Default Environment keeps getting marked as deleted (Issue #7562)

@barthttps://github.com/bart are you editing the Dataflow that is used for the PBI or the Dataflow that is used for bringing the data into the envt?

— Reply to this email directly, view it on GitHubhttps://github.com/microsoft/coe-starter-kit/issues/7562#issuecomment-1929439869, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJ645VDOS6SAU2VQA4H26P3YSIP3DAVCNFSM6AAAAABCRBSE3OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRZGQZTSOBWHE. You are receiving this because you commented.Message ID: @.***>

bartsneary commented 9 months ago

Yes! That last cell on that row shows "Error"

[cid:39aae894-e03f-4293-8b44-a8f332fc2e8b]

Bart Sneary

MTC Architect | 952-837-4560 W or 507-403-3088 M

@.**@.>

3601 West 76th St, Suite 600, Edina, MN 55435 | www.microsoft.com/mtchttp://www.microsoft.com/mtc

[cid:ab3b56e0-7b83-4897-9772-2e84428f53cf]

From: Jenefer Monroe @.> Sent: Tuesday, February 6, 2024 7:47 AM To: microsoft/coe-starter-kit @.> Cc: Bart Sneary @.>; Comment @.> Subject: Re: [microsoft/coe-starter-kit] Data Export : Default Environment keeps getting marked as deleted (Issue #7562)

Or (your picture did not come through) are you saying that is what you editted and you failed at this step? image.png (view on web)https://github.com/microsoft/coe-starter-kit/assets/89584709/1ed7881b-0ac2-49be-b285-0a468757da90

— Reply to this email directly, view it on GitHubhttps://github.com/microsoft/coe-starter-kit/issues/7562#issuecomment-1929455465, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJ645VF3EV2C77GYI3BBO4LYSIQ5PAVCNFSM6AAAAABCRBSE3OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRZGQ2TKNBWGU. You are receiving this because you commented.Message ID: @.***>

Jenefer-Monroe commented 9 months ago

ok we have seen that before, I believe it was because the flow files were malformatted

Please go to the Datalake and find the flow file for an environment where you expect to see flows but do not. You can do this by going to the Storage Browser and searching for the GUID of the envt image

Is the file formatted like image 1 with a single line per flow or like image 2 with a jumbled look?

Image 1 : Well formatted image

Image 2 : Not well formatted image

bartsneary commented 9 months ago

Jenefer,

I didn't look through all environments, but did check Personal Productivity (that has no record right now in the COE Kit Admin App, so no Flows show up either). I also checked about 30 others going back from today in descending last modified date order from the data lake. All were well formatted.

I'm fine waiting until tomorrow when the Feb update comes out, updating everything to the latest versions and seeing if the default environment keeps getting deleted after that as part of the BYODL processes. If it matters, according to audit logs, this started happening on 1-8-24. I know I updated all the Dataflow definitions the last time I installed the kit and all set up fine as I set things up. I thought that was on 1-4 or 1-5. It's kinda weird that all the Dataflows last published dates show in the tool tonight as 3-14-23.

Refreshing the Dataflows manually seems to bring it back, but the next automated run seems to make it disappear again.

Bart

From: Jenefer Monroe @.> Sent: Tuesday, February 6, 2024 10:57 AM To: microsoft/coe-starter-kit @.> Cc: Bart Sneary @.>; Comment @.> Subject: Re: [microsoft/coe-starter-kit] Data Export : Default Environment keeps getting marked as deleted (Issue #7562)

ok we have seen that before, I believe it was because the flow files were malformatted

Please go to the Datalake and find the flow file for an environment where you expect to see flows but do not. You can do this by going to the Storage Browser and searching for the GUID of the envt image.png (view on web)https://github.com/microsoft/coe-starter-kit/assets/89584709/160575cf-225f-4d31-b7db-88fc079f3ed7

Is the file formatted like image 1 with a single line per flow or like image 2 with a jumbled look?

Image 1 : Well formatted image.png (view on web)https://github.com/microsoft/coe-starter-kit/assets/89584709/dbf210ec-8544-4e82-a03e-e6196f0685fc

Image 2 : Not well formatted image.png (view on web)https://github.com/microsoft/coe-starter-kit/assets/89584709/589de3df-db6c-4101-ba16-05fd530c56c0

— Reply to this email directly, view it on GitHubhttps://github.com/microsoft/coe-starter-kit/issues/7562#issuecomment-1930360256, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJ645VDU6KDGAFGJYLM5CZTYSJOHFAVCNFSM6AAAAABCRBSE3OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZQGM3DAMRVGY. You are receiving this because you commented.Message ID: @.***>

Jenefer-Monroe commented 9 months ago

Great lets do that. And if you have an issue please post a new one and just reference this as it will close when we ship.