microsoft / coe-starter-kit

Other
737 stars 216 forks source link

[ALM Accelerator - Feature]: Option to remove environment variable values on deploy #4215

Closed LukasSchauffele closed 1 year ago

LukasSchauffele commented 1 year ago

Is your feature request related to a problem? Please describe.

I will add my question/request for the ALM accelerator to start with even though the underlying issue is a bug with the platform.

Is it possible to remove the environment variable values in a downstream environment before importing a solution into an environment? Maybe using hooks?

The reason I want to do this is because I am sometimes getting an error on import when the environment variable in the deploymentSettings.json and the environment variable value in the downstream environment do not match.

I've actually already posted about this error before - #3101 - and raised a ticket with support (2207120030000937 and 2211160030000704) but haven't gotten anywhere. The response from support was to delete the value of the environment variable manually before importing a solution.

This screenshot shows the current value of the environment variable in the downstream environment I am importing in, the value is 8e197461-21d1-429b-9c53-e7792b2ce555:

CleanShot 2022-11-17 at 11 38 08

The value for this environment variable in my deploymentSettings-demo.json matches. The solution successfully imports.

CleanShot 2022-11-17 at 11 37 20

Once I change the value in deploymentSettings-demo.json to something other than what's currently in the environment the import fails (I've added the full error message below).

CleanShot 2022-11-17 at 11 40 55

If I remove the environment variable value manually and then re-run the import pipeline, everything works.

CleanShot 2022-11-17 at 11 45 49

CleanShot 2022-11-17 at 11 46 59

Describe the solution you'd like

Ideally, pac should be able to import into an environment with environment variables specified in the --settings-file without issues. Can you please pass this on to the pac team?

In the mean time, could someone please assist me in writing a hook to clear the environment variable values in the downstream environment? The current process is:

  1. Run pipeline and cross my fingers
  2. Sometimes it runs without issues, sometimes it fails
  3. Manually delete environment variable value in environment
  4. Re-run pipeline

Describe alternatives you've considered

Continue to manually delete environment variables.

Additional context?

Error message:

Error: Exception type: System.ServiceModel.FaultException`1[Microsoft.Xrm.Sdk.OrganizationServiceFault]
Message: Cannot insert duplicate key exception when executing non-query: System.Data.SqlClient.SqlCommand Exception: System.Data.SqlClient.SqlException (0x80131904): Violation of UNIQUE KEY constraint 'ndx_envvarval_schemaname'. Cannot insert duplicate key in object 'dbo.EnvironmentVariableValueBase'. The duplicate key value is (bis_CapexRegisterAppID, 0, Jan  1 1900 12:00AM).
The statement has been terminated.
   at Microsoft.Crm.CrmDbConnection.HandleExecutionExceptionAbstract(Exception exception, String commandText, IDbConnection connection, IDbCommand command, IDbTransaction transaction, Boolean isOpen, Int32 commandTimeout, IConnectionRetryPolicy localRetryPolicy, Int32& retryCount)
   at Microsoft.Crm.CrmDbConnection.InternalExecuteWithRetryAbstract[TResult](Func`1 executeMethod, IDbCommand command, IConnectionRetryPolicy retryPolicy)
   at Microsoft.Crm.CrmDbConnection.<>c__DisplayClass229_0.<InternalExecuteNonQuery>b__0()
   at Microsoft.PowerApps.CoreFramework.ActivityLoggerExtensions.Execute[TResult](ILogger logger, EventId eventId, ActivityType activityType, Func`1 func, IEnumerable`1 additionalCustomProperties)
   at Microsoft.Xrm.Telemetry.XrmTelemetryExtensions.Execute[TResult](ILogger logger, XrmTelemetryActivityType activityType, Func`1 func)
   at Microsoft.Crm.Platform.Server.DataEngine.SqlDataAccessManager.ExecuteNonQuery(IDbCommand command, ISqlExecutionContext context, Int32 sourceLineNumber, String memberName, String sourceFilePath)
   at Microsoft.Crm.BusinessEntities.BusinessProcessObject.ExecuteNonQueryAbstract(IDbCommand command, ISqlExecutionContext context)
   at Microsoft.Crm.BusinessEntities.BusinessProcessObject.ExecuteNonQueryAbstract(IDbCommand command, ExecutionContext context)
   at Microsoft.Crm.Platform.SolutionAwareComponents.UpdateExecutor.ExecuteUpdate(IDbCommand command, ExecutionContext context)
ClientConnectionId:cc361d21-7b46-4ea6-93c8-7d9bf66cd26c
Error Number:2627,State:1,Class:14
ClientConnectionId before routing:8fa74187-a45d-4cd8-8464-18865794837b
Routing Destination:f2506fb14b87.tr369.australiasoutheast1-a.worker.database.windows.net,11001Detail:
<OrganizationServiceFault xmlns:i="http://www.w3.org/2001/XMLSchema-instance"
  xmlns="http://schemas.microsoft.com/xrm/2011/Contracts">
  <ActivityId>0aee171d-7612-47cd-9461-a4c9d83537f0</ActivityId>
  <ErrorCode>-2147220937</ErrorCode>
  <ErrorDetails xmlns:d2p1="http://schemas.datacontract.org/2004/07/System.Collections.Generic">
    <KeyValuePairOfstringanyType>
      <d2p1:key>ApiExceptionSourceKey</d2p1:key>
      <d2p1:value xmlns:d4p1="http://www.w3.org/2001/XMLSchema" i:type="d4p1:string">Plugin/Microsoft.Crm.WebServices.ImportXmlService</d2p1:value>
    </KeyValuePairOfstringanyType>
    <KeyValuePairOfstringanyType>
      <d2p1:key>ApiSourceActivityKey</d2p1:key>
      <d2p1:value xmlns:d4p1="http://www.w3.org/2001/XMLSchema" i:type="d4p1:string"></d2p1:value>
    </KeyValuePairOfstringanyType>
    <KeyValuePairOfstringanyType>
      <d2p1:key>ApiExceptionOwnerKey</d2p1:key>
      <d2p1:value xmlns:d4p1="http://www.w3.org/2001/XMLSchema" i:type="d4p1:string">OwnershipNotFoundInExceptionOwnerSettings</d2p1:value>
    </KeyValuePairOfstringanyType>
    <KeyValuePairOfstringanyType>
      <d2p1:key>ApiOriginalExceptionKey</d2p1:key>
      <d2p1:value xmlns:d4p1="http://www.w3.org/2001/XMLSchema" i:type="d4p1:string">Microsoft.Crm.BusinessEntities.CrmDuplicateRecordException: Cannot insert duplicate key exception when executing non-query: System.Data.SqlClient.SqlCommand Exception: System.Data.SqlClient.SqlException (0x80131904): Violation of UNIQUE KEY constraint 'ndx_envvarval_schemaname'. Cannot insert duplicate key in object 'dbo.EnvironmentVariableValueBase'. The duplicate key value is (bis_CapexRegisterAppID, 0, Jan  1 1900 12:00AM).
The statement has been terminated.
at Microsoft.Crm.CrmDbConnection.HandleExecutionExceptionAbstract(Exception exception, String commandText, IDbConnection connection, IDbCommand command, IDbTransaction transaction, Boolean isOpen, Int32 commandTimeout, IConnectionRetryPolicy localRetryPolicy, Int32&amp; retryCount)
at Microsoft.Crm.CrmDbConnection.InternalExecuteWithRetryAbstract[TResult](Func`1 executeMethod, IDbCommand command, IConnectionRetryPolicy retryPolicy)
at Microsoft.Crm.CrmDbConnection.&lt; &gt; c__DisplayClass229_0.&lt; InternalExecuteNonQuery&gt; b__0()
at Microsoft.PowerApps.CoreFramework.ActivityLoggerExtensions.Execute[TResult](ILogger logger, EventId eventId, ActivityType activityType, Func`1 func, IEnumerable`1 additionalCustomProperties)
at Microsoft.Xrm.Telemetry.XrmTelemetryExtensions.Execute[TResult](ILogger logger, XrmTelemetryActivityType activityType, Func`1 func)
at Microsoft.Crm.Platform.Server.DataEngine.SqlDataAccessManager.ExecuteNonQuery(IDbCommand command, ISqlExecutionContext context, Int32 sourceLineNumber, String memberName, String sourceFilePath)
at Microsoft.Crm.BusinessEntities.BusinessProcessObject.ExecuteNonQueryAbstract(IDbCommand command, ISqlExecutionContext context)
at Microsoft.Crm.BusinessEntities.BusinessProcessObject.ExecuteNonQueryAbstract(IDbCommand command, ExecutionContext context)
at Microsoft.Crm.Platform.SolutionAwareComponents.UpdateExecutor.ExecuteUpdate(IDbCommand command, ExecutionContext context)
ClientConnectionId:cc361d21-7b46-4ea6-93c8-7d9bf66cd26c
Error Number:2627, State:1, Class:14
ClientConnectionId before routing:8fa74187-a45d-4cd8-8464-18865794837b
Routing Destination:f2506fb14b87.tr369.australiasoutheast1-a.worker.database.windows.net, 11001 ---&gt; Microsoft.Crm.BusinessEntities.CrmDuplicateRecordException: Cannot insert duplicate key exception when executing non-query: System.Data.SqlClient.SqlCommand Exception: System.Data.SqlClient.SqlException (0x80131904): Violation of UNIQUE KEY constraint 'ndx_envvarval_schemaname'. Cannot insert duplicate key in object 'dbo.EnvironmentVariableValueBase'. The duplicate key value is (bis_CapexRegisterAppID, 0, Jan  1 1900 12:00AM).
The statement has been terminated.
at Microsoft.Crm.CrmDbConnection.HandleExecutionExceptionAbstract(Exception exception, String commandText, IDbConnection connection, IDbCommand command, IDbTransaction transaction, Boolean isOpen, Int32 commandTimeout, IConnectionRetryPolicy localRetryPolicy, Int32&amp; retryCount)
at Microsoft.Crm.CrmDbConnection.InternalExecuteWithRetryAbstract[TResult](Func`1 executeMethod, IDbCommand command, IConnectionRetryPolicy retryPolicy)
at Microsoft.Crm.CrmDbConnection.&lt; &gt; c__DisplayClass229_0.&lt; InternalExecuteNonQuery&gt; b__0()
at Microsoft.PowerApps.CoreFramework.ActivityLoggerExtensions.Execute[TResult](ILogger logger, EventId eventId, ActivityType activityType, Func`1 func, IEnumerable`1 additionalCustomProperties)
at Microsoft.Xrm.Telemetry.XrmTelemetryExtensions.Execute[TResult](ILogger logger, XrmTelemetryActivityType activityType, Func`1 func)
at Microsoft.Crm.Platform.Server.DataEngine.SqlDataAccessManager.ExecuteNonQuery(IDbCommand command, ISqlExecutionContext context, Int32 sourceLineNumber, String memberName, String sourceFilePath)
at Microsoft.Crm.BusinessEntities.BusinessProcessObject.ExecuteNonQueryAbstract(IDbCommand command, ISqlExecutionContext context)
at Microsoft.Crm.BusinessEntities.BusinessProcessObject.ExecuteNonQueryAbstract(IDbCommand command, ExecutionContext context)
at Microsoft.Crm.Platform.SolutionAwareComponents.UpdateExecutor.ExecuteUpdate(IDbCommand command, ExecutionContext context)
ClientConnectionId:cc361d21-7b46-4ea6-93c8-7d9bf66cd26c
Error Number:2627, State:1, Class:14
ClientConnectionId before routing:8fa74187-a45d-4cd8-8464-18865794837b
Routing Destination:f2506fb14b87.tr369.australiasoutheast1-a.worker.database.windows.net, 11001
at Microsoft.Crm.Platform.SolutionAwareComponents.UpdateExecutor.ExecuteUpdate(IDbCommand command, ExecutionContext context)
at Microsoft.Crm.Platform.SolutionAwareComponents.UpdateFullComponentInstanceAction.Execute(ExecutionContext context)
at Microsoft.Crm.Platform.SolutionAwareComponents.ComponentStateTransition.&lt; &gt; c__DisplayClass4_0.&lt; Execute&gt; b__1()
at Microsoft.PowerApps.CoreFramework.ActivityLoggerExtensions.Execute(ILogger logger, EventId eventId, ActivityType activityType, Action action, IEnumerable`1 additionalCustomProperties)
at Microsoft.Xrm.Telemetry.XrmTelemetryExtensions.Execute(ILogger logger, XrmTelemetryActivityType activityType, Action action)
at Microsoft.Crm.Platform.SolutionAwareComponents.ComponentStateTransition.Execute(Int32 componentType, Guid objectId, Object operationParameter, BusinessComponentState currentState, ExecutionContext context)
at Microsoft.Crm.BusinessEntities.BusinessProcessObject.DoUpdate(IBusinessEntity entity, FilterExpression filter, ExecutionContext context)
at Microsoft.Crm.BusinessEntities.BusinessProcessObject.&lt; &gt; c__DisplayClass271_0.&lt; UpdateWithPipelineAndExtensions&gt; b__0()
at Microsoft.PowerApps.CoreFramework.ActivityLoggerExtensions.Execute(ILogger logger, EventId eventId, ActivityType activityType, Action action, IEnumerable`1 additionalCustomProperties)
at Microsoft.Xrm.Telemetry.XrmTelemetryExtensions.Execute(ILogger logger, XrmTelemetryActivityType activityType, Action action)
at Microsoft.Crm.BusinessEntities.BusinessProcessObject.&lt; &gt; c__DisplayClass257_0.&lt; Update&gt; b__0()
at Microsoft.PowerApps.CoreFramework.ActivityLoggerExtensions.Execute(ILogger logger, EventId eventId, ActivityType activityType, Action action, IEnumerable`1 additionalCustomProperties)
at Microsoft.Xrm.Telemetry.XrmTelemetryExtensions.Execute(ILogger logger, XrmTelemetryActivityType activityType, Action action)
at Microsoft.Crm.Tools.ImportExportPublish.GenericImportHandler.ImportComponent(IEntity businessDataEntity)
at Microsoft.Crm.ObjectModel.SerializedSolutionOperation.ExecuteSolutionOperation(LinkedList`1 batches, Action`1 solutionOperation)
at Microsoft.Crm.ObjectModel.SolutionSorter.BuildAndExecuteBatches(IList`1 entities, Action`1 solutionOperation, Action`2 removeLinkAction)
at Microsoft.Crm.Tools.ImportExportPublish.GenericImportHandler.RunImport(List`1 componentsToImport, XmlNode rootNode, Guid parentInstanceId, String currentFilePath)
at Microsoft.Crm.Tools.ImportExportPublish.GenericImportHandler.ImportItem()
at Microsoft.Crm.Tools.ImportExportPublish.ImportHandler.ImportInternal()
at Microsoft.PowerApps.CoreFramework.ActivityLoggerExtensions.Execute(ILogger logger, EventId eventId, ActivityType activityType, Action action, IEnumerable`1 additionalCustomProperties)
at Microsoft.Xrm.Telemetry.XrmTelemetryExtensions.Execute(ILogger logger, XrmTelemetryActivityType activityType, Action action)
at Microsoft.Crm.Tools.ImportExportPublish.RootImportHandler.ImportAndUpdateProgress(ImportHandler ih)
at Microsoft.Crm.Tools.ImportExportPublish.RootImportHandler.ProcessHandlers(String[] importEntities, Hashtable ht, String&amp; curPath, ImportHandler&amp; ihForCurrentPath, CounterList listCounters)
at Microsoft.Crm.Tools.ImportExportPublish.RootImportHandler.RunImport(String[] importEntities)
at Microsoft.Crm.Tools.ImportExportPublish.ImportXml.RunImport(String[] importEntities)
at Microsoft.Crm.Tools.ImportExportPublish.ImportXml.RunImport()
at Microsoft.Crm.Tools.ImportExportPublish.ImportXml.RunImportInternal(Boolean withSolutionManifest)
at Microsoft.Crm.Tools.ImportExportPublish.ImportXml.&lt; &gt; c__DisplayClass144_0.&lt; RunImport&gt; b__0()
at Microsoft.PowerApps.CoreFramework.ActivityLoggerExtensions.Execute(ILogger logger, EventId eventId, ActivityType activityType, Action action, IEnumerable`1 additionalCustomProperties)
at Microsoft.Xrm.Telemetry.XrmTelemetryExtensions.Execute(ILogger logger, XrmTelemetryActivityType activityType, Action action)
at Microsoft.Crm.WebServices.ImportXmlService.ImportSolutionSkipCapableInternal(Boolean overwriteUnmanagedCustomizations, Boolean publishWorkflows, Byte[] customizationFile, Guid importJobId, Boolean convertToManaged, Boolean skipProductUpdateDependencies, Boolean holdingSolution, ExecutionContext context, Boolean skipQueueRibbonJob, LayerDesiredOrder layerDesiredOrder, Boolean asyncRibbonProcessing, EntityCollection componentParameters, Boolean isTemplateMode, String templateSuffix, SolutionParameters solutionParameters, String templateDisplayNamePrefix)
at Microsoft.Crm.WebServices.ImportXmlService.ImportSolutionSkipCapable(Boolean overwriteUnmanagedCustomizations, Boolean publishWorkflows, Byte[] customizationFile, Guid importJobId, Boolean convertToManaged, Boolean skipProductUpdateDependencies, Boolean holdingSolution, ExecutionContext context, Boolean skipQueueRibbonJob, LayerDesiredOrder layerDesiredOrder, Boolean asyncRibbonProcessing, EntityCollection componentParameters, Boolean isTemplateMode, String templateSuffix, SolutionParameters solutionParameters, String templateDisplayNamePrefix)
-- - End of inner exception stack trace ---
at Microsoft.Crm.Extensibility.VersionedPluginProxyStepBase.Execute(PipelineExecutionContext context)
at Microsoft.Crm.Extensibility.PipelineInstrumentationHelper.Execute(Boolean instrumentationEnabled, String stopwatchName, ExecuteWithInstrumentation action, PipelineExecutionContext context)
at Microsoft.Crm.Extensibility.Pipeline.&lt; &gt; c__DisplayClass5_0.&lt; RunStep&gt; b__0()</d2p1:value>
    </KeyValuePairOfstringanyType>
    <KeyValuePairOfstringanyType>
      <d2p1:key>ApiStepKey</d2p1:key>
      <d2p1:value xmlns:d4p1="http://schemas.microsoft.com/2003/10/Serialization/" i:type="d4p1:guid">1b830950-e106-4ee1-b3fd-d348cb65dc8d</d2p1:value>
    </KeyValuePairOfstringanyType>
    <KeyValuePairOfstringanyType>
      <d2p1:key>ApiDepthKey</d2p1:key>
      <d2p1:value xmlns:d4p1="http://www.w3.org/2001/XMLSchema" i:type="d4p1:int">2</d2p1:value>
    </KeyValuePairOfstringanyType>
    <KeyValuePairOfstringanyType>
      <d2p1:key>ApiActivityIdKey</d2p1:key>
      <d2p1:value xmlns:d4p1="http://schemas.microsoft.com/2003/10/Serialization/" i:type="d4p1:guid">0aee171d-7612-47cd-9461-a4c9d83537f0</d2p1:value>
    </KeyValuePairOfstringanyType>
    <KeyValuePairOfstringanyType>
      <d2p1:key>ApiPluginSolutionNameKey</d2p1:key>
      <d2p1:value xmlns:d4p1="http://www.w3.org/2001/XMLSchema" i:type="d4p1:string">System</d2p1:value>
    </KeyValuePairOfstringanyType>
    <KeyValuePairOfstringanyType>
      <d2p1:key>ApiStepSolutionNameKey</d2p1:key>
      <d2p1:value xmlns:d4p1="http://www.w3.org/2001/XMLSchema" i:type="d4p1:string">System</d2p1:value>
    </KeyValuePairOfstringanyType>
    <KeyValuePairOfstringanyType>
      <d2p1:key>ApiExceptionCategory</d2p1:key>
      <d2p1:value xmlns:d4p1="http://www.w3.org/2001/XMLSchema" i:type="d4p1:string">ClientError</d2p1:value>
    </KeyValuePairOfstringanyType>
    <KeyValuePairOfstringanyType>
      <d2p1:key>ApiExceptionMessageName</d2p1:key>
      <d2p1:value xmlns:d4p1="http://www.w3.org/2001/XMLSchema" i:type="d4p1:string">DuplicateRecord</d2p1:value>
    </KeyValuePairOfstringanyType>
    <KeyValuePairOfstringanyType>
      <d2p1:key>ApiExceptionHttpStatusCode</d2p1:key>
      <d2p1:value xmlns:d4p1="http://www.w3.org/2001/XMLSchema" i:type="d4p1:int">412</d2p1:value>
    </KeyValuePairOfstringanyType>
    <KeyValuePairOfstringanyType>
      <d2p1:key>0</d2p1:key>
      <d2p1:value xmlns:d4p1="http://www.w3.org/2001/XMLSchema" i:type="d4p1:string"></d2p1:value>
    </KeyValuePairOfstringanyType>
    <KeyValuePairOfstringanyType>
      <d2p1:key>HasRetried</d2p1:key>
      <d2p1:value xmlns:d4p1="http://www.w3.org/2001/XMLSchema" i:type="d4p1:string">True</d2p1:value>
    </KeyValuePairOfstringanyType>
    <KeyValuePairOfstringanyType>
      <d2p1:key>RetryExecutionStatus</d2p1:key>
      <d2p1:value xmlns:d4p1="http://www.w3.org/2001/XMLSchema" i:type="d4p1:string">Microsoft.Crm.Core.Extensions.Retry.RetryExecutionStatus</d2p1:value>
    </KeyValuePairOfstringanyType>
  </ErrorDetails>
  <HelpLink>http://go.microsoft.com/fwlink/?LinkID=398563&amp; error=Microsoft.Crm.CrmException%3a80040237&amp; client=platform</HelpLink>
  <Message>Cannot insert duplicate key exception when executing non-query: System.Data.SqlClient.SqlCommand Exception: System.Data.SqlClient.SqlException (0x80131904): Violation of UNIQUE KEY constraint 'ndx_envvarval_schemaname'. Cannot insert duplicate key in object 'dbo.EnvironmentVariableValueBase'. The duplicate key value is (bis_CapexRegisterAppID, 0, Jan  1 1900 12:00AM).
The statement has been terminated.
at Microsoft.Crm.CrmDbConnection.HandleExecutionExceptionAbstract(Exception exception, String commandText, IDbConnection connection, IDbCommand command, IDbTransaction transaction, Boolean isOpen, Int32 commandTimeout, IConnectionRetryPolicy localRetryPolicy, Int32&amp; retryCount)
at Microsoft.Crm.CrmDbConnection.InternalExecuteWithRetryAbstract[TResult](Func`1 executeMethod, IDbCommand command, IConnectionRetryPolicy retryPolicy)
at Microsoft.Crm.CrmDbConnection.&lt; &gt; c__DisplayClass229_0.&lt; InternalExecuteNonQuery&gt; b__0()
at Microsoft.PowerApps.CoreFramework.ActivityLoggerExtensions.Execute[TResult](ILogger logger, EventId eventId, ActivityType activityType, Func`1 func, IEnumerable`1 additionalCustomProperties)
at Microsoft.Xrm.Telemetry.XrmTelemetryExtensions.Execute[TResult](ILogger logger, XrmTelemetryActivityType activityType, Func`1 func)
at Microsoft.Crm.Platform.Server.DataEngine.SqlDataAccessManager.ExecuteNonQuery(IDbCommand command, ISqlExecutionContext context, Int32 sourceLineNumber, String memberName, String sourceFilePath)
at Microsoft.Crm.BusinessEntities.BusinessProcessObject.ExecuteNonQueryAbstract(IDbCommand command, ISqlExecutionContext context)
at Microsoft.Crm.BusinessEntities.BusinessProcessObject.ExecuteNonQueryAbstract(IDbCommand command, ExecutionContext context)
at Microsoft.Crm.Platform.SolutionAwareComponents.UpdateExecutor.ExecuteUpdate(IDbCommand command, ExecutionContext context)
ClientConnectionId:cc361d21-7b46-4ea6-93c8-7d9bf66cd26c
Error Number:2627, State:1, Class:14
ClientConnectionId before routing:8fa74187-a45d-4cd8-8464-18865794837b
Routing Destination:f2506fb14b87.tr369.australiasoutheast1-a.worker.database.windows.net, 11001</Message>
  <Timestamp>2022-11-16T05:31:56.7717362Z</Timestamp>
  <ExceptionRetriable>false</ExceptionRetriable>
  <ExceptionSource i:nil="true" />
  <InnerFault>
    <ActivityId>0aee171d-7612-47cd-9461-a4c9d83537f0</ActivityId>
    <ErrorCode>-2147220937</ErrorCode>
    <ErrorDetails xmlns:d3p1="http://schemas.datacontract.org/2004/07/System.Collections.Generic" />
    <HelpLink i:nil="true" />
    <Message>Cannot insert duplicate key exception when executing non-query: System.Data.SqlClient.SqlCommand Exception: System.Data.SqlClient.SqlException (0x80131904): Violation of UNIQUE KEY constraint 'ndx_envvarval_schemaname'. Cannot insert duplicate key in object 'dbo.EnvironmentVariableValueBase'. The duplicate key value is (bis_CapexRegisterAppID, 0, Jan  1 1900 12:00AM).
The statement has been terminated.
at Microsoft.Crm.CrmDbConnection.HandleExecutionExceptionAbstract(Exception exception, String commandText, IDbConnection connection, IDbCommand command, IDbTransaction transaction, Boolean isOpen, Int32 commandTimeout, IConnectionRetryPolicy localRetryPolicy, Int32&amp; retryCount)
at Microsoft.Crm.CrmDbConnection.InternalExecuteWithRetryAbstract[TResult](Func`1 executeMethod, IDbCommand command, IConnectionRetryPolicy retryPolicy)
at Microsoft.Crm.CrmDbConnection.&lt; &gt; c__DisplayClass229_0.&lt; InternalExecuteNonQuery&gt; b__0()
at Microsoft.PowerApps.CoreFramework.ActivityLoggerExtensions.Execute[TResult](ILogger logger, EventId eventId, ActivityType activityType, Func`1 func, IEnumerable`1 additionalCustomProperties)
at Microsoft.Xrm.Telemetry.XrmTelemetryExtensions.Execute[TResult](ILogger logger, XrmTelemetryActivityType activityType, Func`1 func)
at Microsoft.Crm.Platform.Server.DataEngine.SqlDataAccessManager.ExecuteNonQuery(IDbCommand command, ISqlExecutionContext context, Int32 sourceLineNumber, String memberName, String sourceFilePath)
at Microsoft.Crm.BusinessEntities.BusinessProcessObject.ExecuteNonQueryAbstract(IDbCommand command, ISqlExecutionContext context)
at Microsoft.Crm.BusinessEntities.BusinessProcessObject.ExecuteNonQueryAbstract(IDbCommand command, ExecutionContext context)
at Microsoft.Crm.Platform.SolutionAwareComponents.UpdateExecutor.ExecuteUpdate(IDbCommand command, ExecutionContext context)
ClientConnectionId:cc361d21-7b46-4ea6-93c8-7d9bf66cd26c
Error Number:2627, State:1, Class:14
ClientConnectionId before routing:8fa74187-a45d-4cd8-8464-18865794837b
Routing Destination:f2506fb14b87.tr369.australiasoutheast1-a.worker.database.windows.net, 11001</Message>
    <Timestamp>2022-11-16T05:31:56.7717362Z</Timestamp>
    <ExceptionRetriable>false</ExceptionRetriable>
    <ExceptionSource i:nil="true" />
    <InnerFault i:nil="true" />
    <OriginalException i:nil="true" />
    <TraceText i:nil="true" />
  </InnerFault>
  <OriginalException i:nil="true" />
  <TraceText i:nil="true" />
</OrganizationServiceFault>
mikefactorial commented 1 year ago

@LukasSchauffele would you mind reposting this on https://github.com/microsoft/powerplatform-build-tools? In the meantime it should be easy enough to delete environment variable values in the downstream environment using the deploy-solution-import-pre-hook.yml hook. Some pseudo code below. NOTE: This hasn't been tested or validated writing from memory.

$config = Get-Content '$(DeploymentSettingsPath)' | ConvertFrom-Json
. "$env:POWERSHELLPATH/load-save-pipeline-parameters.ps1"
$parameters = Read-Pipeline-Parameters "$(Agent.BuildDirectory)\deploy-pipeline-parameters.json"

. "$env:POWERSHELLPATH/dataverse-webapi-functions.ps1"
$dataverseHost = Get-HostFromUrl "$parameters.serviceConnectionUrl"

foreach ($var in $config.EnvironmentVariables) { 
    $odataQuery = 'environmentvariablevalues?$filter=schemaname eq ''' + $var.LogicalName + '''&$select=environmentvariablevalueid'
    $response = Invoke-DataverseHttpGet $(SpnToken) $dataverseHost $odataQuery
    foreach ($varValue in $response ) { 

       #Call delete against the API for each $varValue.environmentvariablevalueid
       #https://learn.microsoft.com/en-us/power-apps/developer/data-platform/webapi/update-delete-entities-using-web-api#basic-delete
    }

}
RajeevPentyala commented 1 year ago

@LukasSchauffele Looks like issue has been fixed at platform level and I could not reproduce this issue. Can you retest and let us know the outcome?

FYI @mikefactorial

CoEStarterKitBot commented 1 year ago

@LukasSchauffele This has been fixed in the latest release. Please install the latest version of the toolkit following the instructions for installing updates. Note that if you do not remove the unmanaged layers as described there you will not receive updates from us.

LukasSchauffele commented 1 year ago

I know it's been a while since I first opened this issue but I wanted to share my working solution in case someone finds it useful.

First, we need to update the extension point as described in the documentation. This runs before deployment so update the Pipelines/Templates/Hooks/deploy-solution-pre-hook.yml file and add a path to the new file:

# The step(s) below can be replaced with your custom steps. You must have at least one step in your hook. The step(s) below demonstrate examples of custom hooks.
# Be sure when upgrading to the latest pipelines that you don't overwrite your custom hooks with these samples hooks
# Parameters passed to the parent pipeline (deploy-Solution.yml) can be accessed via the following command.
# . "$env:POWERSHELLPATH/load-save-pipeline-parameters.ps1"
# $parameters = Read-Pipeline-Parameters "$(Agent.BuildDirectory)\deploy-pipeline-parameters.json"
steps:
  - script: echo Deploy Solution Pre Hook
    displayName: 'Deploy Solution Pre Hook'
    enabled: false

  - template: CustomTemplates/delete-environment-variables.yml

Second, create the Pipelines/Templates/Hooks/CustomTemplates/delete-environment-variables.yml file. We first query the environment variable definition before we delete the environment variable values:

# Delete environment variables
steps:
- pwsh: |
    $config = Get-Content '$(DeploymentSettingsPath)' | ConvertFrom-Json
    . "$env:POWERSHELLPATH/load-save-pipeline-parameters.ps1"
    $parameters = Read-Pipeline-Parameters "$(Agent.BuildDirectory)\deploy-pipeline-parameters.json"
    . "$env:POWERSHELLPATH/dataverse-webapi-functions.ps1"
    $dataverseHost = Get-HostFromUrl "$parameters.serviceConnectionUrl"
    foreach ($envVar in $config.EnvironmentVariables) {
        # Get Environment Variable Definition 
        $odataQuery = 'environmentvariabledefinitions?$filter=schemaname eq ''' + $envVar.SchemaName + '''&$select=environmentvariabledefinitionid'
        $response = Invoke-DataverseHttpGet $(SpnToken) $dataverseHost $odataQuery
        foreach ($envVarDefinition in $response.value ) {
            # Get Environment Variable Values 
            $odataQuery = 'environmentvariablevalues?$filter=_environmentvariabledefinitionid_value eq ''' + $envVarDefinition.environmentvariabledefinitionid + '''&$select=environmentvariablevalueid'
            $response = Invoke-DataverseHttpGet $(SpnToken) $dataverseHost $odataQuery
            foreach ($envVarValue in $response.value) {
                $odataQuery = 'environmentvariablevalues(' + $envVarValue.environmentvariablevalueid + ')'
                Invoke-DataverseHttpDelete $(SpnToken) $dataverseHost $odataQuery
            } 
        }
    }
  displayName: 'Delete Environment Variables'

Finally, the function Invoke-DataverseHttpDelete doesn't exist yet and we need to create it. Add the below snippet at the end of PowerShell/dataverse-webapi-functions.ps1 @mikefactorial - I would be happy to create a PR for this if you think it could be useful.

<#
This function invokes Dataverse web api DELETE.
#>
function Invoke-DataverseHttpDelete {
    param (
        [Parameter(Mandatory)] [String]$token,
        [Parameter(Mandatory)] [String]$dataverseHost,
        [Parameter(Mandatory)] [String]$requestUrlRemainder
    )
    $headers = Set-DefaultHeaders $token
    $requestUrl = Set-RequestUrl $dataverseHost $requestUrlRemainder
    $response = Invoke-RestMethod $requestUrl -Method 'DELETE' -Headers $headers
    return $response
}