search

microsoft / azure-pipelines-tasks

Tasks for Azure Pipelines
https://aka.ms/tfbuild
MIT License
3.47k stars 2.6k forks source link

AzureRmWebAppDeploy: Xml Variable Substitution does not work with connectionStrings #3495

Closed mikeycooper closed 7 years ago

mikeycooper commented 7 years ago

We are using the 3.x preview AzureRmWebAppDeploy task. All app settings are replaced as expected. However, connection strings are not substituted. Looking at webdeployment-common/xmlvariablesubstitutionutility.ts, it looks like it's only looking for attributes named key and value, whereas the connectionStrings section uses name and connectionString.

mvvsubbu commented 7 years ago

Can you share your sample web.config snippet and debug logs by setting system.debug to true

mikeycooper commented 7 years ago

Snippets of log where it's reading out the connection string from our variable called RavenReadModels and then do the substitution on connectionStrings:

2017-01-27T10:05:06.8141117Z ##[debug]XmlVariableSubstitution=true
2017-01-27T10:05:12.3187894Z ##[debug]RavenReadModels=********
2017-01-27T10:05:12.3811443Z ##[debug]Initiated variable substitution in config file : C:/agent/_work/r10/a/temp_web_package_2733884963672608/Web.config
2017-01-27T10:05:12.3811443Z ##[debug]Detecting file encoding using BOM
2017-01-27T10:05:12.3811443Z ##[debug]Unable to detect File encoding using BOM
2017-01-27T10:05:12.3811443Z ##[debug]Detecting file encoding without BOM
2017-01-27T10:05:12.3811443Z ##[debug]Unable to find node with tag 'applicationSettings' in provided xml file.
2017-01-27T10:05:12.3811443Z ##[debug]Processing substitution for xml node : appSettings
2017-01-27T10:05:12.3811443Z ##[debug]Processing substitution for xml node : connectionStrings
2017-01-27T10:05:12.3811443Z ##[debug]Unable to find node with tag 'configSections' in provided xml file.

Connection string snippet of web.config:

  <connectionStrings>
    <add name="RavenReadModels" connectionString="Url = http://sample:1234;Database=abc" />
  </connectionStrings>

RavenReadModels should be substituted with the value of the variable, but because xmlvariablesubstitutionutility is only looking for a node with attributes named key and value, it never finds attributes on connection string nodes.

mvvsubbu commented 7 years ago

RavenReadModels will be substituted with the value of "name" release variable. Isn't it happening like that?

mikeycooper commented 7 years ago

The connection string node with a name of RavenReadModels should have its connectionString attribute substituted with the value of the RavenReadModels environment variable. This is not happening. Even the scenario that you're describing would not happen, because xmlvariablesubstitutionutility.ts only looks for key and value pairs. When processing the connectionStrings section, it needs to look for name and connectionString pairs.

mvvsubbu commented 7 years ago

XML variable substitution supports the following two formats, please let us if it doesn't as described below.

Format 1:

If release variable has ValueOfWebsiteName = Contoso defined the following transformation happens.

<tag key=”WebsiteName” value=”ValueOfWebsiteName” /> --> <tag key=”WebsiteName” value=”Contoso” />

Format 2:

If release variable has connectionStrinng = localhost\SQLSERVER, websiteName = Contoso, name = xyz defined the following transformation happens.

<tag name="NodeName" websiteName=”ValueOfWebsiteName” connectionStrinng=”valueOfConnectionString” /> -->    <tag name="xyz" websiteName=”Contoso” connectionStrinng=”localhost\SQLSERVER” />
mikeycooper commented 7 years ago

I tested with a variable named connectionString and that did replace it. I'm not sure how I'm supposed to replace different connection strings though...

<connectionStrings>
  <add name="RavenReadModels" connectionString="Url = http://localhost:123;Database=SampleDev" />
  <add name="LoggingDb" connectionString="Url = http://localhost:999;Database=LogsDev" />
</connectionStrings>

There seems to be no way to set it so that RavenReadModels has a connection string of Url = http://prod;Database=Prod and LoggingDb has a connection string of http://log;Database=ProdLog. The current mechanism only supports 1 connection string?

mvvsubbu commented 7 years ago

Can you try using format 1 for now. Looping our PM to note feedback on this.

mvvsubbu commented 7 years ago

@mikeycooper

We have added this enhancement in PR 3538 (https://github.com/Microsoft/vsts-tasks/pull/3538). This will be available in next deployment for your to try out which will be 3 - 4 weeks from now.

I am closing the issue, feel free to re-open if there is some issue after this enhancement as well.

mikeycooper commented 7 years ago

We have v3.0.7 of the task now, but it looks like it's HTML-encoding the variable during substitution. The connection string being set is:

Url=https://primary;Database=db1;ApiKey=11111111-1111-1111-1111-111111111111;Failover = {Url:&apos;https://secondary&apos;, ApiKey:&apos;11111111-1111-1111-1111-111111111111&apos;}

when it should be:

Url=https://primary;Database=db1;ApiKey=11111111-1111-1111-1111-111111111111;Failover = {Url:'https://secondary', ApiKey:'11111111-1111-1111-1111-111111111111'}

vincent1173 commented 7 years ago

@mikeycooper , the above changes failed your deployment ? If so, Can you please share the details from where you viewed the changes (temp folder from Agent Build / your app control flow)?

Can you please check whether the web.config has the expected changes from Kudu Console ? Follow the below steps to view file in Kudu console. open https://.scm.azurewebsites.net

  1. In the header menu, select 'Debug Console' -> 'cmd'
  2. cd into D:/home/site/wwwroot
  3. Click Edit option (to the left side) of web.config image
mikeycooper commented 7 years ago

Deployment succeeded. Just the transformed and substituted value is incorrect (and possibly invalid) because the apostrophes are being replaced by &apos;.

I confirmed it by looking at the web.config inside the App Service Editor. There, I see the connection string with &apos; in it.

vincent1173 commented 7 years ago

@mikeycooper , thanks for the info. We found the root cause for this issue and working on it.

vincent1173 commented 7 years ago

@mikeycooper , As a workaround , can you please provide the values without single quotes ? (If quotes doesn't matter for your connection string) Other possible workaround is to set the value in Web..config and enabling XML Transformation.

BerendWouters commented 7 years ago

We encounter the same issue.

vincent1173 commented 7 years ago

@BerendWouters , thanks for reporting the issue. Can you please share config details where the issue is encountered ? In this way, we can be sure that it is due to encoding issue.

BerendWouters commented 7 years ago

We transform the web.release.config to web.config, and the perform some variable substitutions. All the other substitutions are successful. 

<connectionStrings xdt:Transform="Replace">
    <add name="AdminConnection" connectionString="{{AdminConnection}}" providerName="System.Data.SqlClient"/>
  </connectionStrings>
vincent1173 commented 7 years ago

@BerendWouters , Can you please tell us what is the expected and actual value substituted ? Are you providing the RD value in below format for XML variable substitution ? image

Please share the debug logs by enabling 'system.debug=true' to RM_Customer_Queries@Microsoft.com

BerendWouters commented 7 years ago

I am indeed providing the AdminConnection name and value parameters accordingly, in that format. Expected: the correct connectionString, but the actual result is still {{AdminConnection}}

vincent1173 commented 7 years ago

Please share your debug logs to RM_Customer_Queries@Microsoft.com.

vincent1173 commented 7 years ago

@mikeycooper , the issue is resolved and the fix will be available in upcoming deployments.

Code4Kicks commented 7 years ago

When will this be working? I'm on 3.* and variable substitution is not working. I've added variables to the environment definition but it never changes in the web.config file on release. Please advise, thank you.

vincent1173 commented 7 years ago

@Code4Kicks , this issue is fixed long back. Can you please share the debug logs for the release ?

Code4Kicks commented 7 years ago

Here are the pertinent details I believe. The environment variable is AdminConnection. THere are many more but just included that one. Maybe I'm using it wrong? Thanks for the reply...

2017-06-07T22:04:09.5246737Z ##[section]Starting: Initialize Job 2017-06-07T22:04:09.5266736Z Prepare release directory. 2017-06-07T22:04:09.5276737Z ReleaseId=47, TeamProjectId=xxxxxxxxxxx, ReleaseDefinitionName=Azure Release 2017-06-07T22:04:09.5416736Z Release folder: d:\a\r1\a 2017-06-07T22:04:09.5556732Z Environment variables available are below. Note that these environment variables can be referred to in the task (in the ReleaseDefinition) by replacing "_" with "." e.g. AGENT_NAME environment variable can be referenced using Agent.Name in the ReleaseDefinition: [ADMINCONNECTION] --> [TESTIAN]

2017-06-07T22:04:13.9582253Z ##[section]Starting: Deploy Azure App Service 2017-06-07T22:04:13.9692257Z ============================================================================== 2017-06-07T22:04:13.9692257Z Task : Azure App Service Deploy 2017-06-07T22:04:13.9692257Z Description : Update Azure App Service using Web Deploy / Kudu REST APIs 2017-06-07T22:04:13.9692257Z Version : 3.3.6 2017-06-07T22:04:13.9692257Z Author : Microsoft Corporation 2017-06-07T22:04:13.9692257Z Help : More Information 2017-06-07T22:04:13.9692257Z ============================================================================== 2017-06-07T22:04:15.3810851Z xxxxxxxxxxxxxxx exists true 2017-06-07T22:04:16.0160849Z Got connection details for Azure App Service:'xxxxxxxxxxxxx' 2017-06-07T22:04:16.8120850Z XML variable substitution applied successfully.

vincent1173 commented 7 years ago

@Code4Kicks , Can you please set system.debug to true in variable section and rerun the release ? Please share the full logs from the new release.

Petermo commented 7 years ago

I'm having a similiar issue. Not related to AzureRmWebAppDeploy but to regular IIS WebApp Deploy, however I'm guessing the substitution mechanism is the same. The other variables work just fine, but not the connectionString. Below are the debug log lines that seem to be telling me that replacement of the connectionString worked. (However it didn't after checking the web.config which is actually deployed)

2017-06-20T15:52:15.7653854Z ##[debug]Initiated variable substitution in config file : C:\vstsagent\A1\_work\r1\a\temp_web_package_16421769512817264\Content\d_C\a\1\s\UI\obj\Release\Package\PackageTmp\Web.config
2017-06-20T15:52:15.7653854Z ##[debug]Detecting file encoding using BOM
2017-06-20T15:52:15.7754157Z ##[debug]Unable to find node with tag 'applicationSettings' in provided xml file.
2017-06-20T15:52:15.7754157Z ##[debug]Processing substitution for xml node : appSettings
2017-06-20T15:52:15.7754157Z ##[debug]Updating value for key=ImportLocationwith token_value: CONFIG_FILE_SETTINGS_TOKEN(ImportLocation)
2017-06-20T15:52:15.7754157Z ##[debug]Updating value for key=ExportLocationwith token_value: CONFIG_FILE_SETTINGS_TOKEN(ExportLocation)
2017-06-20T15:52:15.7754157Z ##[debug]Updating value for key=FilesLocationwith token_value: CONFIG_FILE_SETTINGS_TOKEN(FilesLocation)
2017-06-20T15:52:15.7754157Z ##[debug]Updating value for key=TextkernelAddresswith token_value: CONFIG_FILE_SETTINGS_TOKEN(TextkernelAddress)
2017-06-20T15:52:15.7783820Z ##[debug]Processing substitution for xml node : connectionStrings
2017-06-20T15:52:15.7783820Z ##[debug]Detecting file encoding using BOM
2017-06-20T15:52:15.7793835Z ##[debug]Parameters file - Replacing value for name: Dynamics-Web.config Connection String with : PARAM_FILE_REPLACE_TOKEN(Dynamics-Web.config Connection String)
2017-06-20T15:52:15.7803861Z ##[debug]Parameters file - Replacing value for temp_name: PARAM_FILE_REPLACE_TOKEN(Dynamics-Web.config Connection String)
2017-06-20T15:52:15.7833838Z ##[debug]Parameter file C:\vstsagent\A1\_work\r1\a\temp_web_package_16421769512817264\parameters.xml updated.
2017-06-20T15:52:15.7843817Z ##[debug]Substituting connectionString value for name=Dynamics with token_value: CONFIG_FILE_CONN_STRING_TOKEN(Dynamics)
2017-06-20T15:52:15.7843817Z ##[debug]Unable to find node with tag 'configSections' in provided xml file.
2017-06-20T15:52:15.7863815Z ##[debug]Substituting original value in place of temp_name: CONFIG_FILE_SETTINGS_TOKEN(ImportLocation)
2017-06-20T15:52:15.7863815Z ##[debug]Substituting original value in place of temp_name: CONFIG_FILE_SETTINGS_TOKEN(ExportLocation)
2017-06-20T15:52:15.7863815Z ##[debug]Substituting original value in place of temp_name: CONFIG_FILE_SETTINGS_TOKEN(FilesLocation)
2017-06-20T15:52:15.7863815Z ##[debug]Substituting original value in place of temp_name: CONFIG_FILE_SETTINGS_TOKEN(TextkernelAddress)
2017-06-20T15:52:15.7863815Z ##[debug]Substituting original value in place of temp_name: CONFIG_FILE_CONN_STRING_TOKEN(Dynamics)
2017-06-20T15:52:15.7893840Z ##[debug]Config file C:\vstsagent\A1\_work\r1\a\temp_web_package_16421769512817264\Content\d_C\a\1\s\UI\obj\Release\Package\PackageTmp\Web.config updated.
vincent1173 commented 7 years ago

@Petermo , Are you trying to deploy MSBuild package ? If so, ConnectionString attributes in Web.config is parameterized by default. Note that the substitution has no effect on connectionString attributes as the value is overridden during deployment by 'Parameters.xml or 'SetParameters.xml' files. You can disable the auto-parameterization by setting /p:AutoParameterizationWebConfigConnectionStrings=False during MSBuild package generation.

If the issue persists, please share the debug logs for the same.

Petermo commented 7 years ago

@vincentdass that did the trick. Never realised that this would be the order in which variables are substituted. Thanks.