g2vinay
commented
4 years ago We need to automate the way we run tests. The steps to run MSI tests in Azure Web Jobs for the identity component are as follows:
prerequisite tools
- Azure CLI
- Powershell (We will run all the commands mentioned in powershell)
- .NET framework 4.5 or above (installed on the machine)
Azure resources
This test requires instances of these Azure resources:
- Azure Key Vault
- Azure Managed Identity
- with secrets/set and secrets/delete permission for the Key Vault
- Azure App Service
- with system assigned identity and user assigned identity
Initial Setup:
PLEASE NOTE: This setup is Java specific since it installs the Java & Maven environments. Other languages will have a different set up.
Set environment variables to simplify copy-pasting
- RESOURCE_GROUP
- name of an Azure resource group
- must be unique in the Azure subscription
- e.g. 'identity-test-rg'
- PLAN_NAME
- name of the app service plan
- must be unique in the resource group
- e.g. 'identity-test-app-service-plan'
- PLAN_SKU
- the type of the plan
- Choose 'S1', # Allowed values are B1, B2, B3, D1, F1, FREE, P1, P1V2, P2, P2V2, P3, P3V2, S1, S2, S3, SHARED.
- LOCATION
- location of the resources
- e.g. "West US 2"
- APP_NAME_SYSTEM_ASSIGNED
- name of an Azure App Service
- must be unique in the resource group
- e.g. 'identity-test-app-system'
- APP_NAME_USER_ASSIGNED
- name of an Azure App Service
- must be unique in the resource group
- e.g. 'identity-test-app-user'
- MANAGED_IDENTITY_NAME
- 3-128 alphanumeric characters
- must be unique in the resource group
- KEY_VAULT_NAME
- 3-24 alphanumeric characters
- must begin with a letter
- must be globally unique
Create resources
-
In a powershell with Azure CLI, create the resource group. This step is optional, skip if resource group already exists.
az group create -n $RESOURCE_GROUP -l $LOCATION -
Create the app service plan
az appservice plan create -n $PLAN_NAME -g $RESOURCE_GROUP -l $LOCATION --sku $PLAN_SKU -
Create a web app
az webapp create -n $APP_NAME_SYSTEM_ASSIGNED -g $RESOURCE_GROUP --plan $PLAN_NAME -
Enable System assigned identity on the web app.
az webapp identity assign --name $APP_NAME_SYSTEM_ASSIGNED --resource-group $RESOURCE_GROUP -
Create a user identity for the second web app:
az identity create -n $MANAGED_IDENTITY_NAME -g $RESOURCE_GROUP -l $LOCATION -
create the second web app:
az webapp create -n $APP_NAME_USER_ASSIGNED -g $RESOURCE_GROUP --plan $PLAN_NAME -
Assign the user Identity from Step 5 to the second web app.
- Login to portal
- Go to the second web app resource
- Select the Identity tab under Settings,
- Click on User assigned
- Click on Add
- Select your subscription
- Search for the user Identity with name $MANAGED_IDENTITY_NAME and select it from the list.
-
Create a Key Vault:
az keyvault create -n $KEY_VAULT_NAME -g $RESOURCE_GROUP
Get the principal ID for the Web app with system assigned identity:
$principalId=az webapp show -n $APP_NAME_SYSTEM_ASSIGNED -g $RESOURCE_GROUP --query "identity.principalId"
$principalId=$principalId.Replace('"', '')Allow the Web app with system assigned identity to access the key vault secrets:
az keyvault set-policy -n $KEY_VAULT_NAME --object-id $principalId --secret-permissions get listGet the principal ID for the second Web app with user assigned identity:
$principalId=az identity show -n $MANAGED_IDENTITY_NAME -g $RESOURCE_GROUP --query "principalId"
$principalId=$principalId.Replace('"', '')Allow the Web app with user assigned identity to access the key vault secrets:
az keyvault set-policy -n $KEY_VAULT_NAME --object-id $principalId --secret-permissions get listAdd a secret called "secret":
az keyvault secret set --vault-name $KEY_VAULT_NAME -n secret --value "Secret value"Build the executable jar
The executable jar holds the logic to run tests. will be different per language.
- clone azure-sdk-for-java-repo
- Install Build tools
mvn clean install -Dgpg.skip -f eng\code-quality-reports\pom.xml - Install Key vault libraries(paths are relative to the root of the repo)
mvn clean install -Dgpg.skip -Dcheckstyle.skip -Dspotbugs.skip -Drevapi.skip -Dmaven.javadoc.skip=true -DskipTests -f sdk/keyvault/pom.service.xml - Build the jar with dependencies
mvn clean package -Dpackage-with-dependencies -f sdk\e2e\pom.xmlsdk\e2e\target will contain an executable with jar with dependencies.
Package, Deploy and run tests
Run System Assigned Web Job Test
-
Create a directory systemAssignedTest
mkdir systemAssignedTest -
Copy executable jar - azure-e2e-1.0.0-beta.1-jar-with-dependencies.jar from sdk\e2e\target to the systemAssignedTest directory.
-
Setup SystemAssignedTest Executable:
$content="set AZURE_VAULT_URL=https://$KEY_VAULT_NAME.vault.azure.net" $content += "`r`n" $content +="set AZURE_WEBJOBS_TEST_MODE=system" $content += "`r`n" $content +="set AZURE_IDENTITY_TEST_PLATFORM=webjobs" $content += "`r`n" $content +="java -jar azure-e2e-1.0.0-beta.1-jar-with-dependencies.jar" Set-Content -Value $content -Path '<Relative-path-to-systemAssignedTest-directory>\run.bat'
Create system assigned test web job deployment file
$netFrameworkVersion="v<YOUR-DOT-NET-FRAMEWORK-VERSION>"
Add-Type -Path "C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\$netFrameworkVersion\System.IO.Compression.FileSystem.dll"
$systemAssignedWebJobzip = "<OUTPUT_PATH>\systemwebjob.zip"
if(Test-path $systemAssignedWebJobzip) {Remove-item $systemAssignedWebJobzip}
[io.compression.zipfile]::CreateFromDirectory("<path-to-systemAssignedTest-directory>", $systemAssignedWebJobzip)- Build user credentials for deployment
$user = az webapp deployment list-publishing-profiles -n $APP_NAME_SYSTEM_ASSIGNED -g $RESOURCE_GROUP ` --query "[?publishMethod=='MSDeploy'].userName" -o tsv
$pass = az webapp deployment list-publishing-profiles -n $APP_NAME_SYSTEM_ASSIGNED -g $RESOURCE_GROUP `
--query "[?publishMethod=='MSDeploy'].userPWD" -o tsv$creds = "$($user):$($pass)"
$encodedCreds = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes($creds))
$basicAuthValue = "Basic $encodedCreds"Deploy the system assigned web job
$ZipHeaders = @{
Authorization = $basicAuthValue
"Content-Disposition" = "attachment; filename=run.cmd"
}
# upload the system assigned job using the Kudu WebJobs API
Invoke-WebRequest -Uri https://$APP_NAME_SYSTEM_ASSIGNED.scm.azurewebsites.net/api/triggeredwebjobs/Systemassignedjob -Headers $ZipHeaders `
-InFile $systemAssignedWebJobzip -ContentType "application/zip" -Method Put- Run the Web Job Test::
Configure auth header
$Headers = @{ Authorization = $basicAuthValue }
Run the system assigned web job
$resp = Invoke-WebRequest -Uri "https://$APP_NAME_SYSTEM_ASSIGNED.scm.azurewebsites.net/api/triggeredwebjobs/Systemassignedjob/run?arguments=eggs bacon" -Headers $Headers `
-Method Post -ContentType "multipart/form-data"Print the web job output log: Wait for 30 -40 seconds for job to execute, to get the full log. You can run the below command multiple times too, to see the progress of the job.
if ($resp.RawContent -match "\nLocation\: (.+)\n")
{
$historyLocation = $matches[1]
$hist = Invoke-RestMethod -Uri $historyLocation -Headers $Headers -Method Get
# $hist has status, start_time, end_time, duration, error_url etc
# get the logs from output_url
Invoke-RestMethod -Uri $hist.output_url -Headers $Headers -Method Get
}Success Output:
Run User Assigned Web job Test
-
Create a directory userAssignedTest
mkdir userAssignedTest -
Copy executable jar - azure-e2e-1.0.0-beta.1-jar-with-dependencies.jar from sdk\e2e\target to the userAssignedTest directory.
-
Get the client id of the user assigned identity
$clientId=az identity show -n $MANAGED_IDENTITY_NAME -g $RESOURCE_GROUP --query "clientId" $clientId=$clientId.Replace('"', '') -
Setup UserAssignedTest Executable: In powershell, run following:
$content="set AZURE_VAULT_URL=https://$KEY_VAULT_NAME.vault.azure.net" $content += "`r`n" $content +="set AZURE_WEBJOBS_TEST_MODE=user" $content += "`r`n" $content +="set AZURE_CLIENT_ID=$clientId" $content += "`r`n" $content +="set AZURE_IDENTITY_TEST_PLATFORM=webjobs" $content += "`r`n" $content +="java -jar azure-e2e-1.0.0-beta.1-jar-with-dependencies.jar" Set-Content -Value $content -Path '<Relative-Path-To-userAssignedTest-directory>\run.bat'Create user assigned test web job deployment file
$netFrameworkVersion="v<YOUR-DOT-NET-FRAMEWORK-VERSION>" Add-Type -Path "C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\$netFrameworkVersion\System.IO.Compression.FileSystem.dll"
$userAssignedWebJobzip = "
5. Build user credentials for deployment
```powershell
$user = az webapp deployment list-publishing-profiles -n $APP_NAME_USER_ASSIGNED -g $RESOURCE_GROUP `
--query "[?publishMethod=='MSDeploy'].userName" -o tsv$pass = az webapp deployment list-publishing-profiles -n $APP_NAME_USER_ASSIGNED -g $RESOURCE_GROUP `
--query "[?publishMethod=='MSDeploy'].userPWD" -o tsv$creds = "$($user):$($pass)"
$encodedCreds = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes($creds))
$basicAuthValue = "Basic $encodedCreds"Deploy the system assigned web job
$ZipHeaders = @{
Authorization = $basicAuthValue
"Content-Disposition" = "attachment; filename=run.cmd"
}
# upload the user assigned job using the Kudu WebJobs API
Invoke-WebRequest -Uri https://$APP_NAME_USER_ASSIGNED.scm.azurewebsites.net/api/triggeredwebjobs/Userassignedjob -Headers $ZipHeaders `
-InFile $userAssignedWebJobzip -ContentType "application/zip" -Method Put-
Run the User assigned Web Job Test:: Configure auth header
$Headers = @{ Authorization = $basicAuthValue }Run the user assigned web job
$resp = Invoke-WebRequest -Uri "https://$APP_NAME_USER_ASSIGNED.scm.azurewebsites.net/api/triggeredwebjobs/Userassignedjob/run?arguments=eggs bacon" -Headers $Headers ` -Method Post -ContentType "multipart/form-data"
Print the web job output log: Wait for 30 -40 seconds for job to execute, to get the full log. You can run the below command multiple times too, to see the progress of the job.
if ($resp.RawContent -match "\nLocation\: (.+)\n")
{
$historyLocation = $matches[1]
$hist = Invoke-RestMethod -Uri $historyLocation -Headers $Headers -Method Get
# $hist has status, start_time, end_time, duration, error_url etc
# get the logs from output_url
Invoke-RestMethod -Uri $hist.output_url -Headers $Headers -Method Get
}Success output:
XuGuang-Yao
sophiajt
kurtzeborn
joshfree
catalinaperalta
Menghua1
v-jiaodi
github-actions[bot]
v-xuto
Tracking issue for step-by-step instructions for how to manually run e2e tests for Azure WebJobs hosted environments for each of the azure-sdk-for-* languages. This issue will be used by our Vendor team for manually running tests in this scenario, and will be tracked on the EngSys backlog for eventual automation.
Assigning to @g2vinay who will create the initial instructions specific to Java Azure SDKs. Other SDK languages will be appended to this issue as further comments.