Investigate PoweShell live test coverage

[sergey-shandar]

Investigate coverage and health of PowerShell live tests.

[sergey-shandar]

https://github.com/Azure/azure-powershell/tree/preview/src/ResourceManager

[sergey-shandar]

See also https://github.com/Azure/azure-rest-api-specs-tests/issues/18

[sergey-shandar]

Live Test Report for ResourceManager

• AnalysisServices
  • 4 failed
  • 4 passed
• ApiManagement is broken. Some wired error related to PowerShell cmdlet not found.
• Automation
  • 3 failed
  • 62 passed
• Backup is broken. Cannot validate argument on parameter Vault.
• Batch
  • 32 failed
  • 150 passed
• Billing
  • 10 failed
• CDN: 1 of 19. Test failed due to a non-empty string.
• CognitiveServices: 1 of 12. The term New-AzureRmCognitiveServiceAccount is not recognized.
• Compute is broken, requires SUSE:SLES:12-SP1:latest image.
• Consumption: 3 of 9. Assertion failed because expected 10 does not match actual 0
• ContainerRegistry: 1 of 3. The term Get-AzureRmResourceProvider is not recognized.
• DataFactory:
  • 14 failed - Brazil South is not available.
  • 43 passed
  • 2 skipped
• DataLakeAnalytics: 18 failed. PowerShell cmdlet problems.
• DataLakeStore: 14 passed.
• DevTestLab: 5 failed. Could not load file or assembly Newtonsoft.Json, Version=9.0.0.0
• Dns:
  • 42 failed. `Test failed due to a non-empty error stream
  • 1 passed.
• EventHub.
  • 4 failed.
  • 1 passed
• HDInsight
  • 1 skipped
  • 48 passed
• Insights
  • 34 failed. Loading an assembly required for interactive user authentication failed. Make sure assembly Microsoft.IdentityModel.Clients.ActiveDirectory.WindowsForms...
  • 32 passed
• IotHub
  • 1 failed. Loading an assembly required for interactive user authentication failed. Make sure assembly Microsoft.IdentityModel.Clients.ActiveDirectory.WindowsForms...
• KeyVault
  • 10 failed. System.Collections.Generic.KeyNotFoundException.
  • 20 skipped.
  • 17 passed.
• LogicApp
  • 18 failed Expected exception doesn't contain expected text
  • 45 passed
• MachineLearning
  • 8 failed Assertion failed
• Media
  • 1 failed. Non-empty error stream
  • 1 passed.
• Network
  • 1 passed
  • 18 skipped
  • 61 failed Non-empty error stream
  • 1 frezed
• NotificationHubs
  • 3 failed Non-empty error stream
  • 1 passed
• OperationalInsights
  • 12 failed HTTP Status Code: Forbidden Error Message
  • 1 passed
• PowerBIEmbedded
  • 8 passed
• Profile
  • 17 failed
  • 116 passed
• RecoveryServices
  • 1 failed
• RecoveryServices.Backup
  • 25 failed The running command stopped
• RecoveryServices.SiteRecovery
  • 12 failed The running command stopped
• RedisCache
  • 3 failed
  • 9 passed
• Relay
  • 5 failed
  • 1 passed
• Resources
  • 80 failed
  • 27 skipped
  • 59 passed
• Scheduler
  • 8 passed
• ServiceManagement
  • 3 failed Operation returned an invalid status code NotFound
• ServiceBus
  • 6 failed File Not Found ResourceManagerStartup.ps1
  • 1 passed
• ServiceFabric
  • 10 failed KeyNotFound
• SiteRecovery
  • 13 failed Resource Group is not associated with the selected Subscription
• Sql
  • 144 failed Non-empty error stream some tests took more than 16 hours
  • 2 skipper
  • 45 passed
• Storage
  • 3 failed The provided location eastus2euap is not available
  • 9 passed
• StreamAnalytics
  • 1 failed NotFound
• TrafficManager
  • 41 failed
  • 4 passed
• UsageAggregates
  • 1 passed
• Webstites
  • 35 failed
  • 5 skipped

[sergey-shandar]

list.md