[HOW-TO] Run Tests in Azure DevOps Release Pipeline

[mgr1054]

Question

Hey Guys,

I'm trying to run some UI Tests in a AzureDevOps Release Pipeline. I followed the Tutorial in the Wiki but the pipeline keeps throwing errors.

Pipeline error:

2019-11-04T17:22:12.5729017Z [command]C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow\..\..\..\Extensions\TestPlatform\vstest.console.exe "@C:\Users\VssAdministrator\AppData\Local\Temp\tmp4DFA.tmp"
2019-11-04T17:22:13.1264811Z Microsoft (R) Test Execution Command Line Tool Version 15.9.1
2019-11-04T17:22:13.1264994Z Copyright (c) Microsoft Corporation.  All rights reserved.
2019-11-04T17:22:13.3784157Z vstest.console.exe "d:\a\r1\a\UCI_Test\drop\UCI_Test.dll"
2019-11-04T17:22:13.3784743Z /Settings:"d:\a\_temp\huinqc34g4n.runsettings"
2019-11-04T17:22:13.3785172Z /Logger:"trx"
2019-11-04T17:22:13.3789847Z /TestAdapterPath:"d:\a\r1\a"
2019-11-04T17:22:13.3790392Z /Diag:"d:\a\_temp\Attempt--1_og5nwg.diag"
2019-11-04T17:22:19.6167216Z Starting test execution, please wait...
2019-11-04T17:22:19.6172853Z Logging Vstest Diagnostics in file: d:\a\_temp\Attempt--1_og5nwg.diag
2019-11-04T17:24:25.9782566Z Failed   WEBOpenEmployee
2019-11-04T17:24:25.9782833Z Error Message:
2019-11-04T17:24:25.9782881Z  Test method UCI_Test.Web.Human_Resource.OpenEmployee.WEBOpenEmployee threw exception: 
2019-11-04T17:24:25.9783019Z System.NullReferenceException: Object reference not set to an instance of an object.
2019-11-04T17:24:25.9783112Z Stack Trace:
2019-11-04T17:24:25.9783162Z     at Microsoft.Dynamics365.UIAutomation.Api.Navigation.<>c__DisplayClass16_0.<OpenSubArea>b__0(IWebDriver driver)
2019-11-04T17:24:25.9783260Z    at Microsoft.Dynamics365.UIAutomation.Browser.DelegateBrowserCommand`1.ExecuteCommand(IWebDriver driver, Object[] params)
2019-11-04T17:24:25.9783314Z    at Microsoft.Dynamics365.UIAutomation.Browser.BrowserCommand`1.Execute[T1,T2,T3,T4,T5,T6,T7,T8,T9](IWebDriver driver, T1 p1, T2 p2, T3 p3, T4 p4, T5 p5, T6 p6, T7 p7, T8 p8, T9 p9)
2019-11-04T17:24:25.9783364Z    at Microsoft.Dynamics365.UIAutomation.Browser.BrowserPage.Execute[TResult](BrowserCommandOptions options, Func`2 delegate)
2019-11-04T17:24:25.9783555Z    at Microsoft.Dynamics365.UIAutomation.Api.Navigation.OpenSubArea(String area, String subArea, Int32 thinkTime)
2019-11-04T17:24:25.9783623Z    at UCI_Test.Web.Human_Resource.OpenEmployee.WEBOpenEmployee() in d:\a\1\s\Testing\UCI_Test\UCI_Test\Web\Human Resource\OpenEmployee.cs:line 21
2019-11-04T17:24:26.7413016Z Results File: d:\a\_temp\TestResults\VssAdministrator_fv-az44_2019-11-04_17_24_25.trx
2019-11-04T17:24:26.7436649Z Total tests: 1. Passed: 0. Failed: 1. Skipped: 0.
2019-11-04T17:24:26.7451366Z Test execution time: 2.0532 Minutes
2019-11-04T17:24:26.7507602Z ##[error]Test Run Failed.

Code

using System;
using System.Security;
using Microsoft.Dynamics365.UIAutomation.Api;
using Microsoft.Dynamics365.UIAutomation.Browser;
using Microsoft.VisualStudio.TestTools.UnitTesting;

namespace UCI_Test.Web.Human_Resource
{
    [TestClass]
    public class OpenEmployee
    {
        private readonly SecureString _username = System.Configuration.ConfigurationManager.AppSettings["OnlineUsername"].ToSecureString();
        private readonly SecureString _password = System.Configuration.ConfigurationManager.AppSettings["OnlinePassword"].ToSecureString();
        private readonly Uri _xrmUri = new Uri(System.Configuration.ConfigurationManager.AppSettings["OnlineCrmUrl"].ToString());

        [TestMethod]
        public void WEBOpenEmployee()
        {
            using (var okBrowser = new Browser(Config.Options))
            {
                okBrowser.GoToXrmUri(_xrmUri);
                okBrowser.Navigation.OpenSubArea("Human Resources", "Employee");
                okBrowser.ThinkTime(2000);
                okBrowser.Grid.SwitchView("All Employees");
                okBrowser.Grid.OpenRecord(0);

            }
        }
     }
}

App Settings

  <appSettings>
    <add key="OnlineUsername" value="username" />
    <add key="OnlinePassword" value="password" />
    <add key="OnlineCrmUrl" value="url" />
    <add key="AzureKey" value="" />
    <!-- IE,Chrome,Firefox,Edge-->
    <add key="BrowserType" value="Chrome" />
    <add key="ClientSettingsProvider.ServiceUri" value="" />
  </appSettings>

Config

 public static BrowserOptions Options = new BrowserOptions
        {
            BrowserType = (BrowserType)Enum.Parse(typeof(BrowserType), Type),
            PrivateMode = true,
            FireEvents = false,
            Headless = false,
            UserAgent = false,
            DefaultThinkTime = 2000
        };

[AngelRodriguez8008]

Hi @mgr1054,

Did you install your agent in Interactive Mode?

br Angel A.

[mgr1054]

Hey @AngelRodriguez8008,

I don't have a self-hosted agent. The tests are supposed to run on a Microsoft Agent (vs2017-win2016).

Thanks

[joserito]

Hello @mgr1054,

Check this article about special considerations for running UI tests in Azure DevOps.

[Shakevg]

Hello @mgr1054,

Did this tests passed locally? Test should work on Agent (vs2017-win2016). First step was passed (opening url). Per log test just cannot Open SubArea. It can be different reasons, the most likely reason is slow page loading.

You can understand the reason using 2 options:

1. Add additional sleep before OpenSubArea okBrowser.GoToXrmUri(_xrmUri); okBrowser.ThinkTime(10000); okBrowser.Navigation.OpenSubArea("Human Resources", "Employee");
2. Make screenshot of test fail and attach it to test https://stackoverflow.com/questions/10995017/attach-an-image-to-a-test-report-in-mstest It will be available on Attachments tab in Azure devOps test results tab) https://docs.microsoft.com/en-us/azure/devops/pipelines/test/collect-screenshots-and-video?view=azure-devops

Change your code to the next:

public TestContext TestContext { get; set; }
public Browser OkBrowser;

[TestInitialize]
public void TestInitialize()
{
    OkBrowser = new Browser(Config.Options);
}

[TestCleanup]
public void TestCleanup()
{
    if (TestContext.CurrentTestOutcome == UnitTestOutcome.Failed)
    {
        Screenshot screenShot = ((ITakesScreenshot)OkBrowser.Driver).GetScreenshot();

        string screenshotsFile = Path.Combine(TestContext.DeploymentDirectory, "screenshot.png");
        screenShot.SaveAsFile(screenshotsFile, ScreenshotImageFormat.Png);

        TestContext.AddResultFile(screenshotsFile);
    }
}

[TestMethod]
public void WEBOpenEmployee()
{
    OkBrowser.GoToXrmUri(_xrmUri);
    OkBrowser.Navigation.OpenSubArea("Human Resources", "Employee");
    OkBrowser.ThinkTime(2000);
    OkBrowser.Grid.SwitchView("All Employees");
    OkBrowser.Grid.OpenRecord(0);
}

If it is really related with slow loading you can fix using:

• sleep (option 1)
• add additional waiter after okBrowser.GoToXrmUri(_xrmUri) step

[AngelRodriguez8008]

@Shakevg what is the advantage of not to use:

  string screenshotsFile = Path.Combine(TestContext.DeploymentDirectory, filename);
  _client.Browser.TakeWindowScreenShot(screenshotsFile, ScreenshotImageFormat.Png);

That save directly the file into the desired location.

[Shakevg]

@AngelRodriguez8008 Agree that it is more simple, but internally the same. Missed this existing method.