Native support for Azure Devops parallelization

markgoho commented 5 years ago

Current behavior:

Supports "out of the box" parallelization for most popular CI environments.

Desired behavior:

Support Azure Devops

Cf. https://docs.cypress.io/guides/guides/parallelization.html#Linking-CI-machines-for-parallelization-or-grouping

https://github.com/cypress-io/cypress/blob/master/packages/server/lib/util/ci_provider.js

We need Azure Devops to be supported here.

Note: Azure Devops is the new name for Visual Studio Team Services (VSTS) which was previously called Visual Studio Online and Team Foundation Services. There has been some work done to support this throughout Cypress, but as far as I can tell, nothing in the parallelization links listed above.

Might be helpful: https://docs.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=designer#multi-configuration

jennifer-shehane commented 5 years ago

Discussion on this also here: https://github.com/cypress-io/cypress-documentation/issues/922

cc @bahmutov

bahmutov commented 5 years ago

@markgoho can I ask what is NOT working for you? We are extracting the variables from team foundation variables, so the dashboard should be able to show the commit information.

is the parallelization not using a variable to tie different machines running in parallel together? Which variable would you use to link runs? You can try doing it yourself with --ci-build-id

The reason we looked at Azure Devops and only created a basic example is that Azure DevOps does NOT provide caching of installed dependencies, making npm install command run from zero on each build, which is pretty painful.

markgoho commented 5 years ago

hey @bahmutov thanks for the quick response!

The dashboard is reading the e2e tests just fine. I'm primarily interested in seeing parallelization supported natively like the 10 or so other CI environments.

Dependency re-installs are a little painful (2.5 min or so on their hosted ubuntu agent) but does that have an impact on running the spec files in parallel on your system?

bahmutov commented 5 years ago

ok, for Azure Devops, the builds are not linked by an environment right away? which variable would you pass with --ci-build-id to link them like shown here https://on.cypress.io/parallelization?

The dependencies install / lack of Docker support is painful but does not affect the test run itself. But especially for parallel mode, if you spend 3 minutes to install NPM dependencies to spend 30 seconds to run tests ... that is super painful, especially compared to other CIs

markgoho commented 5 years ago

We'd probably pass the build number, which is like 20190306.5

I'm not 100% how the parallelization is determined, but let's say I had 10 spec files and 5 machines available. Your service spins up those other machines, installs dependencies and runs, say, 2 spec files per machine.

So the question is: do the parallelized tests (with the extra dependency installation) take less time to run than running them in sequence but with just 1 dependency installation?

Yeah, wtf Azure Devops? I know they have a proposal for caching but it's probably weeks to months away. I think this is probably all academic until they get their caching strategy in place. For now it's kind of pointless to do parallelization with Azure.

bahmutov commented 5 years ago

I have finally created a nice parallel example on Azure CI pipelines, here is the config file https://github.com/cypress-io/cypress-example-kitchensink/blob/master/azure-ci.yml

Builds https://cypress-io.visualstudio.com/cypress-example-kitchensink/_build/results

To parallelize I have used the variable BUILD_BUILDNUMBER which is the same among all parallel tasks in a job. The other variables relevant are starting with AGENT, BUILD and SYSTEM

AGENT_ACCEPTTEEEULA=True
AGENT_BUILDDIRECTORY=/home/vsts/work/1
AGENT_DISABLELOGPLUGIN_TESTFILEPUBLISHERPLUGIN=true
AGENT_DISABLELOGPLUGIN_TESTRESULTLOGPLUGIN=false
AGENT_HOMEDIRECTORY=/home/vsts/agents/2.149.2
AGENT_ID=6
AGENT_JOBNAME=Job1
AGENT_JOBSTATUS=Succeeded
AGENT_MACHINENAME=fv-az619
AGENT_NAME=Hosted Agent
AGENT_OS=Linux
AGENT_OSARCHITECTURE=X64
AGENT_RETAINDEFAULTENCODING=false
AGENT_ROOTDIRECTORY=/home/vsts/work
AGENT_TEMPDIRECTORY=/home/vsts/work/_temp
AGENT_TOOLSDIRECTORY=/opt/hostedtoolcache
AGENT_VERSION=2.149.2
AGENT_WORKFOLDER=/home/vsts/work
BUILD_ARTIFACTSTAGINGDIRECTORY=/home/vsts/work/1/a
BUILD_BINARIESDIRECTORY=/home/vsts/work/1/b
BUILD_BUILDID=202
BUILD_BUILDNUMBER=20190410.14
BUILD_BUILDURI=vstfs:///Build/Build/202
BUILD_CONTAINERID=2222494
BUILD_DEFINITIONNAME=cypress-io.cypress-example-kitchensink
BUILD_DEFINITIONVERSION=5
BUILD_QUEUEDBY=GitHub
BUILD_QUEUEDBYID=7039aca1-88b6-4fb4-ab41-ea1084848cc5
BUILD_REASON=IndividualCI
BUILD_REPOSITORY_CLEAN=false
BUILD_REPOSITORY_GIT_SUBMODULECHECKOUT=False
BUILD_REPOSITORY_ID=cypress-io/cypress-example-kitchensink
BUILD_REPOSITORY_LOCALPATH=/home/vsts/work/1/s
BUILD_REPOSITORY_NAME=cypress-io/cypress-example-kitchensink
BUILD_REPOSITORY_PROVIDER=GitHub
BUILD_REPOSITORY_URI=https://github.com/cypress-io/cypress-example-kitchensink
BUILD_REQUESTEDFOR=Gleb Bahmutov
BUILD_REQUESTEDFOREMAIL=gleb.bahmutov@gmail.com
BUILD_REQUESTEDFORID=892b9acc-1dc9-60c5-846d-406494e0f162
BUILD_SOURCEBRANCH=refs/heads/master
BUILD_SOURCEBRANCHNAME=master
BUILD_SOURCESDIRECTORY=/home/vsts/work/1/s
BUILD_SOURCEVERSION=4a0361b0e393b8c4a79f73213893115544f67592
BUILD_SOURCEVERSIONAUTHOR=Gleb Bahmutov
BUILD_SOURCEVERSIONMESSAGE=set up Azure CI parallel example, close #109
BUILD_STAGINGDIRECTORY=/home/vsts/work/1/a
SYSTEM=build
SYSTEM_ARTIFACTSDIRECTORY=/home/vsts/work/1/a
SYSTEM_COLLECTIONID=5e0090d5-c5b9-4fab-8fd8-ce288e9fb666
SYSTEM_COLLECTIONURI=https://cypress-io.visualstudio.com/
SYSTEM_CULTURE=en-US
SYSTEM_DEBUG=false
SYSTEM_DEFAULTWORKINGDIRECTORY=/home/vsts/work/1/s
SYSTEM_DEFINITIONID=2
SYSTEM_DEFINITIONNAME=cypress-io.cypress-example-kitchensink
SYSTEM_ENABLEACCESSTOKEN=SecretVariable
SYSTEM_HOSTTYPE=build
SYSTEM_ISSCHEDULED=False
SYSTEM_JOBATTEMPT=1
SYSTEM_JOBDISPLAYNAME=Job1
SYSTEM_JOBID=3677b765-f636-593c-68e0-4ea700fd016f
SYSTEM_JOBIDENTIFIER=Cypress_e2e_tests.Job1
SYSTEM_JOBNAME=Job1
SYSTEM_JOBPARALLELISMTAG=Public
SYSTEM_JOBPOSITIONINPHASE=1
SYSTEM_PARALLELEXECUTIONTYPE=MultiMachine
SYSTEM_PHASEDISPLAYNAME=Cypress_e2e_tests
SYSTEM_PHASEID=9df2adbc-b881-59c8-9500-86e4319f3a78
SYSTEM_PHASENAME=Cypress_e2e_tests
SYSTEM_PIPELINESTARTTIME=2019-04-10 17:52:15-04:00
SYSTEM_PLANID=dd1598ce-6132-4ee9-98b7-65b953a077a8
SYSTEM_PULLREQUEST_ISFORK=False
SYSTEM_SERVERTYPE=Hosted
SYSTEM_TASKDEFINITIONSURI=https://cypress-io.visualstudio.com/
SYSTEM_TASKDISPLAYNAME=Run Cypress tests
SYSTEM_TASKINSTANCEID=12d105dc-0f9b-5efb-be0f-f60b28c1af97
SYSTEM_TASKINSTANCENAME=CmdLine3
SYSTEM_TEAMFOUNDATIONCOLLECTIONURI=https://cypress-io.visualstudio.com/
SYSTEM_TEAMFOUNDATIONSERVERURI=https://cypress-io.visualstudio.com/
SYSTEM_TEAMPROJECT=cypress-example-kitchensink
SYSTEM_TEAMPROJECTID=e4ae616f-aacc-4d73-a5d5-861780c4d469
SYSTEM_TIMELINEID=dd1598ce-6132-4ee9-98b7-65b953a077a8
SYSTEM_TOTALJOBSINPHASE=10
SYSTEM_WORKFOLDER=/home/vsts/work

How do we decide if this is Azure CI vs Team Foundation?
use BUILD_BUILDNUMBER as CI build id
we need a new profile for Azure CI so we do not display "Team Foundation" in the dashboard

Currently in https://dashboard.cypress.io/#/projects/4b7344/runs/5210/specs it is

bahmutov commented 5 years ago

Related: in build https://cypress-io.visualstudio.com/cypress-example-kitchensink/_build/results?buildId=197 I have printed all environment variables

AGENT_ACCEPTTEEEULA=True
AGENT_ACCEPTTEEEULA=True
AGENT_BUILDDIRECTORY=/home/vsts/work/1
AGENT_BUILDDIRECTORY=/home/vsts/work/1
AGENT_DISABLELOGPLUGIN_TESTFILEPUBLISHERPLUGIN=true
AGENT_DISABLELOGPLUGIN_TESTFILEPUBLISHERPLUGIN=true
AGENT_DISABLELOGPLUGIN_TESTRESULTLOGPLUGIN=false
AGENT_DISABLELOGPLUGIN_TESTRESULTLOGPLUGIN=false
AGENT_HOMEDIRECTORY=/home/vsts/agents/2.149.2
AGENT_HOMEDIRECTORY=/home/vsts/agents/2.149.2
AGENT_ID=6
AGENT_ID=6
AGENT_JOBNAME=Job1
AGENT_JOBNAME=Job1
AGENT_JOBSTATUS=Succeeded
AGENT_JOBSTATUS=Succeeded
AGENT_MACHINENAME=fv-az619
AGENT_MACHINENAME=fv-az619
AGENT_NAME=Hosted Agent
AGENT_NAME=Hosted Agent
AGENT_OS=Linux
AGENT_OS=Linux
AGENT_OSARCHITECTURE=X64
AGENT_OSARCHITECTURE=X64
AGENT_RETAINDEFAULTENCODING=false
AGENT_RETAINDEFAULTENCODING=false
AGENT_ROOTDIRECTORY=/home/vsts/work
AGENT_ROOTDIRECTORY=/home/vsts/work
AGENT_TEMPDIRECTORY=/home/vsts/work/_temp
AGENT_TEMPDIRECTORY=/home/vsts/work/_temp
AGENT_TOOLSDIRECTORY=/opt/hostedtoolcache
AGENT_TOOLSDIRECTORY=/opt/hostedtoolcache
AGENT_VERSION=2.149.2
AGENT_VERSION=2.149.2
AGENT_WORKFOLDER=/home/vsts/work
AGENT_WORKFOLDER=/home/vsts/work
ANDROID_HOME=/usr/local/lib/android/sdk
ANDROID_SDK_ROOT=/usr/local/lib/android/sdk
ANT_HOME=/usr/share/ant
AZURE_HTTP_USER_AGENT=VSTS_5e0090d5-c5b9-4fab-8fd8-ce288e9fb666_build_2_0
BOOST_ROOT=/usr/local/share/boost/1.69.0
BOOST_ROOT_1_69_0=/usr/local/share/boost/1.69.0
BUILD_ARTIFACTSTAGINGDIRECTORY=/home/vsts/work/1/a
BUILD_BINARIESDIRECTORY=/home/vsts/work/1/b
BUILD_BUILDID=197
BUILD_BUILDNUMBER=20190410.9
BUILD_BUILDURI=vstfs:///Build/Build/197
BUILD_CONTAINERID=2222244
BUILD_DEFINITIONNAME=cypress-io.cypress-example-kitchensink
BUILD_DEFINITIONVERSION=3
BUILD_QUEUEDBY=GitHub
BUILD_QUEUEDBYID=7039aca1-88b6-4fb4-ab41-ea1084848cc5
BUILD_REASON=IndividualCI
BUILD_REPOSITORY_CLEAN=false
BUILD_REPOSITORY_GIT_SUBMODULECHECKOUT=False
BUILD_REPOSITORY_ID=cypress-io/cypress-example-kitchensink
BUILD_REPOSITORY_LOCALPATH=/home/vsts/work/1/s
BUILD_REPOSITORY_NAME=cypress-io/cypress-example-kitchensink
BUILD_REPOSITORY_PROVIDER=GitHub
BUILD_REPOSITORY_URI=https://github.com/cypress-io/cypress-example-kitchensink
BUILD_REQUESTEDFOR=Gleb Bahmutov
BUILD_REQUESTEDFOREMAIL=gleb.bahmutov@gmail.com
BUILD_REQUESTEDFORID=892b9acc-1dc9-60c5-846d-406494e0f162
BUILD_SOURCEBRANCH=refs/heads/master
BUILD_SOURCEBRANCHNAME=master
BUILD_SOURCESDIRECTORY=/home/vsts/work/1/s
BUILD_SOURCEVERSION=8ede71f53e86fd9bc98dfdda593ed824e17c629b
BUILD_SOURCEVERSIONAUTHOR=Gleb Bahmutov
BUILD_SOURCEVERSIONMESSAGE=print all vars
BUILD_STAGINGDIRECTORY=/home/vsts/work/1/a
CHROME_BIN=/usr/bin/google-chrome
COMMON_TESTRESULTSDIRECTORY=/home/vsts/work/1/TestResults
CONDA=/usr/share/miniconda
DOTNET_SKIP_FIRST_TIME_EXPERIENCE=1
ENDPOINT_URL_SYSTEMVSSCONNECTION=https://cypress-io.visualstudio.com/
GIT_TERMINAL_PROMPT=0
GOROOT=/usr/local/go1.12.1
GOROOT_1_10_X64=/usr/local/go1.10
GOROOT_1_11_X64=/usr/local/go1.11
GOROOT_1_12_1_X64=/usr/local/go1.12.1
GOROOT_1_9_X64=/usr/local/go1.9
GRADLE_HOME=/usr/share/gradle
HOME=/home/vsts
JAVA_HOME=/usr/lib/jvm/zulu-8-azure-amd64
JAVA_HOME_11_X64=/usr/lib/jvm/zulu-11-azure-amd64
JAVA_HOME_12_X64=/usr/lib/jvm/zulu-12-azure-amd64
JAVA_HOME_7_X64=/usr/lib/jvm/zulu-7-azure-amd64
JAVA_HOME_8_X64=/usr/lib/jvm/zulu-8-azure-amd64
LANG=en_US.UTF-8
LEIN_HOME=/usr/local/lib/lein
LEIN_JAR=/usr/local/lib/lein/self-installs/leiningen-2.9.1-standalone.jar
M2_HOME=/usr/share/apache-maven-3.6.0
MSDEPLOY_HTTP_USER_AGENT=VSTS_5e0090d5-c5b9-4fab-8fd8-ce288e9fb666_build_2_0
PATH=/opt/hostedtoolcache/node/10.15.3/x64/bin:"/home/vsts/.dotnet/tools:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin"
PWD=/home/vsts/work/1/s
SHLVL=1
SYSTEM=build
SYSTEM_ARTIFACTSDIRECTORY=/home/vsts/work/1/a
SYSTEM_COLLECTIONID=5e0090d5-c5b9-4fab-8fd8-ce288e9fb666
SYSTEM_COLLECTIONURI=https://cypress-io.visualstudio.com/
SYSTEM_CULTURE=en-US
SYSTEM_DEBUG=false
SYSTEM_DEFAULTWORKINGDIRECTORY=/home/vsts/work/1/s
SYSTEM_DEFINITIONID=2
SYSTEM_DEFINITIONNAME=cypress-io.cypress-example-kitchensink
SYSTEM_ENABLEACCESSTOKEN=SecretVariable
SYSTEM_HOSTTYPE=build
SYSTEM_ISSCHEDULED=False
SYSTEM_JOBATTEMPT=1
SYSTEM_JOBDISPLAYNAME=Job1
SYSTEM_JOBID=3677b765-f636-593c-68e0-4ea700fd016f
SYSTEM_JOBIDENTIFIER=Cypress_e2e_tests.Job1
SYSTEM_JOBNAME=Job1
SYSTEM_JOBPARALLELISMTAG=Public
SYSTEM_JOBPOSITIONINPHASE=1
SYSTEM_PARALLELEXECUTIONTYPE=MultiMachine
SYSTEM_PHASEDISPLAYNAME=Cypress_e2e_tests
SYSTEM_PHASEID=9df2adbc-b881-59c8-9500-86e4319f3a78
SYSTEM_PHASENAME=Cypress_e2e_tests
SYSTEM_PIPELINESTARTTIME=2019-04-10 17:21:48-04:00
SYSTEM_PLANID=8b99b100-c664-41f2-94ac-2c7403f2832c
SYSTEM_PULLREQUEST_ISFORK=False
SYSTEM_SERVERTYPE=Hosted
SYSTEM_TASKDEFINITIONSURI=https://cypress-io.visualstudio.com/
SYSTEM_TASKDISPLAYNAME=Run Cypress tests
SYSTEM_TASKINSTANCEID=12d105dc-0f9b-5efb-be0f-f60b28c1af97
SYSTEM_TASKINSTANCENAME=CmdLine3
SYSTEM_TEAMFOUNDATIONCOLLECTIONURI=https://cypress-io.visualstudio.com/
SYSTEM_TEAMFOUNDATIONSERVERURI=https://cypress-io.visualstudio.com/
SYSTEM_TEAMPROJECT=cypress-example-kitchensink
SYSTEM_TEAMPROJECTID=e4ae616f-aacc-4d73-a5d5-861780c4d469
SYSTEM_TIMELINEID=8b99b100-c664-41f2-94ac-2c7403f2832c
SYSTEM_TOTALJOBSINPHASE=3
SYSTEM_WORKFOLDER=/home/vsts/work
TASK_DISPLAYNAME=Run Cypress tests
TF_BUILD=True
USER=vsts
VCPKG_INSTALLATION_ROOT=/usr/local/share/vcpkg
VSTS_AGENT_PERFLOG=/home/vsts/agentperf
VSTS_PROCESS_LOOKUP_ID=vsts_df2b1fa7-38ce-43e1-b9d5-cef2808e140d
_=/usr/bin/env
agent.jobstatus=Succeeded

cypress-bot[bot] commented 5 years ago

Released in 3.3.0.

cypress-io / cypress

Native support for Azure Devops parallelization #3657

Current behavior:

Desired behavior: