App Center Test Task failed with 'Cannot find test-cloud.exe'

[omanuke]

Environment

- If using VSTS, provide the account name, team project name, build definition name/build number: 

　　account name: miyasaka_masahiko@tworks.co.jp 　　team project name: TestAppCenter 　　build definition name/build number: TestAppCenter-Xamarin.Android-CI (1) / Build 20180404.1

• Agent - Hosted or Private:

  • If using Hosted agent, provide agent queue name: Hosted VS2017

Issue Description

'Test with Visual Studio App Center' failed on Prepare Tests. The log said test-cloud.exe couldn't be found. My settings for App Center Test is like below. I tried to add Addition option "--uitest-tools-dir" which the log says about with the value '$(APPCENTER_SOURCE_DIRECTORY)/packages/Xamarin.UITest./tools' or '$APPCENTER_SOURCE_DIRECTORY/packages/Xamarin.UITest./tools'. But it doesn't work. '$APPCENTER_SOURCE_DIRECTORY/packages/Xamarin.UITest.*/tools' is written here

Error logs

******

Starting: Test with Visual Studio App Center

****** \============================================================================== Task : App Center Test Description : Test app packages with Visual Studio App Center. Version : 1.131.0 Author : Microsoft Corporation Help : For help with this task, visit the Visual Studio App Center support site. \==============================================================================

C:\Windows\system32\cmd.exe /D /S /C "D:\a_tasks\AppCenterTest_ad5cd22a-be4e-48bb-adce-181a32432da5\1.131.0\node_modules.bin\appcenter.cmd test prepare uitest --artifacts-dir D:\a\1\a\AppCenterTest --app-path D:/a/1/b/Release/com.companyname.TestAppCenter.apk --build-dir D:\a\1\b/Release/test-assembly/test.dll --quiet" Preparing tests... failed. Error: Cannot find test-cloud.exe, which is required to prepare UI tests. We have searched for directory "packages\Xamarin.UITest.\tools" inside "D:\a\1\b/Release/test-assembly/test*.dll" and all of its parent directories. Please use option "--uitest-tools-dir" to manually specify location of this tool. Minimum required version is "2.2.0".

Error: D:\a_tasks\AppCenterTest_ad5cd22a-be4e-48bb-adce-181a32432da5\1.131.0\node_modules.bin\appcenter.cmd failed with return code: 3

[omanuke]

I set "$(build.sourcesdirectory)/packages/Xamarin.UITest.2.2.3/tools" on 'Test tools directory'. Now, test-cloud.exe could be found successfully. But specifying Xamarin.UITest.X.X.X is fragile. Is there a good configuration way?

[owenniblock]

Thanks @omanuke - I think you're the second person to request this feature (to my knowledge). I'll pass it on to my PM and get it recorded as a feature request.

[hubertusgr]

I also believe this is very fragile +1

[kumarpun]

I spent 2 days figuring out testcloud.dll but got no success please make a easy way to do UI test from app center for xamarin.forms it's really very important to me.

[gsstudio]

This is annoying, I can not find information. appcenter throw me this error ... Cannot find test-cloud.exe, which is required to prepare UI tests.

[Abdul-waseem123]

@omanuke you can use "$(build.sourcesdirectory)/packages/Xamarin.UITest.*/tools" to take any version

[vtkumaresan]

do we have solution? what is the correct build directory path?

iOS: Error: /usr/local/bin/appcenter failed with return code: 3

Android: Error: D:\a_tasks\AppCenterTest_ad5cd22a-be4e-48bb-adce-181a32432da5\1.137.0\node_modules.bin\appcenter.cmd failed with return code: 3

[kumarpun]

Kumaresan P you can use "$(build.sourcesdirectory)/ packages/Xamarin.UITest.*/tools" to take any version

On Mon, Aug 6, 2018 at 6:08 PM, Kumaresan P notifications@github.com wrote:

do we have solution?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Microsoft/vsts-tasks/issues/6868#issuecomment-410689918, or mute the thread https://github.com/notifications/unsubscribe-auth/AjYshox6FZy4Yji5kT_hXGRaDkP5VUl7ks5uODVWgaJpZM4TGJoY .

[vtkumaresan]

packages folder not found in finder but showing in project. so how do we set above path.

[owenniblock]

You need to set this ($(build.sourcesdirectory)/ packages/Xamarin.UITest.*/tools) in your Test tools directory on the App Center Test task.

[kumarpun]

You need to set this ($(build.sourcesdirectory)/ packages/Xamarin.UITest.*/tools) in your Test tools directory on the App Center Test task.

On Tue, Aug 7, 2018 at 12:41 PM, Kumaresan P notifications@github.com wrote:

packages folder not found in finder but showing in project. so how do we set above path. [image: screen shot 2018-08-07 at 12 24 42 pm] https://user-images.githubusercontent.com/3971168/43759487-06dbbd26-9a3d-11e8-8c2c-0befe24d60ad.png

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Microsoft/vsts-tasks/issues/6868#issuecomment-410953632, or mute the thread https://github.com/notifications/unsubscribe-auth/AjYshh_3GVoprUTkKhwdwLXfWbw-sKhhks5uOToGgaJpZM4TGJoY .

[vtkumaresan]

Getting same error,

`2018-08-07T10:54:11.5087690Z Error: Cannot find test-cloud.exe, the path specified by "--uitest-tools-dir" was not found.

2018-08-07T10:54:11.5107630Z Please check that "/Users/vsts/agent/2.136.1/work/1/s/packages/Xamarin.UITest.*/tools" is a valid directory and contains test-cloud.exe.

2018-08-07T10:54:11.5124110Z Minimum required version is "2.2.0".

2018-08-07T10:54:11.5277090Z ##[error]Error: /usr/local/bin/appcenter failed with return code: 3 `

[vtkumaresan]

Today tried different types of url but still getting same error, $(build.sourcesdirectory)/packages/Xamarin.UITest.2.2.4/tools $(build.sourcesdirectory)/packages/xamarin.uitest/2.2.2/tools $(build.sourcesdirectory)/packages/Xamarin.UITest.*/tools $(build.sourcesdirectory)/packages/Xamarin.UITest.2.2.2/tools $(build.sourcesdirectory)/packages/Xamarin.UITest.2.2.3/tools

[owenniblock]

What's in $(build.sourcesdirectory)? You could add a Powershell (or Bash) script to output the directory contents. Maybe you need to be looking somewhere else?

[vtkumaresan]

what could be the correct path? shall use '$(APPCENTER_SOURCE_DIRECTORY)/' ? where the 'test-cloud.exe' will be located. 'Cannot find test-cloud.exe, the path specified by "--uitest-tools-dir" was not found.'

[owenniblock]

Just a thought - I assume you're building the source (so downloading the necessary nuget packages) before running the tests?

The packages folder should be wherever you'd expect it to be based on your project folder structure.

[vtkumaresan]

Test initiated after Build generation for Continuous Testing. 'Packages' folder not found in project location. I'm using Xamarin.Form for mobile application. So trying to set 'Packages' folder through 'Test tools directory'

[kumarpun]

You need to set this ($(build.sourcesdirectory)/ packages/Xamarin.UITest.*/tools) in your Test tools directory on the App Center Test task.

On Sun, May 13, 2018 at 4:37 PM, gsstudio notifications@github.com wrote:

This is annoying, I can not find information. appcenter throw me this error ... Cannot find test-cloud.exe, which is required to prepare UI tests.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Microsoft/vsts-tasks/issues/6868#issuecomment-388618032, or mute the thread https://github.com/notifications/unsubscribe-auth/AjYshiejZ9nYs6CisAGE_C2FixXsxWN-ks5tyBBfgaJpZM4TGJoY .

[vtkumaresan]

tried already got below error message, 2018-08-07T10:54:11.5070140Z Preparing tests... failed. 2018-08-07T10:54:11.5087690Z Error: Cannot find test-cloud.exe, the path specified by "--uitest-tools-dir" was not found. 2018-08-07T10:54:11.5107630Z Please check that "/Users/vsts/agent/2.136.1/work/1/s/packages/Xamarin.UITest.*/tools" is a valid directory and contains test-cloud.exe.

[owenniblock]

Does the build process definitely build the UITest project? (If it's just building the app project this might be the case).

And also are there any build logs in VSTS which show where nuget packages are being restored to? That might help us find the tools :)

[vtkumaresan]

Thank you @owenniblock . Got nuget package url in build log, /Users/vsts/.nuget/packages/xamarin.uitest/2.2.2

[owenniblock]

No problem - sorry I didn't think of doing that sooner! Let me know if Xamarin.UITest.* works or whether you have to add the version number :)

[fuzzybinary]

To follow up a bit on this -- it seems like Xamarin.UITest no longer restores to the packages directory, but now only restores to the global nuget cache. Not sure when that changed.

However, in our case we try to run test as part of a pipeline after the build is complete, not as part of the same step. It would be nice if this step allowed you to specify a version of Xamarin UITest and pulled its tools down directly, instead of relying on them to be there from a nuget restore.

[Scriptwarrior1027]

I have tried every permutation of wild carding this path and still no joy. Seems that the task only runs with the hardcoded path which is fragile.

($(build.sourcesdirectory)/packages/Xamarin.UITest.2.2.6/tools/

My example is running against a host MacOS bulid machine.

[fuzzybinary]

@Scriptwarrior1027 What we ended up doing is putting in a Nuget Task to download the specific version of Xamarin.UITest we want, then leave the "Test Tools Directory" blank and placing the installed package where the Task could find it (in packages)

Our Nuget Command is: install Xamarin.UITest -Version $(Xamarin.UITest.Version) -OutputDirectory "$(System.DefaultWorkingDirectory)\\drop\packages"

[spontoreau]

Any updates?

[Minneth]

I managed to sort this by providing ./packages/Xamarin.UITest.2.2.6/tools as the path, but I've then found myself where it can't find the apk/ipa.

[Minneth]

It seems that ./packages/Xamarin.UITest.*/tools works too.

[nichojo89]

@owenniblock I've been tirelessly working on this error as well. I think I am also unable to find test-cloud.exe because my build process is maybe building the android project & not building as a solution. I've seen you mention this solution on a few threads but I dont understand how to do it? Is this just adding the project to the configuration manager of the sln in visual studio? If so I've already done this, but I am hoping I am just misunderstanding.

[owenniblock]

@nichojo89 - you could probably achieve this in a few ways. The way I usually favour is to add a separate build step which specifically builds the test assets (if you see the screenshot in the original issue - this would be the build solution... step.

You may also need the NuGet Restore step before build if you don't have it already.

[Minneth]

@owenniblock - This is a limitation in AppCenter - The iOS builds use the .sln but the Android use the .csproj. What I ended up doing was having an appcenter-post-build.sh script at solution level for my iOS build, and another one at project level for Android. It works as it should now.

One other thing to note is that for android builds you will probably want to either check in your UITests/bin/ folder so the assembly is available for the build (since it's built at project level as mentioned above) or you could add an msbuild command to your appcenter-post-build.sh script to actually compile your UITest project before you kick off your tests.

I hope this information helps everyone, I had a nightmare with this but the above seems to work flawlessly once it's set up with the correct paths.

[pulimento]

The test-cloud.exe can be located at 'packages/foo' in projects that uses the old project structure (projects that use packages.config). For new projects (new .csproj formats), there is no such file in the path of the project. The only way I found to make it work on AppCenter is to use it from the NuGet package cache (/Users/vsts/.nuget/packages/xamarin.uitest/2.2.7), but this is ridiculously fragile ATM.

[asbpp]

Environment

- If using VSTS, provide the account name, team project name, build definition name/build number: 

　　account name: miyasaka_masahiko@tworks.co.jp 　　team project name: TestAppCenter 　　build definition name/build number: TestAppCenter-Xamarin.Android-CI (1) / Build 20180404.1

• Agent - Hosted or Private:

  • If using Hosted agent, provide agent queue name: Hosted VS2017

Issue Description

'Test with Visual Studio App Center' failed on Prepare Tests. The log said test-cloud.exe couldn't be found. My settings for App Center Test is like below. I tried to add Addition option "--uitest-tools-dir" which the log says about with the value '$(APPCENTER_SOURCEDIRECTORY)/packages/Xamarin.UITest./tools' or '$APPCENTER_SOURCEDIRECTORY/packages/Xamarin.UITest./tools'. But it doesn't work. '$APPCENTER_SOURCE_DIRECTORY/packages/Xamarin.UITest.*/tools' is written here

Error logs

---

Starting: Test with Visual Studio App Center

---

============================================================================== Task : App Center Test Description : Test app packages with Visual Studio App Center. Version : 1.131.0 Author : Microsoft Corporation Help : For help with this task, visit the Visual Studio App Center support site.

C:\Windows\system32\cmd.exe /D /S /C "D:\a_tasks\AppCenterTest_ad5cd22a-be4e-48bb-adce-181a32432da5\1.131.0\node_modules.bin\appcenter.cmd test prepare uitest --artifacts-dir D:\a\1\a\AppCenterTest --app-path D:/a/1/b/Release/com.companyname.TestAppCenter.apk --build-dir D:\a\1\b/Release/test-assembly/test.dll --quiet" Preparing tests... failed. Error: Cannot find test-cloud.exe, which is required to prepare UI tests. We have searched for directory "packages\Xamarin.UITest.*\tools" inside "D:\a\1\b/Release/test-assembly/test.dll" and all of its parent directories. Please use option "--uitest-tools-dir" to manually specify location of this tool. Minimum required version is "2.2.0".

Error: D:\a_tasks\AppCenterTest_ad5cd22a-be4e-48bb-adce-181a32432da5\1.131.0\node_modules.bin\appcenter.cmd failed with return code: 3

Hello, we are facing same problem and unable to resolved it.

[owenniblock]

Hi @asbpp - the build directory must point to a directory (so in your case you'll need to remove the *test*.dll

If there are DLLs which you don't want to upload to the Test Cloud - I would recommend either altering the build process which generated these assets (my preferred approach) or adding a step between build and upload which creates a clean directory for upload.

If that still doesn't work - you'll need to locate the directory which contains test-cloud.exe on your VSTS instance and point to it using the --uitest-tools-dir parameter.

[DeanFaizal]

Am also seeing this. Can confirm the path is:

/Users/vsts/.nuget/packages/xamarin.uitest/2.2.7/tools

[owenniblock]

If you add a line to your script to output the contents on this directory - what does it show?

Something like:

ls /Users/vsts/.nuget/packages/xamarin.uitest/2.2.7/tools

[fume]

This is still a problem as of today. All the proposed workarounds could work but are fragile. At least document the workarounds while searching/finding/developing a REAL solution!

thanks, regards

[asbpp]

This is resolved. We weren’t doing it right.

On Apr 11, 2019, at 7:31 AM, Fulvio notifications@github.com wrote:

This is still a problem as of today. All the proposed workarounds could work but are fragile. At least document the workarounds while searching/finding/developing a REAL solution!

thanks, regards

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Microsoft/azure-pipelines-tasks/issues/6868#issuecomment-482077388, or mute the thread https://github.com/notifications/unsubscribe-auth/AXPz5hKO0AutvcWo0jwRX6od87xTNSwvks5vfx0DgaJpZM4TGJoY.

[jixer]

I don't know that anyone has mentioned this workaround yet, but instead of specifying a direct path (including version number) for Test tools directory, use a CopyFiles task to copy '$(build.sourcesdirectory)/packages/Xamarin.UITest.*/tools/**' to '$(build.sourcesdirectory)/tools'. Then you can just hardcode the path without version number and a version update on the project won't break the build.

[owenniblock]

Thanks @jixer - that's a great workaround.

[Brosten]

@jixer Copy from packages folder is still a problem when no packages folder exists, isn't it? I'm building in Azure DevOps, pushing to AppCenter. Is there a working solution for this? All my attempts (with paths given above) has failed.

[swathg]

@Brosten I am following same approach as yours and got this solution -

1. Keep test-cloud.exe in the Azure DevOps repo
2. Copy the exe file into a packages folder in build pipeline
3. Refer this as path in Test tools directory. eg - $(System.DefaultWorkingDirectory)/Mobile-Android/drop/packages

This is working fine for me :)

[grockland]

Have to agree with @Brosten. There is still a problem when AppCenter retrieves from DevOps repo and the package or .vsts or .nuget folders don't exist. Can't locate --uitest-tools-dir or test-tools.exe on the AppCenter server for use in Test with the appcenter-post-build.sh script and the CLI template to run UITests. Shouldn't this be automated in the AppCenter Test UI?

[acastr7]

A work around that works for me without having to check in test-cloud.exe is:

Update your Nuget restore step and set a destination directory.

Then inside your App Center test step, point the tools directory to that location.

You could probably wildcard to that path so you don't have to hardcode the version.

[cmcnicholas]

Still all suggested workarounds here are fragile, I'm of the opinion that the devops task should handle fetching the test utils, this is still a nightmare to configure and I'm not having much joy.

[github-actions[bot]]

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days

[jmegner]

I think I have a workaround that is not so fragile: use a PowerShell script with wildcards to get the tools folder regardless of Xamarin.UITest version. I have a XamarinPipelineDemo repo with README that covers this issue, and here is a relevant snippet...

- task: NuGetCommand@2
  displayName: 'nuget-restore solution'
  inputs:
    restoreSolution: '$(solution)'
    restoreDirectory: '$(nugetPackageDir)'

# ... lots of steps omitted here ...

# default nodejs version (v12) is not compatible with stuff used in AppCenterTest task
# https://github.com/microsoft/appcenter-cli/issues/696#issuecomment-553218361
- task: UseNode@1
  displayName: 'Use Node 10.15.1'
  condition: and(succeeded(), eq(variables.wantAppCenter, true))
  inputs:
    version: 10.15.1

- pwsh: |
    $uiTestToolsDir = (Get-ChildItem "$(nugetPackageDir)/Xamarin.UITest/*/tools")[0].FullName
    Write-Output "##vso[task.setvariable variable=uiTestToolsDir]$uiTestToolsDir"
  displayName: 'find Xamarin.UITest tools directory'
  condition: and(succeeded(), eq(variables.wantAppCenter, true))

- task: AppCenterTest@1
  condition: and(succeeded(), eq(variables.wantAppCenter, true))
  inputs:
    appFile: '$(finalApkPathSigned)'
    frameworkOption: uitest
    uiTestBuildDirectory: '$(uiTestDir)'
    uiTestToolsDirectory: '$(uiTestToolsDir)'

[jmegner]

I have even better news: building a Xamarin.UITest project puts a test-cloud.exe into the output folder alongside all the generated dlls. Here's a simpler pipeline snippet:

variables:
  - name: uiTestDir
    value: '$(Build.SourcesDirectory)/XamarinPipelineDemo.UITest'
  - name: uiTestAssemblyDir
    value: '$(uiTestDir)/bin/$(buildConfiguration)'

# ... lots of steps omitted here ...

- task: MSBuild@1
  displayName: 'build ui tests'
  inputs:
    solution: '**/*UITest*.csproj'
    configuration: '$(buildConfiguration)'

# default nodejs version (v12) is not compatible with stuff used in AppCenterTest task
# https://github.com/microsoft/appcenter-cli/issues/696#issuecomment-553218361
- task: UseNode@1
  displayName: 'Use Node 10.15.1'
  inputs:
    version: 10.15.1

- task: AppCenterTest@1
  continueOnError: true
  inputs:
    appFile: '$(finalApkPathSigned)'
    appSlug: 'JacobEgnerDemos/XamarinPipelineDemo' # orgname or username, then '/', then app name
    devices: 'JacobEgnerDemos/demo_device_set' # uses same orgname or username, then '/', then device set name
    frameworkOption: 'uitest'
    serverEndpoint: 'AppCenterConnectionUserBasedFullAccess' # make a App Center user API token, then add service connection in Azure DevOps
    uiTestBuildDirectory: '$(uiTestAssemblyDir)' # directory that contains the uitest assemblies, not the build directory
    uiTestToolsDirectory: '$(uiTestAssemblyDir)' # build process puts test-cloud.exe in assembly dir

Full sample repo with writeup on GitHub and Azure DevOps.