How to run specific combination of categories?

[IeuanWalker]

I'm running this command, to run the tests in appcenter -

appcenter test run uitest --app "Cardiff-Council/Cardiff-Gov-1" --devices 8ad25728 --app-path C:\WS\Azure-CardiffCouncilICT\App\CardiffMobileApp\CardiffMobileApp.Android\bin\Release\uk.gov.cardiff.CardiffGov.Android-Signed.apk --test-series "master" --locale "en_GB" --build-dir C:\WS\Azure-CardiffCouncilICT\App\App.UITest\bin\Release --uitest-tools-dir C:\Users\ieuan\.nuget\packages\xamarin.uitest\4.1.3\tools --include-category AppStartup&&Dark --async

I want to run the tests that include the category AppStartup and Dark

When i run the above command it runs all tests with the category AppStartup but ignores Dark. After the tests have ran in appcenter i get this error back in the console - 'Dark' is not recognized as an internal or external command

Locally using NUnit i am able to do this, which works -

nunit3-console App.UITest.dll --where="cat==AppStartup && cat==Dark"

[MikhailSuendukov]

Hi @IeuanWalker and thanks for reaching out to us. Have you tried using AppStartup&&Dark inside quotes? For example "AppStartup&&Dark".

[IeuanWalker]

@MikhailSuendukov not sure did try quite a few variations. I've finished for the day now, so will give that a go in the morning

[IeuanWalker]

@MikhailSuendukov no doesnt work, error saying no tests would be run

C:\Users\ieuan>appcenter test run uitest --app "Cardiff-Council/Cardiff-Gov-1" --devices 8ad25728 --app-path C:\WS\Azure-CardiffCouncilICT\App\CardiffMobileApp\CardiffMobileApp.Android\bin\Release\uk.gov.cardiff.CardiffGov.Android-Signed.apk --test-series "master" --locale "en_GB" --build-dir C:\WS\Azure-CardiffCouncilICT\App\App.UITest\bin\Release --uitest-tools-dir C:\Users\ieuan\.nuget\packages\xamarin.uitest\4.1.3\tools --include-category "AppStartup&&Dark" --async
\ Preparing tests... WARNING: Testing tests built for .NET 6 and above is not yet supported on Test Cloud.

- Preparing tests... No tests would be run in Xamarin Test Cloud, reasons:
Filtered.: App.UITest.Tests.AppStartup.AppStartupTests(Android,English,Dark).AppSetup_FillDetails, App.UITest.Tests.AppStartup.AppStartupTests(Android,English,Dark).AppSetup_IllDoItLater, App.UITest.Tests.AppStartup.AppStartupTests(Android,Welsh,Light).AppSetup_FillDetails, App.UITest.Tests.AppStartup.AppStartupTests(Android,Welsh,Light).AppSetup_IllDoItLater, App.UITest.Tests.AppStartup.AppStartupTests(iOS,English,Dark).AppSetup_FillDetails, App.UITest.Tests.AppStartup.AppStartupTests(iOS,English,Dark).AppSetup_IllDoItLater, App.UITest.Tests.AppStartup.AppStartupTests(iOS,Welsh,Light).AppSetup_FillDetails, App.UITest.Tests.AppStartup.AppStartupTests(iOS,Welsh,Light).AppSetup_IllDoItLater, App.UITest.Tests.FlyTipping.FlyTippingTests(Android,English,Dark).FlyTipping_FillDetailsFromSettings, App.UITest.Tests.FlyTipping.FlyTippingTests(Android,English,Dark).FlyTipping_NoDetailsFromSettings, App.UITest.Tests.FlyTipping.FlyTippingTests(Android,Welsh,Light).FlyTipping_FillDetailsFromSettings, App.UITest.Tests.FlyTipping.FlyTippingTests(Android,Welsh,Light).FlyTipping_NoDetailsFromSettings, App.UITest.Tests.FlyTipping.FlyTippingTests(iOS,English,Dark).FlyTipping_FillDetailsFromSettings, App.UITest.Tests.FlyTipping.FlyTippingTests(iOS,English,Dark).FlyTipping_NoDetailsFromSettings, App.UITest.Tests.FlyTipping.FlyTippingTests(iOS,Welsh,Light).FlyTipping_FillDetailsFromSettings, App.UITest.Tests.FlyTipping.FlyTippingTests(iOS,Welsh,Light).FlyTipping_NoDetailsFromSettings

Preparing tests... failed.
Error: Failed to prepare UI Test artifacts using command "test-cloud.exe prepare" with error message:
No tests would run given the current parameters, please try again. If you can't work out how to fix this issue, please contact support.

Further error details: For help, please send both the reported error above and the following environment information to us by going to https://appcenter.ms/apps and opening a support ticket (using the help menu in the upper right corner of the screen)

    Environment: win32
    App Upload Id: Cardiff-Council/Cardiff-Gov-1
    Timestamp: 1685608824049
    Operation: RunUITestsCommand
    Exit Code: 21
    User Email: **********
    User Name: ieuan.walker
    User Id: **********

[IlyaBausovAkvelon]

@IeuanWalker Hi! Did you tried to use multiple --include-category flags? One for each category.

[IeuanWalker]

@IlyaBausovAkvelon just tried it doesnt have the same affect of '&&' So doing -

--include-category AppStartup --include-category Dark

Ran the test that had 'AppStartup' OR 'Dark' as a category

[IlyaBausovAkvelon]

@IeuanWalker so is this solution work for you?

[IeuanWalker]

@IlyaBausovAkvelon No, unfortunately not.

Our app has 30+ sections, and i want to be able to run the tests for a specific section in a specific theme.

With the above it runs a section on all the themes 'AppStartup' and runs all the other sections in 'Dark' theme.

So the above logic is essentially AppStartup || Dark whereas i need AppStartup && Dark

[IlyaBausovAkvelon]

@IeuanWalker I don't think that AppCenter CLI can do that.

[IeuanWalker]

@IlyaBausovAkvelon how come appcenter doesn't use NUit filtering?

I found this doc that shows the syntax for 'Specifying Test Categories to Include or Exclude' but that doesnt even work - https://docs.nunit.org/2.6.2/consoleCommandLine.html

f.e. running this command -

appcenter test run uitest --app "Cardiff-Council/Cardiff-Gov-1" --devices 8ad25728 --app-path C:\WS\Azure-CardiffCouncilICT\App\CardiffMobileApp\CardiffMobileApp.Android\bin\Release\uk.gov.cardiff.CardiffGov.Android-Signed.apk --test-series "master" --locale "en_GB" --build-dir C:\WS\Azure-CardiffCouncilICT\App\App.UITest\bin\Release --uitest-tools-dir C:\Users\ieuan\.nuget\packages\xamarin.uitest\4.1.4\tools --include-category Dark+AppStartup

I get this error back -

No tests would run given the current parameters, please try again. If you can't work out how to fix this issue, please contact support.

[DmitriyKirakosyan]

Hi @IeuanWalker , unfortunately test-cloud CLI (which is used by the appcenter-cli) doesn't support such filters.

Please provide additional information on your project. What Xamarin.UITest version is used? Have you tried to run nunit3-console command with include argument instead of where, like it is shown on the picture in your last message? Would it work for you?

[DmitriyKirakosyan]

Closing as stale.