UI tests don't work with --test_filter unless ProductModuleName matches source lib

[wendyliga]

did bazel support running testing with spesific case only?

I know xcodebuild has this option called -only-testing, which can be used by passing an array of Test Bundle/Test Suite/Test Cases.

do bazel have something equivalent with that?

[jszumski]

Yes via --test_filter but I believe it's up to your test runner to use it correctly.

--test_filter=TestClass/testName should work with xctestrunner

[keith]

Please submit an issue if this doesn't work!

[wendyliga]

after looking to --test_filter, I find it only working for ios_unit_test and didn't work for ios_ui_test

from this example repo

bazel test ios/DummyTest --test_filter=ios_DummyTest_Source.DummyTests/test

this will run DummyTest and filter only DummyTests/test

but

bazel test ios/DummyUITest --test_filter=ios_DummyUITest_Source.UITest/test

do you guys have any idea? I try multiple different combinations of class bundle names and function, but none work.

[keith]

Can you show the output when your test runs so we can see the names of things to verify that matches? it's possible this isn't implemented correctly / at all for UI tests, I'm not sure

[wendyliga]

@keith this is the log, running full test

[keith]

Ah I think you need to update rules_apple / xctestrunner to include this fix https://github.com/google/xctestrunner/commit/24ba1517795e6e1c99d361988b95e51a48489087

[keith]

If your target names have dashes in them you also need https://github.com/google/xctestrunner/pull/42

[wendyliga]

I solved the issue by using this naming conversion

{bundle_name}/{class_name}/{function_name}

previous

bazel test ios/DummyUITest --test_filter=ios_DummyUITest_Source.UITest/test

working

bazel test ios/DummyUITest --test_filter=ios_DummyUITest_Source/UITest/test

[wendyliga]

but i got crash after that

Command line invocation:
    /Applications/Xcode_13.2.1.app/Contents/Developer/usr/bin/xcodebuild test-without-building -xctestrun /var/folders/tg/5hc1y5wn5qg4gnfhktnyt_kc0000gn/T/test_runner_work_dir.X5J7V7/TEST_ROOT/xctestrun.plist -destination id=874334EE-E8C9-4322-9CAC-F088B66C54DF -derivedDataPath /private/var/tmp/_bazel_wendy.liga/8284c4a94ccbbb579b9e33d50d39273c/execroot/__main__/bazel-out/darwin_arm64-fastbuild/testlogs/ios/DummyUITest/DummyUITest/test.outputs -resultBundlePath /private/var/tmp/_bazel_wendy.liga/8284c4a94ccbbb579b9e33d50d39273c/execroot/__main__/bazel-out/darwin_arm64-fastbuild/testlogs/ios/DummyUITest/DummyUITest/test.outputs/test.xcresult

User defaults from command line:
    IDEBuildOperationResultBundlePath = /private/var/tmp/_bazel_wendy.liga/8284c4a94ccbbb579b9e33d50d39273c/execroot/__main__/bazel-out/darwin_arm64-fastbuild/testlogs/ios/DummyUITest/DummyUITest/test.outputs/test.xcresult
    IDEDerivedDataPathOverride = /var/tmp/_bazel_wendy.liga/8284c4a94ccbbb579b9e33d50d39273c/execroot/__main__/bazel-out/darwin_arm64-fastbuild/testlogs/ios/DummyUITest/DummyUITest/test.outputs
    IDEPackageSupportUseBuiltinSCM = YES
    XCTHTestRunSpecificationPath = /var/folders/tg/5hc1y5wn5qg4gnfhktnyt_kc0000gn/T/test_runner_work_dir.X5J7V7/TEST_ROOT/xctestrun.plist

** INTERNAL ERROR: Uncaught exception **
Uncaught Exception: *** -[__NSSetM addObject:]: object cannot be nil
Stack:
  0   __exceptionPreprocess (in CoreFoundation)
  1   objc_exception_throw (in libobjc.A.dylib)
  2   -[__NSCFString characterAtIndex:].cold.1 (in CoreFoundation)
  3   -[__NSSetM addObject:].cold.2 (in CoreFoundation)
  4   -[__NSSetM addObject:] (in CoreFoundation)
  5   -[XCTTestIdentifierSetBuilder addTestIdentifier:] (in XCTestCore)
  6   -[XCTHTestRunSpecification initWithDictionaryRepresentation:error:] (in XCTHarness)
  7   -[XCTHTestRunCoder_V1 testRunSpecificationsInDictionaryRepresentation:removingPathPlaceholders:error:] (in XCTHarness)
  8   -[XCTHTestRunCoder_V1 testRunFileWithDictionaryRepresentation:removingPathPlaceholders:error:] (in XCTHarness)
  9   +[XCTHTestRunFileSerialization testRunFileWithDictionaryRepresentation:removingPathPlaceholders:error:] (in XCTHarness)
 10   +[XCTHTestRunFileSerialization testRunFileWithContentsOfURL:pathPlaceholders:error:] (in XCTHarness)
 11   specialized static XCTHTestRunFileSerialization.testRunFile(withContentsOf:derivedDataPath:) (in IDEFoundation)
 12   @objc static XCTHTestRunFileSerialization.testRunFile(withContentsOf:derivedDataPath:) (in IDEFoundation)
 13   -[Xcode3CommandLineBuildTool _resolveInputOptionsWithTimingSection:] (in Xcode3Core)
 14   -[Xcode3CommandLineBuildTool run] (in Xcode3Core)
 15   XcodeBuildMain (in libxcodebuildLoader.dylib)
 16   start (in dyld)

error: Result bundle at path /private/var/tmp/_bazel_wendy.liga/8284c4a94ccbbb579b9e33d50d39273c/execroot/__main__/bazel-out/darwin_arm64-fastbuild/testlogs/ios/DummyUITest/DummyUITest/test.outputs/test.xcresult does not exist.
2022-03-01 14:16:11,918 b''
2022-03-01 14:16:11,919 Deleting simulator 874334EE-E8C9-4322-9CAC-F088B66C54DF asynchronously.
2022-03-01 14:16:11,934 Done.

[keith]

i debugged to https://github.com/bazelbuild/rules_apple/pull/1369

[wendyliga]

@keith thanks for the response, so far to make it work

• use the same name for uitest target and swift module_name.
• filtering will work with --test_filter={class_name}/{function_name}, we do not need bundle_name anymore

[keith]

I wonder if using the bundle name == the module name would work as well

[wendyliga]

crash --test_filter={bundle_name}/{class_name}/{function_name}

works --test_filter={bundle_name}.{class_name}/{function_name}

[keith]

Yea I don't think the double slash variation is valid

[keith]

Note we just pass these through to xcodebuild so it's up to it to validate