search

SlatherOrg / slather

Generate test coverage reports for Xcode projects & hook it into CI.
MIT License
1.55k stars 236 forks source link

Slather incompatible with Variant tests #514

Open gaige opened 2 years ago

gaige commented 2 years ago

We are using variant testing in some of our test plans (for example: one plan requiring USan and another using ASan), which results in the following Build directory structure:

Build/Products/Variant-ASan-UBSan/Debug-iphonesimulator/CTUtilsTest.xctest
Build/Products/Variant-USan-UBSan/Debug-iphonesimulator/CTUtilsTest.xctest

In this case the find_binary_files fails to find the variants (reasonable, since there's no way to tell it which one to choose anyway). This problem may be especially acute if using build_without_testing, since you're guaranteed to have the variants carried along in that case.

I've not found a way to be more specific about the binaries in order to accommodate this, so we're running a separate set of coverage tests (which should yield the same results) with a single test binary, as opposed to requiring the variants. However, there may be others who run into this in the future, and I figured I'd see if I could save someone time at least in acknowledging this.

I'm not sure exactly what the fix is here, other than to provide a way to specify variants explicitly.

andrewsexcellent commented 1 year ago

Running against the same issue, the tests create Build/Products/Variant-ASan and then Slather fails with No product binary found