Open jBorkowska opened 1 year ago
Scope: most of the stuff from #1341
In particular:
groups()
must be converted to native JUnit
/xcresult
test suites. In other words: closely replicate Dart test suite hierarchy on nativesetUp()
/ tearDown()
should work again
setUp()
s and tearDown()
s should worksetUpAll()
/ tearDownAll()
should work again
size: L
I'd do iOS first, because I'm more uncertain what we want to do is possible there.
iOS:
XCTestCase
subclasses, so that they're picked by Xcode's test runner.XCTestCase
is possibleAndroid:
XCTestCase
s cannot be nested. Best I've found is Grouping Tests into Substeps with Activities. We could try hacking it somehow Update: I tried and it isn't what we want.
We already dynamically create test case methods by XCTestCase.testInvocations. Now we need the equivalent of this method but for dynamic creation of XCTestCase
classes, and I didn't find any first-party solutions for it.
However, it might be possible to solve it with NSPrincipalClass
, but it very likely doesn't have access to (which we need to retrieve test cases from the Dart side). Update: I checked this and we indeed can use XCTest APIs from XCTest
APIs such as opening appNSPrincipalClass
. Fr example, we can make conform to XCTestObservation
and call XCUIApplication.launch
from XCTestObservation.testBundleWillStart
. Example below:
The test - nothing fancy:
Resources I've found:
NSPrincipal
:
XCTestSuite
– Typically, Xcode automatically manages test suites for you. Only use XCTestSuite if you need to define your own custom test suites programmatically.
– looks interestingXCTest
I managed to start the app under test (using XCUIApplication.launch
) from within PrincipalClass in its testBundleWillStart
callback (from XCTestObservation
, which PrincipalClass conforms to).
Next steps:
XCTestSuite
s can contain other XCTestSuite
s. We need nesting. UPDATE It's likely not possible.I just had a great talk with @zltnDC and he suggested a fresh approach to the test suites problem.
group()
s and lifecycle callbacks. This is already tracked in #1503, but currently as part of #1501.Recreating Dart test hierarchy natively is hard
And maybe even impossible (the research above isn't done). See SO question I asked.
Let's just not do it. Patrol's user doesn't care about how the tests are executed – if they're "flat", or if they're natively grouped. What they do care about is the test report. So let's just give them the test report.
Exampe test hierarchy:
Below is how the report looks like currently (copied from #1341). Let's call it "raw report":
test feature_bravo.1_test.dart ✅
test feature_bravo.2_test.dart ✅
test feature_delta.1_test.dart ✅
test feature_delta.2_test.dart ✅
but we'd like in to look like this (let's call it "enhanced report"):
group feature_bravo
├── test 1_test.dart ✅
└── test 2_test.dart ✅
group feature_delta
├── test 1_test.dart ✅
└── test 2_test.dart ✅
Good news is that the "raw report" contains all information needed to create an "enhanced report" out of it. We could provide a command like patrol reconstruct
to convert the .xml
s and .xcresult
s back to the original hierarchy from Dart.
raw_report.xml --> patrol reconstruct --> enhanced_report.xml
group()
s and lifecycle callbacksThe patrol reconstruct
described above doesn't solve this problem. But discussion about it belongs into #1503 and #1619 (https://github.com/leancodepl/patrol/issues/1619#issuecomment-1677359318).
patrol_cli
. We might be able to work around it by converting the report at the end of test suite, but that's a long shot.
[@bartekpacia here - I'm editing this message sometimes to keep it updated]
setUp
,tearDown
(done in #1721)addTearDown
setUpAll
(done in #1751)tearDownAll