Run tests for multiple apps in one run

[jonasboegner]

Is your feature request related to a problem? Please describe. We have multiple ui5 apps in our repository and want to run all tests in our pipeline. Then create one consolidated test status report and another consolidated coverage report. So in the end having two reports (status/coverage) instead of 2 for each app.

Describe the solution you'd like Specify an apps folder path (or regex at best) where each subfolder (regex result) is treated like the base folder for an app. Run all tests in those apps with options like --split-opa etc. applied and create central report files.

Describe alternatives you've considered Running each app individually and then aggregate the reports would also be possible like described here: https://arnaudbuchholz.github.io/ui5-test-runner/coverage/#aggregate-coverage-for-several-projects but handling this centrally would be much more usable.

Additional context We currently have 32 apps in such a structure (some larger and some smaller). Currently those are run parallel with karma atm.

[ArnaudBuchholz]

Hi @jonasboegner ,

This is an interesting request, I believe this is the second time I hear about it. As of now, I have another major update to the project (long story short, make the runner less bound to qUnit to support other test frameworks), I need to plan this.

In the meantime, I would like to explore deeper the impacts of handling multiple projects, do you mind if we discuss this in this thread ?

Some questions :

• There might be options that are specific to some apps (like aliases), how would you configure this ?
  • One idea could be to use the ui5-test-runner.json in each app folder
• How would you explore the folders ? maybe some do not contain app. I guess the search for apps should be recursive but then we need a clear way to identify which folders are "testable" (and prevent searching deeper).
  • Here again, the ui5-test-runner.json file presence could help
• Consolidating the coverage file can be done, as you pointed out, it is already documented. The main report, however, might be a bit more challenging. Keep in mind that ui5-test-runner also supports custom report generators. What would be your take on this ?
• Should we have an overarching global timeout ?
• For legacy testing (i.e. the runner is responsible of serving the app), this can be achieved but for remote testing (i.e. the app must be started first), this could be a bit challenging. What's your scenario ?

[jonasboegner]

1. Yes in each folder or if they should use a common config one in the base directory. Maybe even both? So configure common values in one central ui5-test-runner.json and app specifics in the app folders?

2. As stated above I would expect a regex search. We also have some folders not containing apps in our main "app" folder. I think ui5-test-runner.json file presence should not be the decider. The regex can handle recursive finds so everything returned by this should imo handled as app folder and an error returned in case the test runner cant run any tests for one result.

3. Just give the report all results (like it would be one app for coverage) just with the app possibly in the path of the covered file. For result reporting like html, junit etc. this should work as well as there are already multiple journeys reflected. Now there would only be more of them.

4. Possibly yes, but I think if we have a timeout for each app, that also should be sufficient and maybe be more stable. With more apps the uncertainty how long all those tests run becomes greater. A timeout for each app implies that the apps are run app by app, respecting the max count of parallelization of course.

5. We run in legacy testing, but for remote testing this should also work. The app specific tests just need to specify where to start the app from and the hosting needs to be handled externally.