Test suite overhaul - Githubissues

The current test suite does not scale. It takes ~1 minute to run tests on a powerful machine, and more on a less powerful machine, which includes the CI machines which might have 2-4 vCPUs at best.

Performance

~- [ ] Large amount of time is spent in invoking maven commands, although most of the time the dependencies are in the local folder. dart-lang/native#742 should fix this.~ ~- [ ] javac is invoked in a few test cases, this can be cached transparently too.~

Update: These seem to take some time during binding generation, but do not occupy large part of our testsuite.

[ ] More tests should be able to be run from same summary. Probably we should implement dart-lang/native#678 .

~Overall, not running any Java based tool in the fast path should be the goal.~

Testing summary

[ ] Summary golden file tests: This can be moved from JUnit/Java to dart, since we are using git diff --no-index anyway to compare. This should also enable us to add more summary tests.
[ ] Summary characteristic tests: Pattern match known details on summary, since most details will be common between asm and doclet summaries.

Examples

Unfortunately some examples are rather android-specific and cannot be frequently run in usual dev workflow.

But pdfbox standalone test can be probably run.

[ ] Create a program which creates a minimal PDF using PDFBox, and run the other (currently existing example) on the same PDF file.

CI

I believe CI shouldn't take 10+ minutes and there should not be 17 CI jobs. (Except those which eg: upload the results).

The worst offender seems to be the android build jobs.

[ ] Explore adding more caches to long-running CI jobs.
- ?? No idea if this handles SDK updates well. ??

~- [ ] Some long running Android builds install multiple versions of the NDK. Remove as many NDK constraints as possible. (Shouldn't everything build with same NDK?).~

[X] check_java_format and test_summarizer should not be different jobs.

~- [ ] test_jni can run in 20s, it's better to merge this and analyze_jni.~

~- [ ] Is there a technical reason why collect coverage is run separately once all tests pass? I think coverage can be collected during first time itself?~ (Error reporting)

dart-lang / native

Test suite overhaul #679

Performance

Testing summary

Examples

CI