CI: Daily execution on a selection of real source codes

[murdos]

Currently openrewrite and its official plugins are only tested with unit tests. I think it would be really beneficial to the project if recipes were tested on a daily basis on real code.

This would allow:

• checking java compatibility. For example java 8 compatibility has already been broken, but it was detected by users, not by automated testing.
• checking that recipes generate code that still builds
• checking that recipes generate code that keep the same behavior: the unit tests of the real code should still run successfully after code transformation

My vision of how it could work is, for a set of selected real open-source projects:

• checkout external project source code
• configure tooling (jdk version, maven or gradle)
• setup openrewrite maven/gradle plugin
• configure plugin to execute all available recipes that don't require parameters (or a predefined list)
• execute rewrite on project
• launch build and tests on transformed source code

The selection of external projects should be based on their characteristics (maven or gradle build, java version, ...).

NB: In a certain way, this is a bit similar to https://github.com/hipster-labs/jhipster-daily-builds

[yeikel]

Should this be Daily or as a standalone check with every PR? (as a set of separate GH action checks)

If there are no changes, we shouldn't schedule any run

[murdos]

These checks often take a lot of time, so running on every PR (change) is probably not a good idea.

[yeikel]

These checks often take a lot of time, so running on every PR (change) is probably not a good idea.

If we do a side job, we'll need to monitor it very often(or receive alerts) to avoid regressions

Maybe we can still have a small CI tasks to do a "smoke test" such as the check for compatibility with Java 8

[timtebeek]

At Moderne we've since implemented internal monitoring, where we run recipes daily on a large collection of projects, with alerting, to ensure there's no (or at least fewer) unnoticed regressions. Thanks for the suggestion!