Comparison to junit4git / storing Test Impact Analysis as Git notes

[sschuberth]

This looks like a great project, and reminds me quite a bit of https://github.com/rpau/junit4git! It'd be interesting to get some comparison about the differences / commonalities regarding the features and technical approaches.

One thing that comes to my mind is the IMO very smart idea of junit4git to store Test Impact Analysis data not as a file that you need to commit (creating a new Git revision), but as a Git note that can be attached to exactly the commit it was generated for, and thus shared easily, even for past commits.

Is that a feature that could be considered for Skippy as well?

[sschuberth]

Reading about the latest 0.0.19 release, this sounds like a good candidate for a SkippyRepositoryExtension interface implementation.

Though if the data can really grow that large, maybe leveraging GitHub action cache would be more feasible... probably https://www.skippy.io/docs/#skippy-in-your-ci-pipeline would be a good place to add docs on how to cache the .skippy directory with GitHub actions without adding it to version control?

[fmck3516]

This looks like a great project, and reminds me quite a bit of https://github.com/rpau/junit4git! It'd be interesting to get some comparison about the differences / commonalities regarding the features and technical approaches.

I was not able to run junit4git. Reason: https://github.com/rpau/junit4git/issues/20

That's why I can't provide a comparison. Let me provide my design goals instead:

• Both the build plugins as well as the JUnit extensions must be thin wrappers around a build and testing tool agnostic core. This allows me to maintain a variety of build and testing tools with little overhead.
• No home-grown byte code analysis. While that’s most probably fun to implement, it’s not sustainable to do so for a small project. I’d rather leave the byte code analysis to the JaCoCo folks.
• Use a mix of static and dynamic analysis. Tools that entirely rely on static analysis struggle with reflection. Tools that entirely rely on dynamic analysis struggle with inlining of code by the compiler (e.g., they don’t detect changed constants).
• Provide extension points in cases where I can’t think of a good 1-size fits all solution (e.g., where and how to store the test impact data).

One thing that comes to my mind is the IMO very smart idea of junit4git to store Test Impact Analysis data not as a file that you need to commit (creating a new Git revision), but as a Git note that can be attached to exactly the commit it was generated for, and thus shared easily, even for past commits.

Regarding storage as Git note: My reasoning was: Why depend on Git if you can easily come up with a solution that is independent of the VCS? That being said, supporting other VCS systems is not a prioriy.

My workflow looks like this:

1. Work on a feature
2. Run the tests
3. Commit & Push

In this scenario, you don't need a separate commit to update the impact data.

Is that a feature that could be considered for Skippy as well?

Yes, absolutely: https://github.com/skippy-io/skippy/issues/158

Reading about the latest 0.0.19 release, this sounds like a good candidate for a SkippyRepositoryExtension interface implementation.

I will take a stab at writing an extension that stores the impact data as Git note.

Though if the data can really grow that large, maybe leveraging GitHub action cache would be more feasible... probably https://www.skippy.io/docs/#skippy-in-your-ci-pipeline would be a good place to add docs on how to cache the .skippy directory with GitHub actions without adding it to version control?

Thanks for the pointer - I haven't used this cache before. I'll check it out.

[fmck3516]

Though if the data can really grow that large

Just looked up some stats: For the Spring Boot core repository (I use a fork as my benchmark), the size of test-impact-analysis.json is roughly 800 kilobytes of data across 17k lines.

[sschuberth]

Thanks for the detailed answers! I believe everything was addressed to the extent possible, and / or there are follow-up issues, so feel free to close this. It's a very good reference post in any case!