search

athena-framework / athena

An ecosystem of reusable, independent components
https://athenaframework.org
MIT License
211 stars 17 forks source link

Spec Coverage Reports #450

Closed Blacksmoke16 closed 1 month ago

Blacksmoke16 commented 1 month ago

After coming across an article from a discussion in Discord a while ago, I found a desire to implement code coverage reporting for Athena. Both as a means to improve the spec coverage, and to track flakiness of said tests. https://github.com/crystal-lang/crystal/issues/14880 would make the former even more powerful by taking some of the guess work out of testing the various macro code employed in the ecosystem.

The plan for this implementation is to leverage https://about.codecov.io/ as the destination of the coverage reports. This isn't set in stone, just seems to be one of the more feature rich solutions for this use case.

However, as per the discussion in the article, and in https://github.com/crystal-lang/crystal/issues/1157#issuecomment-2272283028, it's not as simple as calling crystal spec --cov or something along those lines. Because of this, in order to properly generate the coverage reports, we'll need to have an intermediary file to glob require spec/ of each component, then run that via kcov. This'll generate the report we need to then upload to codecov.

Given Athena is a monorepo, some extra thought must be put into how to best handle this the report generation, mainly given the requirements of codecov, and how CI currently runs the specs:

As I asked about in https://github.com/codecov/feedback/issues/304#issuecomment-2274973159, current plan is:

Use the components feature where by I'll run the specs/generate a report for each component into a dedicated directory that'll then be uploaded all at once via the official GHA which will handle assigning each report to the proper component.

I def have some experimenting to do, so will see how it goes.

Blacksmoke16 commented 1 month ago

After thinking about this more and starting on a prototype I think it's clear we should use components to track the coverage of each component itself. There is then the argument of should we use flags to break things down further? E.g. unit vs compiled specs? In short, I don't think so. AFAIK the main benefit would be seeing that say overall coverage is 95% but that's a result of unit test coverage being 100% and compiled test coverage being 0%.

However, because of the way CI is setup, it probably would make sense to do so just for the heck of it :shrug:. I.e. test_compiled will upload everything all at once, but with a compiled flag, while test_unit will upload everything all at once, but with a unit flag, and only for latest crystal on ubuntu because idt we really need one for each item in the matrix.