search

fermyon / spin

Spin is the open source developer tool for building and running serverless applications powered by WebAssembly.
https://developer.fermyon.com/spin
Apache License 2.0
5.2k stars 247 forks source link

Code coverage flakes #1061

Closed itowlson closed 6 months ago

itowlson commented 1 year ago

We're getting (at least I'm getting) a lot of flakes in the code coverage / Cargo Tarpaulin action. Seems like there's a specific integration test that passes in the normal test runs, but under Tarpaulin fails to connect to its spawned Spin.

@FrankYang0529 I think you are the person who knows about the code coverage stuff - would you be able to take a look please? Thanks!

itowlson commented 1 year ago

Related thought: given that the integration tests already run as part of the build action, could we skip running this rather lengthy action on every iteration of every PR, and instead do it like the audit one, a distinct action running on a schedule against main only in the Fermyon org? Are we actually using the PR-level coverage info?

cc @vdice who was involved in the original code coverage discussion/planning

vdice commented 1 year ago

👍 I'm in support of restricting the code coverage action to run on main only for the near future. Cron approach, as used by audits, seems appropriate.

We talked about utilizing the coverage results on PRs as a follow-up, but as 1. the check isn't entirely reliable (eg #1063 ) and 2. it takes quite a large amount of time to run, I agree it should harden against main first, where we can address both points, before incurring costs to contributors/PRs.

itowlson commented 1 year ago

@FrankYang0529 I have skipped the problematic test for now but if you have any ideas on how to reinstate it that would ge great - thanks!

itowlson commented 6 months ago

Not sure where this ended up but it's been inactive for over a year, and CI seems as happy as CI ever is, so closing.