ci: Add coverage tracking

pulumi / pulumi-yaml

YAML language provider for Pulumi

Apache License 2.0

38 stars 11 forks source link

ci: Add coverage tracking #469

Closed abhinav closed 1 year ago

abhinav commented 1 year ago

Adds a make test_cover target to the Makefile which builds pulumi-language-yaml with coverage instrumentation and runs all tests with coverage tracking.

For CI, adds an coverage option that, when enabled, will run make test_cover instead of make test, and upload the results to codecov. This option is only enabled for tests invoked for PRs and by the /run-acceptance-tests command.

Additionally, similarly to pulumi/pulumi#13334, this adds a workflow that runs tests on master with coverage tracking every 12 hours.

Resolves #464

abhinav commented 1 year ago

Current dependencies on/for this PR:

main
- PR #501
- PR #469 👈

This comment was auto-generated by Graphite.

abhinav commented 1 year ago

From local experimentation on master, it seems that this test is just really long-running. It takes 7+ minutes to delete the aws:ecs:Service for some reason, which eats up the bulk of the test runtime.

abhinav commented 1 year ago

Sounds like some AWS resources taking long to delete is a known thing. That explains the 30m timeout on make test.

codecov-commenter commented 1 year ago

Codecov Report

:exclamation: No coverage uploaded for pull request base (main@dfcddb9). Click here to learn what that means. The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #469   +/-   ##
=======================================
  Coverage        ?   61.90%           
=======================================
  Files           ?       30           
  Lines           ?     7533           
  Branches        ?        0           
=======================================
  Hits            ?     4663           
  Misses          ?     2485           
  Partials        ?      385

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

abhinav commented 1 year ago

Disabling commenting on PRs^

abhinav commented 1 year ago

This is mostly working, but something's off with integration test coverage data.

abhinav commented 1 year ago

Okay, so this turned into a yak shave.

https://github.com/pulumi/pulumi-yaml/issues/464#issuecomment-1672305641 has an update from 3 weeks ago on the tooling side. On top of that, even with the tooling, there was another problem: we SIGKILLed all plugins so we couldn't get plugin coverage data anyway.

However, with the changes I've made to pu/pu recently, namely the following, plugins can now be terminated gracefully.

I've stacked this PR on top of https://github.com/pulumi/pulumi-yaml/pull/501. It also needs a version of pu/pu with the above two PRs merged into it, after which binaries are able to produce meaningful coverage data.

abhinav commented 1 year ago

It works!

The graceful shutdown work in pu/pu recently allows pulumi-language-yaml to write the coverage data.