search

hashicorp / terraform-provider-google

Terraform Provider for Google Cloud Platform
https://registry.terraform.io/providers/hashicorp/google/latest/docs
Mozilla Public License 2.0
2.33k stars 1.73k forks source link

Run SkipIfVCR tests during recording mode #19079

Open melinath opened 3 months ago

melinath commented 3 months ago

What kind of contribution is this issue about?

Pull request UX

Details

We should automatically run SkipIfVcr tests in "Recording" mode, instead of just skipping them. (We could implement this as running all skipped tests in "Recording" mode, since always-skip tests will still be skipped.)

We will likely need to also have a way to actually skip on PRs (but still run on nightlies) for specific tests that truly can't run in that environment (or for handling cases like stampeding herd of PRs accessing a specific API.)

Alternatively: We currently alert users when a test was added but not run (for example because it was skipped in VCR or is GA-only). However, we could try to detect all tests that are run and skipped in VCR and raise them directly where the reviewer will see them, so they can make a more informed decision about whether manual tests are required. The problem with this approach is that it would be easy to miss and require manual intervention by the reviewer to run and then check the tests.

References

https://github.com/GoogleCloudPlatform/magic-modules/pull/11291 introduced a failure in TestAccGKEHubFeature_FleetDefaultMemberConfigConfigManagement because it was not run on the PR.

melinath commented 2 months ago

https://github.com/GoogleCloudPlatform/magic-modules/pull/11153 introduced test failures because the tests were skipped in VCR.