Do not use the word "snapshot" for upgrade test data

[t0yv0]

Hello!

• Vote on this issue by adding a 👍 reaction
• If you want to implement this feature, comment to let us know (we'll work with you on design, scheduling, etc.)

Issue details

Multiple engineers are confused by how upgrade tests work specifically PreviewOnly and snapshot recording. It appears that the use of the word "snapshot" seems to lead people to believe these are golden tests, and recording snapshots should make failing tests pass by accepting the current behavior as the new correct behavior. This is currently not what's happening though.

Suggestion here is to try to come up with a better name and not use "snapshot".

I'm still thinking of a good name..

A verbose explanation of what the providertest system is doing, currently, when recording "snapshots". For baseline dependency version set (which is typically something like the last major provider release), the system runs the program against the cloud to observe the actual behavior and record what is happening. These records are then used to accelerate the tests by mocking out the cloud behavior of the baseline dependencies, so that the test can proceed without hitting the cloud and being slowed down by it. This is currently orthogonal to the test asserts.

Affected area/feature

[t0yv0]

CC @danielrbradley @VenelinMartinov @thomas11 I saw all of you making good suggestions here that "snapshot" is confusing.

[thomas11]

the system runs the program against the cloud to observe the actual behavior and record what is happening.

Sounds like a snapshot to me.

IMO some of the confusion is more about what an "upgrade test" actually does. It's an assert-clean-diff-against-baseline-recorded-snapshot test which is not the most elegant name :)