Add a flag to skip every test that requires credentials

[t0yv0]

Hello!

• Vote on this issue by adding a 👍 reaction
• If you want to implement this feature, comment to let us know (we'll work with you on design, scheduling, etc.)

Issue details

It will be useful to have a way to run all tests that do not require cloud credentials. If not useful internally, I'm sure it will be appreciated by third-party contributors who do not have time to setup test credentials. Currently the framework is setup to run some tests that do end up requiring credentials (E2E quick, PreviewOnly upgrade), but it already is starting to have some tests that can exercise the provider without requiring credentials (PreviewOnly quick) and we expect to have more of these in the future, especially if they are snapshot based.

The concrete suggestion here is to implement a flag, perhaps --provider-unit or --provider-no-creds, that would skip all tests that would fail if credentials are not present. This flag can later on support a Makefile target or some such helper.

Affected area/feature

[t0yv0]

@danielrbradley quick idea from this morning discussion. It's not strictly necessary but I think it can be useful.

[danielrbradley]

I think this is tricky to know because it depends on the provider implementation. Some providers require cloud access to validate credentials during their Configure call while others only check that required fields are set.

[t0yv0]

Perhaps the provider could mark or communicate this to the framework.

Or perhaps we could go the other way and add flags that make providers skip cred verification specifically so that they would be more amenable to testing under this flag.