Increase thoroughness of Provider conformance tests

[negz]

What problem are you facing?

This repository should define two conformance test suites; one for Crossplane distributions and another for Crossplane providers.

How could conformance help solve your problem?

It could flesh out the provider conformance test suite, which currently exists but has no meaningful tests. We should test that:

• [x] Providers install only cluster scoped CRDs.
• [x] Providers install only CRDs that appear to be a managed resource or a provider config.
• [x] Providers install support for at least one managed resource.
• [x] All defined resources are in the "crossplane" category, and managed resources are in the "managed" category.
• [ ] A ProviderConfigUsage exists for each extant (managed resource, provider config) tuple.
• [ ] All managed resources can become 'ready'.
• [ ] All managed resources emit Kubernetes events.
• [ ] All managed resources write connection secrets (though for many these secrets would be empty).

(Edit: consolidated some of the comments below into a checklist here).

[negz]

It's not clear to me whether we should (and how we would) test how managed resources actually behave, given that we don't know which managed resources a provider defines. We could consider requiring that one of each kind of managed resource supported by the provider be created (manually) before the conformance suite is run. This would allow us to test that all managed resources defined by the provider:

• Can become 'ready'.
• Emit Kubernetes events.
• Can write connection secrets (though for many these secrets would be empty).

[negz]

Some other things we'll want to test:

• All defined resources are in the "crossplane" category, and managed resources are in the "managed" category.
• A ProviderConfigUsage exists for each extant (managed resource, provider config) tuple.

[negz]

I'm somewhat skeptical WRT testing for event emittance, just because if there's a bunch going on in an API server it's possible for events to expire and disappear pretty quickly. This may not be an issue given that we don't really expect folks to be testing for conformance using a live Kubernetes cluster.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.