Create a validation/certification tooling.

[jieyu]

The tooling should be in the spec repo so that when we ship a release of the spec, we also ship the corresponding validation/certification tooling.

The goal of the tooling is to test if a CSI plugin is CSI compliant, and conform to the spec.

[akutz]

Hi @jieyu,

I'd like to take a stab at this. Is there a preferred language for the tooling? I think the first step is to determine the ruleset around the tooling. For example:

1. Should it be able to be executed locally?
2. Is it CI-specific?
3. Should it exist for multiple languages?

Etc.

[julian-hj]

We (Cloud Foundry) were intending to donate our certification test suite to CSI, as soon as we have bandwidth to update it for compatibility to the latest version. The intent behind that suite is that it can act as a mock CO and interact with a volume plugin running a real service to take it through the "happy path" flows. It should not require any part of a CO in order to run.

My take on your questions above:

Should it be able to be executed locally?

yes. It should assume network access to a CSI plugin somewhere. The onus would be on the plugin author to install and start up their plugin, and then to run the cert suite somewhere where it can get access to the plugin.

Is it CI-specific?

No. It should be possible for a developer to kick it off. Presumably we will want to eventually hook it up to Jenkins to run CSI against a dummy plugin when folks make changes to the CSI spec repo, but I'd think we just want to make it run manually first. The cloud foundry one is just a ginkgo test suite.

Should it exist for multiple languages?

My vote is that we definitely don't want to maintain multiple certification suites in different languages. Since changes with the spec will typically require corresponding changes to the certification tests, having multiple sets of cert tests would result in a fair bit of overhead later on.

[julian-hj]

AIs:

• Luis will submit an initial PR adding his csi-sanity tests to this spec repo.
• After that PR is accepted, the Cloud Foundry team will follow up with additional PRs to
  1. include Andrew's mock plugin and
  2. introduce fixture files to allow the tests to run against plugins requiring inputs like parameters to the CreateVolume ControllerPublishVolume or NodePublishVolume RPCs in order to function properly.

[julian-hj]

@lpabon checking in...are you still planning to PR csi-sanity?