Plugin Contract Compatibility Check

[erikerikson]

When plugin related contracts are broken, the contract against which a plugin is implemented must be declared and verified for compatibility by artillery-core or else plugins will cease to work without notice to plugin maintainers or guidance to users experiencing a breakage related to a contract incompatibility.

[erikerikson]

we saw this when a project moved from using artillery ^1.8.0 to using ^2.4.6

[hassy]

Yup, agreed. What sort of a mechanism to prevent these breakages would you suggest?

One possibility would be to allow plugins to declare the version of Artillery they have been tested with - that would allow for a warning message to the user at least if a possible incompatibility is detected. (Note that the breaking change was between major semver releases.)

Another solution would be to version the plugin interfaces and require that plugins declare which version they use.

I'm open to any & all suggestions!

[erikerikson]

You absolutely complied with semver. 😀 Definitely just trying to start a productive conversation with this.

I think I would prefer that plugins be queried via a unchanging (across contract versions) static (used prior to construction) method that returns a contract version id or an array of contract version ids. It might be helpful to make the contracts more explicit in concert. Every plugin without this method could be considered version 1- compatible. This would allow Artillery to detect and message about incompatibility or offer a compatible interaction. The plugin loader would need to be robust to incompatible plugins of course.

Another possibility is that the framework or plugins or both could be robust to multiple contract versions. A selected contract ID passed during construction could allow plugins to load the appropriate code for the declared contract. Potentially an array of contract IDs could be given to the constructor instead which could respond with the contract it selected and intends to comply with. The plugin loader could thereby be offering the last N contracts or something like it and plugins could work with multiple versions of artillery-core.

Of course, any of this can be a bit of a PITA but... Usually better for longevity. Maybe there's a npm module for this? Haven't checked yet.

[hassy]

It is an interesting problem to think about, semver really was of no help here since plugins do not depend on core explicitly. Maybe another possible solution is a separate plugin-interface package, to act as a proxy between plugins and core. Version compatibility checking is then "free" in that the major version used by a plugin will have to match that used by Artillery, otherwise a warning can be output.

[erikerikson]

That's interesting. It seems like it might block a robust plugin from working with multiple contracts (for disparate users).

Use case:

User one loves artillery-plugin-A, compatible with contract 1. User two loves artillery-plugin-B, compatible with contract 2.

Both use artillery-plugin-C and that plugin's creator handles contracts 0, 1, and 2.

That way both user one and two can use the same npm installable artillery-plugin-C.

On the other hand, it could allow artillery to handle contract compatibility/translation issues transparently for plugin producers. That could get painful for you over time.