Allow enabling and disabling of filters and plugins with startup configuration

[KirilKabakchiev]

Currently, there is no way to disable specific plugins by using external config (for example env vars, flags, config file). However, there are often SM plugins that would talk to other systems which might often be missing or not accessible in some development environments.

Each filter/plugin should be configurable using the SM environment abstraction (which would mean that using env vars/flags/adding values to application.yml should allow disabling/enabling plugins/filters.

[dzahariev]

Better approach would be to do not have all plugins packed inside the binary, but to allow installation (provisioning) of plugins as part of application initialisation, or by triggering a call to the application itself. The plugins should run as separate processes - similar to gometalinter approach that gets and installs only required plugins and runs the separate processes for each installed plugin. This will bring:

• no "dead" code packaged inside binary - will provide a framework instead monolithic application.
  • only needed components are in place - small memory and storage footprints, better security, better stability, less complexity, faster startup and restart.
  • easy upgrade of separate plugins - each component can be upgraded (released) without requiring release of the Peripli SM.
  • downtime during upgrades will be minimised.

[KirilKabakchiev]

In gometalinter ( I just check the code) - here is what happens:

• they read up a config that says which linters will be used
• based on the config they download (exec go get commands) the linters
• then they put the downloaded binaries from the previous step in PATH
• then they run the binaries (exec them in separate go routines and wait each one to finish)

I don't see how this approach would be benefical for SM. Actually, a couple major drawbacks

• go get-ed binaries won't be in the dep toml/lock files which means no guarantee for reproduceable builds (half dependencies will be downloaded on runtime and won't at all be listed in SM dependencies). E.g. nobody can guarantee about the version of whatever is ran in the binaries.
• execing a binary produces an output. Our plugins take a requesta and return a response. I don't see how runing a binary and its output would be used to intercept efficiently http requests
• I don't think this would affect downtime during updates - if you want to install a new plugin or update an existing one (e.g. download another binary or update an existing one), you still need to restart the go process a.k.a. the application. There should be rolling-updates or blue-green deployments in place for that matter.
• Not to mention that each binary is configured with flags. sm itself is a binary and if each plugin it has to download is a binary (and has flags to be configured with) this would probably make SM's configuration pretty unmanageable
• There will be dead code inside SM's code - someone needs to translate the plugin binary's output to whatever SM intends to do with it which is different for each plugin - and this code needs to be inside SM's code (run a binary -> get output -> do stuff with it). And for the plugins that are currently not used, this will be dead code
• You would still need to update SM's code base if you want to update a plugin (well, depends what the update on the plugin is) - but if configuring the plugin (cmd args) were modified or if the output it produces has changed, sm code base will need to be updated.
• In general running your application multiple times should result in the same thing. The external binaries approach would break that which is really not good, not safe. For example one can release a new version of a plugin -> SM just restarts, because why not (vm crash or w/e) and then SM suddenly stops working because it has picked up a new binary for the plugin which is either corrupted or incompatibaly changed with the previous version)

Rather then downloading binaries i would prefer to have a config file in which we specified which of the already vendored plugins to be enabled/disabled. Then when testing SM, we can write tests that start SM with different plugin configurations. These tests will use the same plugin versions because the plugins are already vendored by the dependency management tool (dep/gomods)