Backends as plugins

[alexott]

Right now, there is a tight coupling between backends and sigma-cli. Although you're defining output formats in the backend, to use them with sigma-cli, you need to modify its code to add new output format(s). It would be easier if backend provided that information to sigma-cli, so no code modification is required.

[thomaspatzke]

Indeed&agreed! This was already the case with good old sigmatools and putting the descriptive information into the backends fits better there than in sigma-cli. I will move them over to the backends and keep this issue open for tracking until done.

[alexott]

Great! If necessary I can help with that. It would be nice to automatically discover backends and pipelines as well. I’m looking for a ways of doing that

[thomaspatzke]

Autodiscovery was also already there in sigmatools.

In sigma-cli I wanted to make this explicit, because backends are external packages here. This makes it hard for the developer to spot, which backends are active there and there is a higher risk that something done in such a backend package could break the autodiscovery. While the monolithic architecture of sigmatools made these issues negligible, they're real for a modular setup like in sigma-cli.

And finally, the Zen of Python states (import this):

Explicit is better than implicit.

😉

[thomaspatzke]

Added it to the backend base class and to the cookiecutter template.

[alexott]

explicit maybe could be better as advice, but right now to add a new backend I need to fork sigma-cli, hack its code, etc. This make extension of it harder, especially in initial phases of development, or when using it for some internal tooling.

[Samrose-Ahmed]

A plugin system would be nice where a user could just do something like sigma add <backend> to add a backend. That would be explicit as well.

[thomaspatzke]

A plugin system would be nice where a user could just do something like sigma add <backend> to add a backend. That would be explicit as well.

After some consideration I think this could be a really good idea because it solves the issue that Sigma CLI releases with major pySigma updates currently depend on the fact that all backends are on the same pySigma release. This adds an increased amount of coordination betwen contributors. This could especially get annoying if backends are abandoned or contributors start to get unresponsive for any reason.

[kelnage]

We're also currently in the process of finalising a release of a backend we have developed, and we were wondering if there was a ETA for the changes around including backends? We're happy to do a PR for sigma-cli in the meantime, but we would like to share our work in the near future if at all possible!

[thomaspatzke]

I've already started to make backends, pipelines and validators pluggable in pySigma. It needs some further work and integration into the CLI. Unfortunately my opensource time was a bit short in the last weeks, but starts to increase now again 😉 Therefore it's always hard for me to give an ETA, but it's in the top three of my Sigma todo list.

[vandir]

Please consider supporting config arguments of custom backends (ref: base.py), that is a way to pass a configuration to the backend from the command line.

[thomaspatzke]

Just created the release for pySigma 0.9.0 that provides a plugin system for backends, processing pipelines and validators. Already started to integrate this into Sigma CLI in the plugin branch. Backend and pipeline projects require some minor updates. Basically, the information contained in the CLI repository has moved to the plugins, as proposed by @alexott.

[thomaspatzke]

pySigma 0.9 is now integrated in the recent versions of Sigma CLI and backends/pipelines are now plugins that can be managed with the sigma plugin subcommand.

[alexott]

Thank you Thomas!