Ideas for an architectural change to add new plugins (db)

[josejuanmontiel]

As comented in this issue here we can discuss options to implements new drivers (or plugins) in a fancy way that could select and reduce (in the build) dependencies, mantaining clear an easy compile/build the code for the standard cases.

I think those are the principal points for this: maintain easy (build and code) but reduce the size (if that it's important in some scenarios)

I read something like this in some place under the machinery proyect (maybe the unexplore for me now to the v2 experiment https://github.com/RichardKnop/machinery#v2-experiment) but backing to this proyect... i would like to explore two options:

1. The standard way in golang (i thought)
2. The wasm aproach e.g this one... that seems an aproach that some project took https://github.com/proxy-wasm/spec/blob/master/docs/WebAssembly-in-Envoy.md

Let's talk

[knadh]

I read something like this in some place under the machinery proyect (maybe the unexplore for me now to the v2 experiment https://github.com/RichardKnop/machinery#v2-experiment) but backing to this proyect... i would like to explore two options:

Machinery is nice. We've used it at work and ended up writing a light weight alternative, https://github.com/kalbhor/Tasqueue which specifically addresses some pitfalls in Machinery. However, it may not be ideal to have to write plugins/servers that depend on a distributed queue. Maybe a simple HTTP webhook system will do? POST from sql-jobber to arbitrary-HTTP-driver-webook endpoint to get results.

1. Standard Go plugin is a really poor system and is only meaningful in a closed environment. All "plugins" have to always be recompiled against master any time the master changes. I'd tried this approach in OTPGateway, but abandoned it for a simple HTTP webhook system because of how impractical and annoying this was: https://github.com/knadh/otpgateway/commit/97baef7e72d3ad6cae68fa0e1f0fe51129963f14

2. WASM is interesting. Haven't toyed with Go/WASM yet.

Thinking about it, a simple HTTP webhook system maybe ideal, just like what I ended up with in OTPGateway. Users can write an HTTP server in any language they prefer and needn't know Go.