Plugin/Module Support

[uberswe]

I want to have some kind of plugin/module system but I am not 100% sure what the best way to implement it is. The admin are could configure plugins for each site and then those plugins define how the admin area of the site should look. I know how Wordpress plugins work and I like the setup but I also know it's not scalable and the API is limited. I want something that meets the following criteria.

• Easy to understand and simple to create modules/plugins
• Can work behind a load balancer
• Can work across multiple hosts, the plugin or module can be on a different hosts from Beubo
• The API should be very open if possible so that we don't have to make new API methods as we implement new features

[uberswe]

Gobs and RPC seems like an interesting and good option for program communication https://golang.org/pkg/net/rpc/

[uberswe]

Is go plugins an option?

• https://golang.org/pkg/plugin/
• https://medium.com/learning-the-go-programming-language/writing-modular-go-programs-with-plugins-ec46381ee1a9

[uberswe]

Still not sure about go plugins, there seems to be some issues as described here https://www.reddit.com/r/golang/comments/b6h8qq/is_anyone_actually_using_go_plugins/

It seems that the intended use-case for plugins is in-tree plugins; when you have code you don't want to always include in the main binary, but you have no problem with it living in the same git repo and getting compiled at the same time. It does not seem to be a good way to let users/customers add in their own code; there are headaches if you try to use them that way; they have to sufficiently emulate your environment when compiling the plugin:

The plugin compiler version must exactly match the program's compiler version. If the program was compiled with 1.11.4, it won't work to compile the plugin with 1.11.5. When distributing a program binary, you must communicate what the compiler version you used is.

Any packages outside of the standard library that are used by both the plugin and the program must have their versions match exactly. This means that when distributing a program binary, you must communicate the exact versions of all dependencies; we accomplish this by publishing the output of GOOS=linux GOARCH=amd64 CGO_ENABLED=1 go list -deps -f='{{if not .Standard}}{{.Module}}{{end}}' ./cmd/the-program. Plugin authors will need to pin those versions in their go.mod.

If there are any packages outside of the standard library that are used by both the plugin and the program, then the plugin must be compiled with the same GOPATH setting as the program (even if using modules). This means that our users who use plugins must set GOPATH=/home/circleci/go, even though they don't have a circleci user.

In order to work on things like Alpine Linux's musl libc6-compat, everything must be compiled for compatibility with LSB 3. Setting _FORTIFY_SOURCE=2 with GNU libc causes the CGO 1.12 runtime to require LSB 4. Several distros (including Ubuntu 14.04, which CircleCI uses) patch their GCC to define _FORTIFY_SOURCE=2 by default. When compiling plugins, users/customers may need to fuss with setting CGO_CPPFLAGS to make things not fall over.

Because plugins use the libc dynamic linker, they force CGO_ENABLED on; cross-compiling is no-longer easy to do. Someone wanting to compile a plugin for the GNU/Linux program binary from their macOS workstation must compile the plugin in a VM (or Docker, since Docker for Mac runs in a VM).

[uberswe]

This should have been more obvious but gRPC is probably the way to go here https://grpc.io/

[uberswe]

Maybe plugins should just be compiled into beubo in the same way caddy and coredns does it https://coredns.io/explugins/

[uberswe]

I have decided to go with grpc and did an initial test implementation in afdbe3665a7b4f39d396bb2b17c5b5ea42e6b214

[uberswe]

I think "modules" and "plugins" can have different purposes and can be useful in different ways. When I write modules I am referring to go packages that can be compiled together with Beubo. When I write plugins I am referring to standalone applications that talk to Beubo over grpc. I think modules can follow the implementation and simplicity that Caddy has as described here.

A module can be useful for adding a small feature such as additional fields to page or site creation forms. However, a plugin can be useful for larger features where you might want to run these features in separate containers or instances. An example is an analytics plugin which only needs to listen to Beubo over grpc. Every time a web request comes in Beubo sends the details of the request to the analytics plugin over grpc and then forgets about it. Beubo can then quickly finish serving the request and the analytics plugin can take as much time as it needs to process the request. The analytics plugin can also have it's own separate database. If we need to show analytics in Beubo then this would be another grpc call from Beubo to the analytics plugin to fetch any data that should be shown.

[uberswe]

I think modules should be part of the core Beubo and plugins should be part of a module which I created a repository for here https://github.com/markustenghamn/beubo-grpc

I have removed all gRPC references from the core Beubo package.

[uberswe]

Just to clarify, I am using the go plugin package to create plugins. This functionality can then be used to add gRPC and REST APIs and whatnot.