Plugin system for obscure/non-universal features

ethantkoenig commented 7 years ago

I propose adding a plugin system to gitea, which would allow gitea users and developers to add new features while keeping the core gitea codebase small and maintainable. Please read my proposal, and let me know what you think.

Motivation

As gitea grows, there will be requests for more and more features. Many features will be relatively peripheral; while necessary/useful to some users, they will not be used by other users. An example of such a feature is #2211. I'm afraid that gitea will eventually become a bloated collection of incoherent features. Even if these features can be enabled/disabled in a config file, they still make the codebase more complicated and harder to maintain.

Meanwhile, many users of gitea may want to perform small-scale changes and customization to their personal forks. These could range from logo changes to small features that are for their specific use-case (and thus wouldn't make sense in upstream gitea). It would be nice if such users could make these changes without facing merge conflicts when they pull from upstream.

I believe plugins could solve both of these problems.

Specifics (I'm open to suggestions and discussion)

Plugins would be in-process (included during compilation). Plugins would interact with "core" gitea in the following ways:

Plugins can update templates (this would happen during start-up). This allows plugins to add new UI elements
Plugins can register new routes
Plugins can install listeners/callbacks that are triggered when various events happened (e.g. new content is pushed to a repo, which could be useful to a plugin for full-text repo search)
Plugins can add new tables to the database. Unfornuately, there doesn't seem to be a clean way for plugins to alter existing tables, since they corresponding struct in "core" gitea would need to be updated.

Unanswered Questions

What if two plugins rely on different version of the same dependency?

I don't have an answer right now, but hope to investigate possible solutions soon. The answer also depends on whether we end up switching to golang/dep (#2218)

sapk commented 7 years ago

We could use the plugin method introduce in Go1.8 : https://golang.org/pkg/plugin/

sapk commented 7 years ago

Futhermore, I propose :

Plugins can add command line command (like: import github/bitbucket/gogs)

This would permit to add script from community wihout bloating main prog.

lafriks commented 7 years ago

@sapk Go plugins are supported only on Linux

sapk commented 7 years ago

@lafriks I have to correct what I have said on compiled plugin and vendored deps : https://github.com/golang/go/issues/20481

This should be fix but not yet ready.

sapk commented 7 years ago

Windows and macos plugin build are an objectif but not yet planned and no one on it. :disappointed: I would still suggest it and explain that it is only possible for linux host in the wait of golang improvement. Maybe there are better solutions ?

andreynering commented 7 years ago

Gogs cross-reference: https://github.com/gogits/gogs/issues/2438

First, I completely agree the we should not bloat Gitea with more and more small features.

About the plugin system, it's also hard to implement. Go has interpreters for other languages we could look:

JavaScript: https://github.com/robertkrimen/otto
Lua: https://github.com/yuin/gopher-lua
Lua: https://github.com/Shopify/go-lua
Bash: https://github.com/mvdan/sh (this one may not be suitable for the plugin system, but for allowing running bash commands that will run on Windows, too)

We should look for an approach that is either easy to maintain in the Gitea core and also for plugin developers.

sapk commented 7 years ago

https://github.com/hashicorp/go-plugin suggested on gogs issue is also a good solution. It support some few good feature like negotiation for compatibility between main and plugins (and is planning "plugin fetching"). And plugin could be develop in various languages just need gRPC interface.

sapk commented 7 years ago

gRPC permit support for plugin triggerring event and not wait to be called.

jonasfranz commented 7 years ago

I think that we only need a better filtering of features and we have to know what we want to implement and what not. But I disagree with you statement that #2211 is only used be a "small minority of users" because it makes gitea interesting for small businesses that need such features. And my apprehension is that the features will not be such good integrated or have a good quality if they come via plugin. I agree that there should be a simple possibility to add small additions like #2221 but I do not think that it would be a good option for (in relation) big features like #2211.

lunny commented 7 years ago

I agree with @JonasFranzDEV enhancement of issues will let small company remove their redmine and such others.

ethantkoenig commented 7 years ago

@JonasFranzDEV @lunny Apologies if I came across as unfairly criticizing the timetracking feature. I didn't mean to say that the feature is useless, but only that it is peripheral to the main purpose of gitea, and that large subset of gitea users do not need it. I've updated the wording in my description accordingly.

I'm not sure "we just need to do a better job deciding what to implement" is a viable solution. Who decides what is worth implementing? What if user X needs feature Y, but it is decided that feature Y will not be implemented? User X's only option is to implement feature Y on his/her own fork, which is problematic as I described in the description.

Finally, I disagree with the "too big for a plugin" argument. The fact that the timetracking feature is large means it will add even more overhead and complexity to the codebase, which I believe further underscores the need for moving it to a plugin.

lafriks commented 7 years ago

@ethantkoenig plugins are good thing to have but also come with additional problems like it is harder to change core functionality/design/structure as such changes can and most probably will break plugins and is harder to test

sapk commented 7 years ago

@lafriks It also give us room for change because with too much function integrate in core we lock the structure of code and giving a standard interface (with versioning) for plugin will allow function to evolve at there own rythm. If a user want to keep a func implemented by a plugin that does not support the last version of core, it become the choice of the user to stay with an older version in the wait of an update of the plugin (or contribute to the update).

I think we all agree to had plugin, we just need to decide what type and how to limit bruden for gitea and plugins developer by starting a roadmap and defining interface and what a plugin can do.

lafriks commented 7 years ago

I agree that there could be plugins for webhooks/ci integration but for extending core functionality and UI it will be too much work to implement that correctly, especially to support multiple platforms etc. Also if plugins starts to be created that adds UI functionality we will start to get bug reports that will be hard to track down if bugs are only reproducible with plugin or combination of plugins enabled.

I think that we should continue to keep Gitea easy to install, run and packed everything is single binary. To make plugins right it would mean to rewrite gitea core to allow that and most probably it would be quite big task with no direct benefit.

tboerger commented 7 years ago

I just want to mention that we already had https://github.com/go-gitea/gitea/issues/661 ;)

tboerger commented 7 years ago

Now my two cents for this topic. I wish we had an grpc based plugin system that is comparable to packer by hashicorp. Of course it's a lot of work, but we can even simplify the core and extract more parts into plugins. To avoid problems with plugin versions we have to apply versioned apis.

When we built the basics we can even offer some kind of marketplace where everybody can offer his stuff.

jonasfranz commented 7 years ago

@tboerger Will the plugins be compatible to newer / older "versioned apis" if gitea updates and the plugin not? Which parts of the core should be extracted into plugins?

tboerger commented 7 years ago

@tboerger Will the plugins be compatible to newer / older "versioned apis" if gitea updates and the plugin not?

That's the whole thing of versioned APIs, older plugins should work with newer core versions, they just can not use newer core API parts.

Which parts of the core should be extracted into plugins?

No fixed list, maybe wikis, timetracking or other future additions like pages and so on. Keeping the core minimalistic and extracting many things into plugins should bring in some additional overhead, but every feature/plugin can be developed and published independent of the core as long as the core already provides the API for it.

jonasfranz commented 7 years ago

I think that a concept like drone's plugin system might be a good idea. But we need a better API since we do not only provide "build steps". So my idea is that every plugin is started as docker container and connected to the plugin API via gRPC like @sapk pointed out (https://github.com/go-gitea/gitea/issues/2222#issuecomment-318653276). There is no installation process for the admin like JDK installation or other dependencies. But perhaps is this concept a little bit to complex.

daviian commented 7 years ago

Since we don't depend on Go 1.7 anymore, at least I thought so we could use the introducted go plugin architecture from 1.8.

A blogpost as an example: https://medium.com/learning-the-go-programming-language/writing-modular-go-programs-with-plugins-ec46381ee1a9

lafriks commented 7 years ago

@daviian we can not use built-in golang plugin architecture as it is not supported on Windows and other gitea build targers

lunny commented 7 years ago

@daviian we should declare v1.3 only support go1.8+ on README or something place.

tboerger commented 7 years ago

I would use some plugin system like packer, that's also grpc based and works also on Windows.

sapk commented 7 years ago

I could try to make a test with https://github.com/hashicorp/go-plugin. Any idea of a plugin that could be implemented ? I was thinking of adding a file interpreter like markdown.

For the plugin architecture, I was thinking of allowing plugin to declare file handling based on regex/ file extension, declare url to be able to responds, declare event to listen for, be able to override some defined function.

lavvy commented 7 years ago

@sapk There are a ton of plugins here http://gitbucket-plugins.github.io/

techknowlogick commented 6 years ago

Another plugin system we could consider is the one that Caddy uses: https://github.com/mholt/caddy/wiki/Extending-Caddy

For the webserver for the docs site it is already using Caddy and its plugin system: https://github.com/go-gitea/gitea/blob/master/docs/Dockerfile#L8

tboerger commented 6 years ago

A compile time plugin system is not something I really like

stevegt commented 6 years ago

Trying to summarize this thread so far -- let me know what I'm missing. I see two viable alternatives for a plugin system:

A single-process internal plugin system similar to Caddy's, where everything runs in one process and each module registers its presence at startup. We could optionally refactor some or all of existing Gitea to make "everything a plugin". In addition to enabling unforseen features while keeping core code small, it enables less risk in refactoring major subsystems, since it encourages the use of internal API between modules.
A microservices architecture, where plugins each run in their own process and use the swagger API to interact with gitea and its db, which keeps gitea monolithic and its bugs relatively well-defined. This also means that, for instance, any plugin would have to maintain its own db for any custom tables or columns, and each plugin might best be run in its own container. We'd provide HTTP callback hooks so gitea can trigger plugins to do their jobs, and plugins would have to have some way of modifying Gitea's HTML.

Variations on these two themes might include runtime versus compile-time linking of the single-process system, or some RPC protocol other than HTTP for the multi-process system -- I'd describe a POSIX IPC plugin system as a variation of the second option, for instance. It would also be possible, though probably not desirable, to break existing Gitea up into microservices as well -- just tossing that out there as a strawman so it can get knocked down for performance reasons. ;-)

But in general, do these two pretty much cover all of the alternatives discussed so far? (I can't imagine any other category being computationally possible, but am willing to be surprised.) ;-)

If nobody can think of anything else, then we may want to split these two alternatives into two different issues, and see where they lead. Right now it's hard to tell which way the consensus is going, I think because we're actually talking about two completely different definitions of what a "plugin" even is.

Thoughts?

jonasfranz commented 5 years ago

I've tested the built-in plugin system of golang and discovered that it might fit very well in our use case. Plugins will be using the already existing database functions in models without having a seperate plugin api interface.

Plugins are going to be ~39MiB big since they include the gitea sources. This is a problem in my mind. Any ideas on this?

tboerger commented 5 years ago

The builtin plugin system doesn't support windows, or has that changed?

Fardinak commented 4 years ago

Plugin support has indeed improved since 1.8, but Windows is still unsupported at this time and it doesn't seem like there's any active effort towards making it happen. However the community seems keen on accepting a pull request from someone who knows actually how to implement it…

Plugin support in v1.13

Currently plugins are only supported on Linux and macOS.

Plugin support in master

Currently plugins are only supported on Linux, FreeBSD, and macOS.

And for reference, here's the issue regarding Windows support: golang/go#19282

Personally I'm mostly in favor of golang plugins or some sort of scripting. RPC just seems like too much trouble to manage. But in any case, I'm looking forward to plugin support.

zeripath commented 4 years ago

Oops, didn't mean to close this.

One thing I think would help would be a more extensible markdown renderer. I understand Hugo has moved to the goldmark renderer.

tboerger commented 4 years ago

I still think it would make sense to implement the plugin api based on grpc and use the hashicorp plugin lib, this is already heavily battle tested.

lunny commented 4 years ago

https://github.com/hashicorp/go-plugin allow only an executable binary plugin and running on the local network. So the plugin have be run on a sandbox(i.e. docker) to avoid it read/write filesystem/database directly.

sapk commented 4 years ago

@lunny I also think that the grpc solution is good because it allow local and remote. If plugin dev choose to acces database and file directly when run locally it is their concern. Maybe later we could choose to isolate (chroot, container, ...) local plugin but I think it would be extra first since it would be platform dependant.

lunny commented 4 years ago

@sapk I don't think that's their concern. I think the security should be considered when we design the plugin system. Once you choose to enable plugin system, gitea could require you installed docker or k8s. All plugins should declare their permissions(read disk/visit network and other etc.)

tboerger commented 4 years ago

I don't get your concerns... That doesn't make sense to me.

lunny commented 4 years ago

@tboerger Plugins will run on your gitea server machine, it can do anything that's what my concerns. Could you explain why did you think that's not a problem?

tboerger commented 4 years ago

They can do whatever they implement... That's the use case for plugins 😂

zeripath commented 4 years ago

Depending on the level of plugin integration, there is a slight issue to do with plugins being run whilst there is an ongoing transaction with the db - but we'd just have to be explicit about when this happens. I would argue that level of integration would be too much to begin with.

If a server owner wants to blow themselves up by having a plugin that erases their entire db that's their call. We should however, try to proffer reasonable interfaces to the plugins to allow them to behave in a safe manner - so for example, offer sensible sanitisation services, perhaps offer some kind of limited js framework or even just best practices, or make it explicit which plugins extra js on the server - I dunno - I think the idea needs some concrete fleshing out to see what would need extra thought.

One thing that would make very tight plugin integration possible would be to move to a (configurable) dependency injection framework (There would be numerous benefits from this - including making it easier to test gitea properly). Go Wire is preferred by the go developers but I can't see how to make that make sense in a very configurable / plugin dominated system - something like a java Spring framework configurable file makes for very configurable system albeit with the costs of domain specific language. Again I dunno - just thoughts.

tboerger commented 4 years ago

Dependency injection doesn't exist in go... How to use wire you can see within drone.

Define some hook endpoints via interfaces, there drone could also give some advice. Start with first parts that can be extended and increase the hooks step by step

Fardinak commented 4 years ago

I think @lunny's concerns are valid, but that problem also exists with Go plugins or most other methods. I don't find requiring docker to be a viable option for a user that needs only a couple of plugins with their instance either. Also since OS support is clearly important, we need to remember that Docker still requires running a resource intensive VM on macOS.

Running plugins in a safe sandbox would be ideal. I consider WASI to be the ultimate solution, but unfortunately it just doesn't seem to be there yet. The closest option would be one of the interpreters mentioned by @andreynering:

About the plugin system, it's also hard to implement. Go has interpreters for other languages we could look:

* JavaScript: https://github.com/robertkrimen/otto
* Lua: https://github.com/yuin/gopher-lua
* Lua: https://github.com/Shopify/go-lua
* Bash: https://github.com/mvdan/sh (this one may not be suitable for the plugin system, but for allowing running bash commands that will run on Windows, too)

I must say, otto looks kinda promising.

Otherwise, it seems that GRPC is the most solid solution currently available. Paired with supervisord and it's RPC API (or maybe a golang implementation of it), they should provide a robust plugin solution for Gitea.

andreynering commented 4 years ago

Yaegi (a Go interpreter written in Go) is also a thing nowadays.

Fardinak commented 4 years ago

Oh that looks really interesting. How feasible would it be to create a limited subset of the stdlib? Although I don't know how that would affect the rest of the libraries, it could for example help disable the plugins from accessing the filesystem, or limit them to only the interface provided by Gitea.

tboerger commented 4 years ago

With all the concerns you should continue with the mono binary. I will unsubscribe from this issue because I currently don't believe this will lead to something useful.

Fardinak commented 4 years ago

Well it seems like the GRPC option makes sense to most. Let me know if I can be of help in any way. We're currently using Go+GRPC on a company project so I have some experience with it.

mewalig commented 3 years ago

Most of comments thus far seem to assume that the solution must be for gitea to dynamically load a module which is not incorporated into the original build of the gitea binary. That seems to have few advantages and many disadvantages, at least as a starting place.

What about just providing a format for modularizing the customization code, which allows for automating the integration of that code into the build process?

So instead of:

Build your gitea binary without customizations
Create your plug-in code, and build the plug-in binaries
Start gitea in some way that dynamically loads the above (assuming you can get this step to work)

you just do:

create your plug-in code, in some specific and known format (maybe this involves specific naming conventions, file + code snippet organization, etc)
Run a generic "merge" script that merges the plug-in code with the base code to generate composite code
Build your gitea binary from the composite code, creating a gitea binary with your customizations directly compiled in

This shifts the problem/challenge from dynamic loading, and all of the associated difficult problems, to a much simpler problem, which is to modify the build process to automatically merge the plug-in code with the base code, to generate the code that is ultimately compiled into the binary. This is not very hard, especially when one can easily define the initial scope to be as narrow as desired, and can also define the organizational structure of the new code to be whatever form is easiest to merge. Furthermore, this can be fairly easily done initially with plain text utilities, and while a more sophisticated approach to merging go code is not particularly trivial, it is also not that hard.

lafriks commented 3 years ago

This could be an option: https://scriggo.com/

mewalig commented 3 years ago

I have a budget of a few hundred thousand USD for developing this gitea code base further, and that number is likely to go higher, and that does not include a decent amount of $ that has already been spent. It's not a huge sum of money, but it can certainly fund a heck of a lot of the requested improvements that have not moved very quickly on this project.

We have already implemented several features that have been requested but not implemented in this code base, and we'd be interested in contributing portions of our changes to this code base, if for no other reason than to benefit from the shared maintenance burden and to give back a bit.

However, the lack of a plug-in system is making it challenging for us to do that in a manner that is not cost-prohibitive.

Any thoughts / suggestions as to how this can be moved forward?

techknowlogick commented 3 years ago

We have already implemented several features that have been requested but not implemented in this code base, and we'd be interested in contributing portions of our changes to this code base, if for no other reason than to benefit from the shared maintenance burden and to give back a bit.

Even if it is a PR with your current code that may have merge conflicts that would be beneficial for the public good, or even just a description of what you have done and how you have implemented it would be good too.

As it seems you have invested $$ of your engineering time into building stuff for gitea for yourself, perhaps funding Gitea maintainers directly for maintenance of Gitea would be beneficial as well, so that you can ensure continued development and support of Gitea. Several of us have Github sponsors, liberapay's, and patreons listed in our profile.

Any thoughts / suggestions as to how this can be moved forward?

Several of us have previously engaged in contract work to expand/maintain Gitea if there is a specific contract you'd like to discuss my email is in my profile, although I can also connect you with others who may be interested as well.

go-gitea / gitea