go-gitea / gitea

Git with a cup of tea! Painless self-hosted all-in-one software development service, including Git hosting, code review, team collaboration, package registry and CI/CD
https://gitea.com
MIT License
44.91k stars 5.48k forks source link

Plugin Mechanism #20126

Open delvh opened 2 years ago

delvh commented 2 years ago

What should it be?

Architecture

Communication: _Gitea :arrow_right: plugin :arrowright: Gitea

Communication: _plugin :arrow_right: Gitea :arrowright: plugin

Plugin Lifecycle (What the plugins can set/ "use")

What data is needed by every/ some plugin(s)?

Why this long text rambling?

  1. Everyone should know without a doubt how it should look like in the end
  2. To make the work for the (eventual) reviewers easier by simply letting them cross-check the actual functionality with its intention
  3. Most of the text I wrote here can be copy-pasted into the docs, making it easier for me to write the docs
  4. To allow for discussion if I missed something while planning this feature
  5. To see if the scope others expect from this feature differs from what I imagine from it
42wim commented 2 years ago

I think https://github.com/hashicorp/go-plugin should also be taking in consideration, afaik the "normal" plugin system has a lot of shortcomings (eg crashing the whole process because of issues with the plugin), that the Hashicorp one does not suffer from.

See also:

delvh commented 2 years ago

Yeah, I've also seen their implementation when looking for a plugin system. However, to me, it appears as if they don't support executing dynamic functions as I think I read the sentence sending functions is not possible anywhere, and then I stopped reading and didn't investigate further as that is something we most likely need. After having read through their docs for a little, it seems I was a bit too fast with that.

Regarding the unexpected crashes: I intended to call every plugin by using something like

func PluginCall(function func(plugin *Plugin)err) {
  for _, plugin := range plugins {
    executePluginCall(plugin)
  }
}

func executePluginCall(plugin *Plugin, function func(plugin *Plugin)err) {
  defer func(){
    if err := recover(); err != nil { // Error handling
    }
  }()
  function(plugin)
}

and thus mitigating the potential for those errors.

So, the question is now, should we use the RPC version which might offer a few benefits and higher latency, or do we stay with the Go native approach that we can completely customize to our liking?

Gusted commented 2 years ago

However, to me, it appears as if they don't support executing dynamic functions as I think I read the sentence sending functions is not possible anywhere, and then I stopped reading and didn't investigate further as that is something we most likely need.

Do you want to compile/execute Go code on-the-fly?

delvh commented 2 years ago

Not really, but we certainly need to handle the custom routes which can best be achieved dynamically, and making this interface-based might be possible, but (a little bit) more work.

Gusted commented 2 years ago

Wish you many luck with this advanced logic.

lunny commented 2 years ago

I prefer the plugin system based on javascript or nodejs because that will allow it include frontend code.

Possbile library be used. https://github.com/dop251/goja, there is also a plugin support nodejs .

delvh commented 2 years ago

Interesting how many different views there are regarding this feature. Now we have three completely different options and no agreement on what to use...

lunny commented 2 years ago

Interesting how many different views there are regarding this feature. Now we have three completely different options and no agreement on what to use...

Use javascript based plugin system, we can implement a plugin market and one click installation. That's impossible for Go based plugin system.

delvh commented 2 years ago

The problem I can see is: We do not even have the same definition of what we want in the end: What you want seems to be a per-user plugin mechanism, what I thought of was a per-instance plugin mechanism...

lunny commented 2 years ago

The problem I can see is: We do not even have the same definition of what we want in the end: What you want seems to be a per-user plugin mechanism, what I thought of was a per-instance plugin mechanism...

I don't think so. I think I'm also talking about per-instance plugin. If I made you confusing, I'm sorry.

jolheiser commented 2 years ago

-1 for the go plugin package. It can't be removed per compatibility promise, but it is absolutely not stable, and there are far too many caveats to get it working properly.

lunny commented 2 years ago

I think we can begin from a theme-kind plugin, which is simple and useful. A plugin could be a zip/tar file with a main.js or index.js and other files like theme.sass, theme.css and images and etc.

wxiaoguang commented 2 years ago

I do not think the Go's plugin package is the future. Is there any success story of using it?

Most large Go projects uses gPRC for inter-process commutation, including plugins, but it's also heavy.

delvh commented 2 years ago

Ok, seems as if the plugin package is generally disliked, so I'll exclude it from the possible options.

I'm still not a fan of a purely frontend-based plugin mechanism as that will have many shortcomings of its own, and I don't know if

plugin market and one click installation.

should be the deciding factor for how to implement the plugin system... Server admins should be sure of what they want for their instance, and not install it with a simple misplaced click. Also, even with the Go-based approach, a plugin "market" would be possible as a third-party tool that simply stores the executables and perhaps some other information about the plugin...

So, as I can see that now, the two possible options are

luwol03 commented 2 years ago

Isn't this library lunny suggested meant for a backend-based plugin mechanism? But instead of writing go you can write js? Or would that be to difficult to build a bridge between the Gitea go code and js code?

lunny commented 2 years ago

Yes, it could expose structs, methods to frontend. i.e. If we expose getdb method, then we can operate database with javascript.

lafriks commented 2 years ago

I think the best way to go would be using grpc so that plugins could be implemented using any language. Plugin could return asset files on init (JavaScript etc) that would need to be injected/used in specific places

jolheiser commented 2 years ago

I would also like to throw yaegi into the mix. I think gRPC is probably the only way to go (unfortunately), but figured yaegi is worth considering as well.

Mai-Lapyst commented 2 years ago

Honestly, I would like to see the native go plugin implementation. Yes it has drawbacks, but it would be fast and admins had full controll over what to use.

But since thats off the table for now, I would rather go with the gRPC approach, since its more open (i.e. you can choose more languages) and imo more secure then javascript.

The problems I see with js plugins is, that it lead to a huge security risk, since js can be so minified or encrypted that you not even need any letters (only parentheses) to write correct js. We then would need an very strict system so no js can accidentally just execute "DROP ALL TABLES;" (or similar) on startup.

Also an interpreter (of any kind) in gitea would bring back the risk of one plugin crashing the whole instance, or to eat up so much performance that gitea stops responding (i.e. a infinite loop in a script somewhere). To mitigate this we then would need to sandbox the plugins into own processes, which would nearly the same as the gRPC approach but with less freedom.

delvh commented 2 years ago

To be fair, you can also DROP ALL TABLES; in Go or consume all available memory. But yeah, I can see your point.

luwol03 commented 2 years ago

Then you have to think twice about an official plug-in market like lunny suggested. Official also indicates that they are save for usage and have no malicious intentions. An easy one click install could potentially break servers from not experienced users who don't understand what the plugin is doing and just pressing install.

To prevent this, this would at least requires some reviewing and analysis of newly published plugins, which is time expensive.

lafriks commented 2 years ago

We could probably get some ideas on how mattermost have implemented their plugin system.

jolheiser commented 2 years ago

We could probably get some ideas on how mattermost have implemented their plugin system.

For posterity: mattermost plugin package It looks like they use go-plugin

mscherer commented 2 years ago

Another inspiration could be woodpecker proposal around extensions https://github.com/woodpecker-ci/woodpecker/pull/915

Trendyne commented 2 years ago

Since nobody has mentioned it yet, projects like Adobe Lightroom, MPV, or Wireshark use Lua for this. You'd read and run Lua files from a directory where they would register events with callbacks or maybe return properties and functions. It's very light. simple, and fast but would make it very easy to make plugins and shouldn't be any harder to implement

Made up possible example:

-- Greet users on their first repo
gitea.on("newRepo", function(repo)
    if #repo.owner.repos == 1 then
        local msg = ("Welcome to our Gitea, %s!"):format(repo.owner.name)
        repo.owner:sendNotifacation(msg)
    end
end)
KN4CK3R commented 2 years ago

The Lua integration could be plugin too which forwards callbacks to Lua.

Lord-Leonard commented 6 months ago

Are there any updates on this one? ^^

lunny commented 6 months ago

For most backend tasks, Gitea Actions can be considered as somewhat plugins.

Lord-Leonard commented 6 months ago

Yeah for my usecase the webhooks are sufficient. I will just have to host another small webserver alongside Gitea. But it would have been nice to just write a plugin and call it a day :)

Sharaf5 commented 2 months ago

Then there are 4 point of views could be considered for plugins