Plugins

[alexec]

It would be good to have a system of plugins that do the following:

• Allow new template types to be easily introduced without impacting core controller code, e.g. HTTP template (#795)
• Tap into the workflow lifecycle, e.g. #4355
• Integrate with third party tools like Kafka.
• Write notification plugins.

Additionally, the ability for users to create plugis would allow us to grow the ecosystem.

Golang Plugin

Golang provides a plugin system, but it very constrained as to how you build and package them:

• Your plugin must be compiled against the same version of Golang
• Your plugin must be compiled against the same version of Argo.
• Plugins do not work with Darwin.
• Must be built using CGO_ENABLED=1

This means effectively plugins must be built in the same source code repo.

However, if we provide a way to package your plugin, they are obviously the fastest and most powerful to write plugins.

HashiCorp Go Plugins

Golang provides an RPC library. HashCorp has formalized this into a way to write bi-directional plugins.

References

Tekton Custom Job:

• https://docs.google.com/document/d/10nQSeIse7Ld4fLg4lhfgUmNKtewfaFNET3zlMdRnBuQ/edit
• https://github.com/tektoncd/community/blob/main/teps/0002-custom-tasks.md

Problems with Golang plugins:

https://www.reddit.com/r/golang/comments/b6h8qq/is_anyone_actually_using_go_plugins/

[Bobgy]

I'd hope plugins can evolve as independently as possible.

Did you ever consider REST/gRPC plugins? It might share some of the issues you mentioned with http templates, but now it's clear they will talk to a single configured trusted server, so some problems like auth and user permissions no longer apply.

[alexec]

Do you mean this?

https://github.com/hashicorp/go-plugin

[Bobgy]

I was only thinking about the problem architecturally and I didn't know hashicorp/go-plugin before. But, that sounds like a great match!

Any packages (including transitive dependencies) outside of the standard library that are used by both the plugin and the program must have their versions match exactly.

(referenced from the reddit link)

The limitation on golang plugin seems unacceptable as a plugin purpose.

EDIT: note, please keep in mind I have limited experience on microservice systems with my own experience. Just talking about my current thoughts and they can be wrong.

[alexec]

More reading material:

• https://7thzero.com/blog/golang-w-sqlite3-docker-scratch-image
• https://stackoverflow.com/questions/15165306/compile-a-static-binary-which-code-there-a-function-gethostbyname
• https://medium.com/@chemidy/create-the-smallest-and-secured-golang-docker-image-based-on-scratch-4752223b7324 (this is very good article)
• https://www.reddit.com/r/golang/comments/53nl4b/why_all_go_binaries_are_staticallylinked/
• https://oddcode.daveamit.com/2018/08/16/statically-compile-golang-binary/
• https://stackoverflow.com/questions/15165306/compile-a-static-binary-which-code-there-a-function-gethostbyname
• https://stackoverflow.com/questions/2725255/create-statically-linked-binary-that-uses-getaddrinfo
• https://github.com/GoogleContainerTools/distroless/blob/master/base/README.md

[alexec]

Golang Plugin (GP) vs HashiCorp Plugin (HP)

Aside: I think we should consider supporting both.

Build

• GP must be built using the same tooling as this binary. Effectively, you must provide a build harness for users to use, and likely build their plugins for them as part of your build. You must build and package with glibc, and CGO_ENABALED=, this is much easier with multi-platform Docker builds.
• HP can happily be built however as they are just binaries.

HP wins by a huge margin.

Performance

• GP is in process.

GP wins.

Language

I believe that HP can be written in non-go.

HP wins.

Boilerplate

• GP has minimal boiler-plate. It is POGO (plain-old Go objects)

GP wins.

Callback

Callbacks are trivial in GP. Not sure about HP.

GP wins.

[Bobgy]

So, talking about use-cases, it seems GP can be a good fit for first party maintained optional plugins.

[alexec]

I think we can and should build an RPC (or JSON RPC) plugin as a Go plugins.

[alexec]

I'm really happy with my PoC. You can build your plugins as a Golang plugin. One of the Golang plugins will proxy requests via JSON-RPC. You can add a sidecar to the workflow-controller that services the RPC requests, so the deployment of the plugin is trivial.

I think JSON-RPC will work for most use cases, but you can always write a Golang plugin if you want.

[alexec]

https://velero.io/docs/v1.5/overview-plugins/

[therealnb]

Use case: We have X tasks to perform on Y endpoints in Z servers. We would like to use a step for each to take advantage of retry policy, alerting and other workflow goodies. XYZ can be 100s in a big system. Each step is a rest operation and completes quickly.

Problem: If there are no failures most of the time is taken in creating the containers. This can take many minutes to do 100 posts. Further more, the first steps are a pause and the last resume, so the workflow duration is down time for this maintenance task.

Solution: We can combine steps, but we lost the 'workflow goodies'. It would be better if we had a way for quick execution. One of

Ability for a step to run a small python script in another context (workflow-manager, or dedicated). An exec step which runs on a previously created daemon. The main reason not to share containers is collisions in file names. This isn't a problem if we aren't creating temporary files and if we are we can use the pod name to name the files.

[Bobgy]

@therealnb do you think https://github.com/argoproj/argo-workflows/issues/795 work for your use-case?

You can configure steps that send a request to your own server, and your server will be able to do whatever things you want.

A limitation could be that this is not suitable for steps that take some time to execute.

[alexec]

I'm closing this issue in favor for #5201 and #4355 which are more narrowly defined (but would have closely relati impl)

[therealnb]

That's fine. #795 would help in our case.