Can't setup go plugins via proxy container's env

mayanksriv commented 4 years ago

Summary

Trying to build a custom go plugin and set it up for ingress controllers. I just copied this source into foo.go https://github.com/Kong/go-plugins/blob/master/go-hello.go and went from there on as if it was a custom plugin. Built the plugin like

go build -buildmode=plugin foo.go

But when setting up the plugin in manifest files via KONG_PLUGINS & KONG_GO_PLUGINS_DIR, the proxy container fails to start. kubectl logs for that container are pasted below which show some Exec format error.

Kong Ingress controller version 0.7.1

Kong or Kong Enterprise version 2.0

Kubernetes version

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.5", GitCommit:"20c265fef0741dd71a66480e35bd69f18351daea", GitTreeState:"clean", BuildDate:"2019-10-15T19:16:51Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-07T21:12:17Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}### Environment

OS : macOS Catalina Running on top of minikube with this custom docker image tagged as custom-kong:latest
```
FROM kong:2.0
COPY foo.so /plugins/foo.so
```

What happened

I loaded a go plugin by setting these env vars on the proxy container

      containers:
      - name: proxy
        image: custom-kong:latest
        env:
        - name: KONG_PLUGINS
          value: bundled,myheader
        - name: KONG_GO_PLUGINS_DIR
          value: /plugins
        - name: KONG_PLUGINS
          value: foo

And I get errors on pod startup like

2020/02/05 16:34:20 failed to open plugin auth_proxy: plugin.Open("/plugins/foo.so"): Error loading shared library /plugins/foo.so: Exec format error
2020/02/05 16:34:20 [error] 1#0: init_by_lua error: /usr/local/share/lua/5.1/kong/db/dao/plugins/go.lua:455: bad argument Kong/kubernetes-ingress-controller#1 to 'ipairs' (table expected, got nil)
stack traceback:
    [C]: in function 'ipairs'
    /usr/local/share/lua/5.1/kong/db/dao/plugins/go.lua:455: in function 'get_plugin'
    /usr/local/share/lua/5.1/kong/db/dao/plugins/go.lua:481: in function 'load_plugin'
    /usr/local/share/lua/5.1/kong/db/dao/plugins.lua:151: in function 'load_plugin_handler'
    /usr/local/share/lua/5.1/kong/db/dao/plugins.lua:227: in function 'load_plugin'
    /usr/local/share/lua/5.1/kong/db/dao/plugins.lua:275: in function 'load_plugin_schemas'
    /usr/local/share/lua/5.1/kong/init.lua:425: in function 'init'
    init_by_lua:3: in main chunk

I removed KONG_PLUGINS env and that started the pod but my plugin was not setup. I could verify by going into the proxy container that /plugins/foo.so existed.

mayanksriv commented 4 years ago

I setup kong locally via homebrew and I think the error over there is more descriptive.

Error: /usr/local/share/lua/5.1/kong/cmd/start.lua:64: 2020/02/05 15:23:41 failed to open plugin foo: plugin.Open("/Users/mayank.srivastava/kong/plugins/foo"): plugin was built with a different version of package github.com/Kong/go-pdk/bridge
nginx: [error] init_by_lua error: /usr/local/share/lua/5.1/kong/db/dao/plugins/go.lua:455: bad argument #1 to 'ipairs' (table expected, got nil)
stack traceback:
    [C]: in function 'ipairs'
    /usr/local/share/lua/5.1/kong/db/dao/plugins/go.lua:455: in function 'get_plugin'
    /usr/local/share/lua/5.1/kong/db/dao/plugins/go.lua:481: in function 'load_plugin'
    /usr/local/share/lua/5.1/kong/db/dao/plugins.lua:151: in function 'load_plugin_handler'
    /usr/local/share/lua/5.1/kong/db/dao/plugins.lua:227: in function 'load_plugin'
    /usr/local/share/lua/5.1/kong/db/dao/plugins.lua:275: in function 'load_plugin_schemas'
    /usr/local/share/lua/5.1/kong/init.lua:425: in function 'init'
    init_by_lua:3: in main chunk

Go version I am using to build the plugin

$ go version
go version go1.13.7 darwin/amd64

gszr commented 4 years ago

Hi @mayanksriv,

Kong ships with a go-pluginserver binary that is built with a specific version of Go (1.13.4) and with a specific version of the go-pdk (v0.3.0).

You have two options:

If you want to build your plugin with a different version of Go (than 1.13.4), you need to recompile the go-pluginserver and replace that in the Kong Docker image; same applies to the go-pdk version
Compile your plugin with the same version of Go used in the bundled go-pluginserver and make sure your plugin also compiles against the same version of the go-pdk

Please try one of the above and let us know how it goes.

Thanks!

javierguerragiraldez commented 4 years ago

Hi @mayanksriv ,

As mentioned, Go does a very strict compiler version check when loading plugins. If you want to develop your own plugins, the easiest is to compile the go-pluginserver yourself. Just do

go get github.com/Kong/go-pluginserver

and the newly compiled executable should go to your $GOROOT/bin/ directory (or $HOME/go/bin if you're gorootless) replace the provided go-pluginserver (at /usr/local/bin/)with this one, and it should work.

mayanksriv commented 4 years ago

@gszr I tried your first option locally on my mac and works. But when I go on minikube, I got the same errors. I built the plugin on golang:1.13.4 docker image so that same OS/arch are used but that failed too. I think Go's plugin arch is a tad bit too strict in a sense that every detail has to be exactly same.

gszr commented 4 years ago

@mayanksriv, could you test rebuilding the pluginserver as well? That can be done with go get github.com/Kong/go-pluginserver; then, you can either place it in /usr/local/bin or set the go_pluginserver_exe property to the pluginserver's path.

mayanksriv commented 4 years ago

@gszr So basically, create a custom docker image from kong:2.0 base and rebuilding pluginserver on it? But shouldn't it be all fine if I built my plugins on Go v1.13.4 with go-pdk v0.3.0 cause that is what the go-pluginserver on kong:2.0 image is built with.

javierguerragiraldez commented 4 years ago

that's what we've tried to do; but unfortunately Go's compatibility checks are extremely picky. Not only compiler and libraries versions, but also system libraries and even some environment variables (unless explicitly overridden.... which isn't trivial to do and remain flexible.

a slightly less intrusive option could be to put your compiled go-pluginserver in a volume, so it's not (totally) a custom image.

mayanksriv commented 4 years ago

So I created a docker image like this and used it as an init container before loading the main proxy container with kong:2.0.

FROM golang:1.13

RUN go get github.com/Kong/go-pluginserver

WORKDIR /go/src/github.com/Kong/go-pluginserver

RUN make

RUN mkdir /pluginserver; mv go-pluginserver /pluginserver/go-pluginserver

And init container spec is like

spec:
      initContainers:
      - name: init-plugins
        image: plugin:latest
        imagePullPolicy: Never
        command:
        - sh
        - -c
        - mv /pluginserver/go-pluginserver /pluginserver_exe
        volumeMounts:
        - name: pluginserver-volume
          mountPath: /pluginserver_exe

And then I specify the following env param to the proxy container:

- name: KONG_GO_PLUGINSERVER_EXE
   value: /pluginserver_exe/go-pluginserver

And I get following errors on pod startup


sh: /pluginserver/go-pluginserver: not found
2020/02/12 21:35:38 [error] 1#0: init_by_lua error: /usr/local/share/lua/5.1/MessagePack.lua:813: missing bytes
stack traceback:
    [C]: in function 'error'
    /usr/local/share/lua/5.1/MessagePack.lua:813: in function 'underflow'
    /usr/local/share/lua/5.1/MessagePack.lua:529: in function 'unpack_cursor'
    /usr/local/share/lua/5.1/MessagePack.lua:843: in function 'unpack'
    /usr/local/share/lua/5.1/kong/db/dao/plugins/go.lua:438: in function 'get_plugin_info'
    /usr/local/share/lua/5.1/kong/db/dao/plugins/go.lua:447: in function 'get_plugin'
    /usr/local/share/lua/5.1/kong/db/dao/plugins/go.lua:481: in function 'load_plugin'
    /usr/local/share/lua/5.1/kong/db/dao/plugins.lua:151: in function 'load_plugin_handler'
    /usr/local/share/lua/5.1/kong/db/dao/plugins.lua:227: in function 'load_plugin'
    /usr/local/share/lua/5.1/kong/db/dao/plugins.lua:275: in function 'load_plugin_schemas'
    /usr/local/share/lua/5.1/kong/init.lua:425: in function 'init'
    init_by_lua:3: in main chunk

ealogar commented 4 years ago

@mayanksriv You should use a docker image for building your plugins of the same type of the kong docker you use... golang:1.13 is not alpine, I think it's ubuntu, but you may use FROM golang:1.13-alpine3.11 if you use kong:2.0.1-alpine as docker for package.

The same error happens to me and I fix in this way

mayanksriv commented 4 years ago

Thanks a lot @ealogar . I got things working based on your suggestion but I still think its a pretty tight dependency. Maybe we can have docker builder images which map to specific kong images and users can use them straightaway to build plugins seamlessly. Plus, I shouldn't have to build the go-pluginserver in the first place. I should just be building my compatible plugins.

gszr commented 4 years ago

This might also be of help: https://github.com/Kong/docker-kong/issues/326.

gszr commented 4 years ago

Closing this, feel free to reopen if needed! Thanks @mayanksriv and @ealogar

Kong / kong