Use vendored dependencies

[nalbion]

Hi,

I've modified your example Makefile slightly (to use govendor) and it generates a zip file which I deploy to Lambda:

• index.so - 2.7MB
• shim.so - 2.8MB
• handler.py
• init.py

index.go:

package main

import "C"

import (
    "fmt"
    "github.com/eawsy/aws-lambda-go-core/service/lambda/runtime"
    "github.com/eawsy/aws-lambda-go-event/service/lambda/runtime/event/apigatewayproxyevt"
)

func Handle(evt apigatewayproxyevt.Event, ctx *runtime.Context) (interface{}, error) {
    fmt.Println(evt)
    return "testing...", nil
}

I deploy with config:

• Runtime: Python 2.7
• Handler: index.Handle

When I test the function in the Lambda web console it responds:

{
  "errorMessage": "Unable to import module 'index'"
}

and the logs say:

Unable to import module 'index': dynamic module does not define init function (initindex)

My Makefile contains:

PACKAGE ?= index
HANDLER ?= index

lambda/$(PACKAGE).zip: clean
    @echo -ne "build..."\\r
    @govendor build -buildmode=plugin -ldflags='-w -s' -o lambda/$(HANDLER).so ./lambda
    @chown $(shell stat -c '%u:%g' .) lambda/$(HANDLER).so
    @echo -ne "build, pack"\\r
    @cd lambda; zip -q $(PACKAGE).zip $(HANDLER).so
    @echo -ne "build, pack, inject"\\r
    @cd lambda; zip -q -r -j $(PACKAGE).zip /shim
    @chown $(shell stat -c '%u:%g' .) lambda/$(PACKAGE).zip
    @echo -ne "DONE!"\\n

[fsenart]

Hi @nalbion, this is due to golang/go#18827.

Basically, the current Docker image embeds a version of the shim which does not use a vendored version of eawsy/aws-lambda-go-core. Waiting for the Golang team to resolve the issue, I will create a new Docker image embedding a version of the shim built with the vendored version of eawsy/aws-lambda-go-core. So that instead of using the latest tag of the current Docker image, you will use another tag embedding the alternate shim.

[fsenart]

@nalbion have you created a new Docker image inherited from ours? with govendor installed on it?

[fsenart]

Ok I have nicer solution without the need for a new Docker image. Add the following to your Makefile in order to use your vendor folder as the only source of truth for the Docker image:

vendor:
        @docker run --rm \
                -v $(PWD)/vendor:/go/src -v $(PWD):/tmp \
                -e "HANDLER=$(HANDLER)" -e "PACKAGE=$(PACKAGE)" \
                eawsy/aws-lambda-go-shim make _all

So the only thing you need to do is make vendor.

PS: Do not use govendor for the build, instead use it to fetch your dependencies into your vendor folder and then use the above target to build your Lambda.

/cc @fahernandez

[nalbion]

This is my Dockerfile:

FROM eawsy/aws-lambda-go-shim

RUN yum -q -e 0 -y install git \
    && go get -u github.com/kardianos/govendor \
    && go get -u github.com/jstemmer/go-junit-report \
    && go get -u github.com/axw/gocov/gocov \
    && go get -u github.com/marinbek/gocov-xml \
    && go get -u github.com/wadey/gocovmerge

# `make docker` first calls `mkdir -p lambda/_gopath/src && cp -r /usr/lib/go_appengine/goroot/src/appengine* lambda/_gopath/src/`
COPY _gopath/src /go/src/

# Deploy
RUN curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && python get-pip.py && rm get-pip.py
RUN pip install --upgrade awscli

The application should be deployable to Lambda or AppEngine. (maybe I should be using build tags to create 2 distinct apps?)

[nalbion]

(This comment started off as something completely different, but evolved into me talking to myself now - might be helpful to others)

According to your docs, you can deploy a zip containing the shim files and foo.so:

func Bar(evt Whatever, ctx *runtime.Context) (interface{}, error) {
    log.Println("Hello, World!")
    return nil, nil
}

to Lambda with config:

Runtime: Python 2.7
Handler: foo.Bar

...ahhh.... my Makefile is wrong!

Your Makefile creates a zip with the structure:

• foo.so
• foo/init.py
• foo/handler.py
• foo/shim.so

foo/__init__.py

• tells python that the foo directory is a package (Lambda is looking for foo.Bar()
• calls shim.open("foo") -> shim_copen() -> shim_goopen() -> plugin.open("foo.so")
  • ...which loads foo.so and saves a Plugin handle as plg
• defines the foo module as an instance of the Handler class in handler.py

handler.py

• the Handler class has a catch-all getter which eventually calls
  • shim.handle() -> hval.Call(args) -> foo.Bar()

...you didn't mention in the README that it also supports environment variables (kind of weird that you need to read them into Python just to write them back out to the go environment now that I think about it...)

[nalbion]

after fixing my Makefile to write the shim files into the $(HANDLER) directory I moved onto the next error:

        if htyp.NumIn() != 2 || htyp.In(1) != ctyp ||
        htyp.NumOut() != 2 || !htyp.Out(1).Implements(reflect.TypeOf((*error)(nil)).Elem()) {
        return errorf("Cannot use handler '%s' with invalid signature", hnm)
    }

If the code provided a different error message for each condition it would be easier to know, but I'm pretty sure that it's getting stuck on htyp.In(1) != ctyp - so does that mean that I've got the wrong github.com/eawsy/aws-lambda-go-core/service/lambda/runtime?

I then tried deleting vendor/github.com/eawsy and adding them to /go/src in my Dockerfile:

FROM eawsy/aws-lambda-go-shim

RUN yum -q -e 0 -y install git \
    && go get -u -d github.com/eawsy/aws-lambda-go-core/... \
    && go get -u -d github.com/eawsy/aws-lambda-go-event/...

...but then it complains:

Unable to import module 'index': plugin.Open: plugin was built with a different version of package github.com/eawsy/aws-lambda-go-core/service/lambda/runtime

I'm not sure which version I should be using. I tried to determine which version you use, but then eventually see this:

$ docker run -it eawsy/aws-lambda-go-shim ls -al /go
total 12
drwxr-xr-x  3 root root 4096 Jan 10 22:49 .
drwxr-xr-x 36 root root 4096 Feb 13 12:59 ..
drwxr-xr-x  2 root root 4096 Jan 10 22:49 {bin,pkg,src}

[fsenart]

@nalbion I tried to explain the inner behavior of the shim outside of this issue in the gitter channel. For your issue, the way you try to find what is the underlying problem is the exact same way that took me to golang/go#18827. The problem is well defined.

In the meantime, please try the workaround which consists in mounting your vendor folder into the Docker image go/src.

I will update the README and the example Makefile soon to add this stuff. If you can confirm me that the solution works for you, I would be perfect. For other discussions please fill another issue or let's talk in the gitter channel.

Thx.

[fsenart]

@nalbion vendoring supported out of the box with the new release :wink:

[nalbion]

Awesome, thanks!