Closed samcday closed 6 years ago
fsenart
commented
7 years ago Hi @samcday, Whaoo what a useful PR. I haven't even heard about this option until now. Very thankful for your PR. I will merge it in the next few days, I have to perform some tests, etc. and will come back soon.
samcday
commented
7 years ago @fsenart no probs! Let me know if I can help any further
fsenart
commented
6 years ago Hi @samcday,
Please apologize for my unavailability. I was under a very heavy workload past month.
I reviewed very carefully this optimization and here are my remarks:
-pkgdir option to totally control the installed artifacts destination instead of using /usr/local/go/pkg/linux_amd64_dynlinkGOPATH. I propose /cache. PKGDIR ?= $(CURDIR)/.cache.It brings us to the following updated Makefile. Please pay attention to the following points:
PKGDIR under windows has to be confirmed. I haven't currently access to this os.perm target has to be updated in this way so that the local directory is under the control of the end user.clean target should be updated to remove all created artifacts.HANDLER ?= handler
PACKAGE ?= $(HANDLER)
ifeq ($(OS),Windows_NT)
GOPATH ?= $(USERPROFILE)/go
GOPATH := /$(subst ;,:/,$(subst \,/,$(subst :,,$(GOPATH))))
CURDIR := /$(subst :,,$(CURDIR))
PKGDIR ?= $(CURDIR)/.cache
RM := del /s /q
else
GOPATH ?= $(HOME)/go
PKGDIR ?= $(CURDIR)/.cache
RM := rm -rf
endif
MAKEFILE = $(word $(words $(MAKEFILE_LIST)),$(MAKEFILE_LIST))
docker:
docker run --rm\
-e HANDLER=$(HANDLER)\
-e PACKAGE=$(PACKAGE)\
-e GOPATH=$(GOPATH)\
-e LDFLAGS='$(LDFLAGS)'\
-v $(CURDIR):$(CURDIR)\
-v $(PKGDIR):/cache\
$(foreach GP,$(subst :, ,$(GOPATH)),-v $(GP):$(GP))\
-w $(CURDIR)\
eawsy/aws-lambda-go-shim:latest make -f $(MAKEFILE) all
.PHONY: docker
all: build pack perm
.PHONY: all
build:
go build -v -buildmode=plugin -ldflags='-w -s $(LDFLAGS)' -pkgdir=/cache -i -o $(HANDLER).so
.PHONY: build
pack:
pack $(HANDLER) $(HANDLER).so $(PACKAGE).zip
.PHONY: pack
perm:
chown -R $(shell stat -c '%u:%g' .) $(HANDLER).so $(PACKAGE).zip /cache
.PHONY: perm
clean:
$(RM) $(HANDLER).so $(PACKAGE).zip $(PKGDIR)
.PHONY: cleanLet me know what do you think about these remarks. And hopefully we will merge this huge optimization very soon. Thank you again for your time and for your contribution.
fsenart
commented
6 years ago See also badass improvements coming with Go 1.10.
samcday
commented
6 years ago Sorry @fsenart - Github doesn't seem to notify me via email anymore.
Your changes LGTM, much more comprehensive approach to the optimization :)
fsenart
commented
6 years ago @samcday can you please update your patch in accordance so that I can merge this long running PR :smile:
PS: In the meantime I will try to find a windows host to be sure that the PKGDIR works as expected.
lion3ls
commented
6 years ago Works on Windows 10 👍.
For windows compatibility some modifications are needed, here is my makefile working on both Windows and Linux:
HANDLER ?= handler
PACKAGE ?= $(HANDLER)
CACHEDIR ?= .cache
ifeq ($(OS),Windows_NT)
GOPATH ?= $(USERPROFILE)/go
GOPATH := /$(subst ;,:/,$(subst \,/,$(subst :,,$(GOPATH))))
CURDIR := /$(subst :,,$(CURDIR))
PKGDIR ?= $(CURDIR)/$(CACHEDIR)
RM := del /S /Q
RMDIR := rd /S /Q
else
GOPATH ?= $(HOME)/go
PKGDIR ?= $(CURDIR)/$(CACHEDIR)
RM := rm -rf
RMDIR := rm -rf
endif
MAKEFILE = $(word $(words $(MAKEFILE_LIST)),$(MAKEFILE_LIST))
docker:
docker run --rm\
-e HANDLER=$(HANDLER)\
-e PACKAGE=$(PACKAGE)\
-e GOPATH=$(GOPATH)\
-e LDFLAGS='$(LDFLAGS)'\
-v $(CURDIR):$(CURDIR)\
-v $(PKGDIR):/cache\
$(foreach GP,$(subst :, ,$(GOPATH)),-v $(GP):$(GP))\
-w $(CURDIR)\
eawsy/aws-lambda-go-shim:latest make -f $(MAKEFILE) all
.PHONY: docker
all: build pack perm
.PHONY: all
build:
go build -v -buildmode=plugin -ldflags='-w -s $(LDFLAGS)' -pkgdir=/cache -i -o $(HANDLER).so
.PHONY: build
pack:
pack $(HANDLER) $(HANDLER).so $(PACKAGE).zip
.PHONY: pack
perm:
chown -R $(shell stat -c '%u:%g' .) $(HANDLER).so $(PACKAGE).zip /cache
.PHONY: perm
clean:
$(RM) $(HANDLER).so $(PACKAGE).zip
$(RMDIR) $(CACHEDIR)
.PHONY: cleanCan't wait on your final PR @samcday 😉
fsenart
commented
6 years ago Thank you very much @samcday. Sadly, we haven't had a chance to integrate this awesome contribution as the new official go runtime is out 🎉
First up, awesome project. Huge fan! :)
I started using it but quickly noticed that the builds for my Lambda function were getting crazy slow as I imported more packages. I did a bit of digging and it turns out running
go build buildmode=plugindoesn't properly cache thepkg/artifacts unless you tell it to with the-ioption. That limitation is kinda hidden away in golang/go#19707 and I didn't find it until the third or fourth frustrated google search ;)So what this does is adds the
-ioption to the build command, and ensures we have a data volume attached to/usr/local/go/pkg/linux_amd64_dynlinkin the container so that the pkg cache is reused between builds.Before this change:
After: