Aws lambda Execution failed - /lib64/libc.so.6: version `GLIBC_2.28' not found

[matrottier]

Trying to run Tegola with AWS lambda.
i got this error when testing : START RequestId: 1340f6e4-ceee-401b-950b-cac0880a4f95 Version: $LATEST /var/task/tegola_lambda: /lib64/libc.so.6: version 'GLIBC_2.28' not found (required by /var/task/tegola_lambda) END RequestId: 1340f6e4-ceee-401b-950b-cac0880a4f95 REPORT RequestId: 1340f6e4-ceee-401b-950b-cac0880a4f95 Duration: 35.30 ms Billed Duration: 36 ms Memory Size: 512 MB Max Memory Used: 6 MB RequestId: 1340f6e4-ceee-401b-950b-cac0880a4f95 Process exited before completing request

i tried the version from github and a compiled version with go 1.3 and go 1.6.
also tried the option "CGO_ENABLED=0" on the compilation without success

[ARolek]

@matrottier hmmm. We're currently using tegola_lambda to run the demo.tegola.io. I'm going to ask the obvious question first, can you confirm you're using the Go runtime in Lambda?

[matrottier]

i think so... there it is my console :

[ARolek]

@matrottier that looks correct to me. When you built locally, did you use GOOS=linux? Does this same error happen when you use the release builds?

[matrottier]

@ARolek it happen with release builds AND locally builds. =/

[matrottier]

***EDITED LOG in an act of utter desperation,
i tested a "layer" with the file "libc.so.6" in the folder "lib/lib64".
2 environments variables: LD_DEBUG | all LD_LIBRARY_PATH | /opt/lib/lib64/

there is the log :

so.2 [0]
         9: symbol=_rtld_global;  lookup in file=/opt/lib/lib64/libc.so.6 [0]
         9: symbol=_rtld_global;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
         9: binding file /lib64/libdl.so.2 [0] to /lib64/ld-linux-x86-64.so.2 [0]: normal symbol `_rtld_global' [GLIBC_PRIVATE]
         9: symbol=__cxa_finalize;  lookup in file=/var/task/tegola_lambda [0]
         9: symbol=__cxa_finalize;  lookup in file=/lib64/libpthread.so.0 [0]
         9: symbol=__cxa_finalize;  lookup in file=/lib64/libdl.so.2 [0]
         9: symbol=__cxa_finalize;  lookup in file=/opt/lib/lib64/libc.so.6 [0]
         9: binding file /lib64/libdl.so.2 [0] to /opt/lib/lib64/libc.so.6 [0]: normal symbol `__cxa_finalize' [GLIBC_2.2.5]
         9: 
         9: relocation processing: /lib64/libpthread.so.0 (lazy)
         9: symbol=_ITM_deregisterTMCloneTable;  lookup in file=/var/task/tegola_lambda [0]
         9: symbol=_ITM_deregisterTMCloneTable;  lookup in file=/lib64/libpthread.so.0 [0]
         9: symbol=_ITM_deregisterTMCloneTable;  lookup in file=/lib64/libdl.so.2 [0]
         9: symbol=_ITM_deregisterTMCloneTable;  lookup in file=/opt/lib/lib64/libc.so.6 [0]
         9: symbol=_ITM_deregisterTMCloneTable;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
         9: symbol=errno;  lookup in file=/var/task/tegola_lambda [0]
         9: symbol=errno;  lookup in file=/lib64/libpthread.so.0 [0]
         9: symbol=errno;  lookup in file=/lib64/libdl.so.2 [0]
         9: symbol=errno;  lookup in file=/opt/lib/lib64/libc.so.6 [0]
         9: binding file /lib64/libpthread.so.0 [0] to /opt/lib/lib64/libc.so.6 [0]: normal symbol `errno' [GLIBC_PRIVATE]
         9: symbol=__resp;  lookup in file=/var/task/tegola_lambda [0]
         9: symbol=__resp;  lookup in file=/lib64/libpthread.so.0 [0]
         9: symbol=__resp;  lookup in file=/lib64/libdl.so.2 [0]
         9: symbol=__resp;  lookup in file=/opt/lib/lib64/libc.so.6 [0]
         9: binding file /lib64/libpthread.so.0 [0] to /opt/lib/lib64/libc.so.6 [0]: normal symbol `__resp' [GLIBC_PRIVATE]
         9: symbol=__libc_stack_end;  lookup in file=/var/task/tegola_lambda [0]
         9: symbol=__libc_stack_end;  lookup in file=/lib64/libpthread.so.0 [0]
         9: symbol=__libc_stack_end;  lookup in file=/lib64/libdl.so.2 [0]
         9: symbol=__libc_stack_end;  lookup in file=/opt/lib/lib64/libc.so.6 [0]
         9: symbol=__libc_stack_end;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
         9: binding file /lib64/libpthread.so.0 [0] to /lib64/ld-linux-x86-64.so.2 [0]: normal symbol `__libc_stack_end' [GLIBC_2.2.5]
         9: symbol=_rtld_global_ro;  lookup in file=/var/task/tegola_lambda [0]
         9: symbol=_rtld_global_ro;  lookup in file=/lib64/libpthread.so.0 [0]
         9: symbol=_rtld_global_ro;  lookup in file=/lib64/libdl.so.2 [0]
         9: symbol=_rtld_global_ro;  lookup in file=/opt/lib/lib64/libc.so.6 [0]
         9: symbol=_rtld_global_ro;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
         9: binding file /lib64/libpthread.so.0 [0] to /lib64/ld-linux-x86-64.so.2 [0]: normal symbol `_rtld_global_ro' [GLIBC_PRIVATE]
         9: symbol=__libc_dl_error_tsd;  lookup in file=/var/task/tegola_lambda [0]
         9: symbol=__libc_dl_error_tsd;  lookup in file=/lib64/libpthread.so.0 [0]
         9: symbol=__libc_dl_error_tsd;  lookup in file=/lib64/libdl.so.2 [0]
         9: symbol=__libc_dl_error_tsd;  lookup in file=/opt/lib/lib64/libc.so.6 [0]
         9: /lib64/libpthread.so.0: error: relocation error: symbol __libc_dl_error_tsd, version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference (fatal)
/var/task/tegola_lambda: relocation error: /lib64/libpthread.so.0: symbol __libc_dl_error_tsd, version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference
END RequestId: d37e3d63-672a-49f9-b3c1-866173d2fb75
REPORT RequestId: d37e3d63-672a-49f9-b3c1-866173d2fb75  Duration: 67.20 ms  Billed Duration: 68 ms  Memory Size: 128 MB Max Memory Used: 6 MB   
RequestId: d37e3d63-672a-49f9-b3c1-866173d2fb75 Process exited before completing request

[matrottier]

oh! locally build v0.13.0 with option "CGO_ENABLED=0" and go 1.16.6 (was 1.13.8 before (ubuntu 20.04) ) seems to work, i got a timeout after 15sec. probably a problem in my configuration of the PG database. I will be able to move forward

[ARolek]

@matrottier I"m going to reopen this issue until I have a chance to investigate why the build pipeline is not producing a binary that just works. Here's the build command:

https://github.com/go-spatial/tegola/blob/126d51113f8ecf362a9235449161a0f62367895c/.github/workflows/on_release_publish.yml#L52-L58

It looks like you disabled CGO, but that's not desirable as Geopkg will not work without it. I'm not quite sure what changed (something with Lambda?) but I would like to make sure no one else has to deal with this juggle.

[ARolek]

@matrottier I was able to recreate your problem, and I'm not quite sure when this started. AWS Lambda runs on AmazonLinux and we're currently compiling on the GH Action runner ubuntu-latest. There's clearly a mismatch of GLIB location / version / expectations. I'm working on updating the CI so we don't encounter this anymore.

Thanks again for reporting this.

[ARolek]

@matrottier can you try the binary output from this Github Action run: https://github.com/go-spatial/tegola/actions/runs/1043951686

I refactored the way we're building tegola_lambda and I'm now using Amazon Linux for that build. I'm hoping you can give it a test and confirm it's working on your end. It's working for me with my tests.

[matrottier]

it works !! (no problem with GLIBC) but i got this error from API gateway, any idea ?
"Internal Server Error The IAM role configured on the integration or API Gateway doesn't have permissions to call the integration. Check the permissions and try again"

[ARolek]

@matrottier excellent! Thanks for checking that out. Regarding the IAM issue, that sounds like it has to do with your resource access control rules setup in your AWS account, not specific to tegola. I have personally found that using the Application Load Balancer easier to set up the API Gateway. Considering giving that a shot and then moving to API Gateway.

[ARolek]

@matrottier I just sent in a PR to fix this if you want to give it a look: https://github.com/go-spatial/tegola/pull/790

[matrottier]

trying to upgrade from 0.16 and i got the same message :

/var/task/bootstrap: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by /var/task/bootstrap)
/var/task/bootstrap: /lib64/libc.so.6: version `GLIBC_2.33' not found (required by /var/task/bootstrap)
/var/task/bootstrap: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by /var/task/bootstrap)
/var/task/bootstrap: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /var/task/bootstrap)

with my own compiled binary, the file of the github 0.17, 0.18 and 0.19. i tested with SAM and in my CDK code i use this option : runtime: lambda.Runtime.GO_1_X. also tried lambda.Runtime.PROVIDED_AL2,

searching for a hint ! :pray:

[ARolek]

@matrottier you're saying the version you download from the tegola releases page is is throwing this error? A few things to note:

• You need to use the al2 runtime now for Go based lambda functions
• Our release page has 2 lambda architectures: arm and amd64. make sure you pick the right architecture for your deployment.

[matrottier]

move along, nothing to see here ! just a old man who has trouble with technology

it's working with al2 and tegola_lambda_amd64.zip thanks for the quick reply @ARolek

[ARolek]

@matrottier nice! and anytime ;-)