search

lambgeo / docker-lambda

AWS Lambda friendly GDAL Docker images and AWS Lambda layer
MIT License
121 stars 17 forks source link

libcurl and libxml2 are missing in amazonlinux 2 #34

Closed vincentsarago closed 2 years ago

vincentsarago commented 3 years ago

😭 running os.system("ldd /opt/bin/gdalinfo") in aws lambda gives: 👇 

        linux-vdso.so.1 (0x00007ffe64732000)
    libgdal.so => /opt/bin/../lib/libgdal.so (0x00007f66747eb000)
    libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f6674396000)
    libgeos_c.so.1 => /opt/bin/../lib/libgeos_c.so.1 (0x00007f667415c000)
    libwebp.so.7 => /opt/bin/../lib/libwebp.so.7 (0x00007f6673ef2000)
    libexpat.so.1 => /opt/bin/../lib/libexpat.so.1 (0x00007f6673cc2000)
    libopenjp2.so.7 => /opt/bin/../lib/libopenjp2.so.7 (0x00007f6673a6a000)
    libnetcdf.so.18 => /opt/bin/../lib/libnetcdf.so.18 (0x00007f6673737000)
    libhdf5.so.200 => /opt/bin/../lib/libhdf5.so.200 (0x00007f6673065000)
    libmfhdf.so.0 => /opt/bin/../lib/libmfhdf.so.0 (0x00007f6672e3b000)
    libdf.so.0 => /opt/bin/../lib/libdf.so.0 (0x00007f6672b8a000)
    libjpeg.so.62 => /opt/bin/../lib/libjpeg.so.62 (0x00007f66728f6000)
    libgeotiff.so.5 => /opt/bin/../lib/libgeotiff.so.5 (0x00007f66726c2000)
    libpng16.so.16 => /opt/bin/../lib/libpng16.so.16 (0x00007f6672491000)
    libpq.so.5 => /opt/bin/../lib/libpq.so.5 (0x00007f6672248000)
    libzstd.so.1 => /opt/bin/../lib/libzstd.so.1 (0x00007f6671fc0000)
    libproj.so.19 => /opt/bin/../lib/libproj.so.19 (0x00007f6671b01000)
    libsqlite3.so.0 => /opt/bin/../lib/libsqlite3.so.0 (0x00007f66717f0000)
    libtiff.so.5 => /opt/bin/../lib/libtiff.so.5 (0x00007f6671572000)
    libdeflate.so.0 => /opt/bin/../lib/libdeflate.so.0 (0x00007f6671365000)
    libz.so.1 => /lib64/libz.so.1 (0x00007f6671150000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6670f32000)
    librt.so.1 => /lib64/librt.so.1 (0x00007f6670d2a000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f6670b26000)
    libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f66708c2000)
    libcurl.so.4 => not found
    libxml2.so.2 => not found
    libm.so.6 => /lib64/libm.so.6 (0x00007f6670582000)
    libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f6670200000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f666ffea000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f666fc3f000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f66753a2000)
    libcurl.so.4 => not found
    libxml2.so.2 => not found
    libgeos-3.8.1.so => /opt/lib/libgeos-3.8.1.so (0x00007f666f874000)
    libhdf5_hl.so.200 => /opt/lib/libhdf5_hl.so.200 (0x00007f666f652000)
    libsz.so.2 => /opt/lib/libsz.so.2 (0x00007f666f43e000)
    libcurl.so.4 => not found
    libcurl.so.4 => not found
    liblzma.so.5 => /var/lang/lib/liblzma.so.5 (0x00007f666f218000)
    libssl.so.10 => /lib64/libssl.so.10 (0x00007f666efa9000)
    libcurl.so.4 => not found
    libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f666ed5d000)
    libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f666ea79000)
    libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f666e875000)
    libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f666e644000)
    libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f666e435000)
    libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f666e231000)
    libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f666e01b000)
    libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f666ddf4000)
vincentsarago commented 3 years ago

fixed in https://github.com/lambgeo/docker-lambda/commit/09782316a0fb8a90eef944aa2b76fee701775c57

ThiagoDiasV commented 2 years ago

After the last updates and the new Dockerfile to use gdal3.5 we are missing libcurl and libxml2 again.

Here's the output of ldd /opt/bin/ogr2ogr in a amazon linux 2 instance running python 3.9 version:

\tlinux-vdso.so.1 (0x00007fff173fd000)
\tlibgdal.so.31 => /opt/bin/../lib/libgdal.so.31 (0x00007fb82bf5d000)
\tlibstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fb82bbdb000)
\tlibm.so.6 => /lib64/libm.so.6 (0x00007fb82b89b000)
\tlibgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fb82b685000)
\tlibc.so.6 => /lib64/libc.so.6 (0x00007fb82b2da000)
\tlibdl.so.2 => /lib64/libdl.so.2 (0x00007fb82b0d6000)
\tlibz.so.1 => /lib64/libz.so.1 (0x00007fb82aec1000)
\tlibcurl.so.4 => not found
\tlibxml2.so.2 => not found
\tlibcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007fb82aa6b000)
\tlibssl.so.10 => /lib64/libssl.so.10 (0x00007fb82a7fc000)
\tlibzstd.so.1 => /opt/bin/../lib/libzstd.so.1 (0x00007fb82a574000)
\tliblzma.so.5 => /var/lang/lib/liblzma.so.5 (0x00007fb82a34e000)
\tlibdeflate.so.0 => /opt/bin/../lib/libdeflate.so.0 (0x00007fb82a13e000)
\tlibblosc.so.1 => /opt/bin/../lib/libblosc.so.1 (0x00007fb829f18000)
\tlibproj.so.22 => /opt/bin/../lib/libproj.so.22 (0x00007fb829a25000)
\tlibexpat.so.1 => /opt/bin/../lib/libexpat.so.1 (0x00007fb8297f5000)
\tlibjpeg.so.62 => /opt/bin/../lib/libjpeg.so.62 (0x00007fb82954e000)
\tlibtiff.so.5 => /opt/bin/../lib/libtiff.so.5 (0x00007fb8292ce000)
\tlibgeotiff.so.5 => /opt/bin/../lib/libgeotiff.so.5 (0x00007fb82909a000)
\tlibpng16.so.16 => /opt/bin/../lib/libpng16.so.16 (0x00007fb828e69000)
\tlibLercLib.so => /opt/bin/../lib/libLercLib.so (0x00007fb828c05000)
\tlibnetcdf.so.18 => /opt/bin/../lib/libnetcdf.so.18 (0x00007fb8288d2000)
\tlibhdf5.so.200 => /opt/bin/../lib/libhdf5.so.200 (0x00007fb828200000)
\tlibsz.so.2 => /opt/bin/../lib/libsz.so.2 (0x00007fb827fec000)
\tlibwebp.so.7 => /opt/bin/../lib/libwebp.so.7 (0x00007fb827d82000)
\tlibsqlite3.so.0 => /opt/bin/../lib/libsqlite3.so.0 (0x00007fb827a71000)
\tlibpq.so.5 => /opt/bin/../lib/libpq.so.5 (0x00007fb827828000)
\tlibopenjp2.so.7 => /opt/bin/../lib/libopenjp2.so.7 (0x00007fb8275d4000)
\tlibpcre.so.1 => /lib64/libpcre.so.1 (0x00007fb827370000)
\tlibgeos_c.so.1 => /opt/bin/../lib/libgeos_c.so.1 (0x00007fb827133000)
\tlibpthread.so.0 => /lib64/libpthread.so.0 (0x00007fb826f15000)
\tlibdf.so.0 => /opt/bin/../lib/libdf.so.0 (0x00007fb826c64000)
\tlibmfhdf.so.0 => /opt/bin/../lib/libmfhdf.so.0 (0x00007fb826a3a000)
\t/lib64/ld-linux-x86-64.so.2 (0x00007fb82d768000)
\tlibgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007fb8267ee000)
\tlibkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007fb82650a000)
\tlibcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007fb826306000)
\tlibk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007fb8260d5000)
\tlibcurl.so.4 => not found
\tlibcurl.so.4 => not found
\tlibhdf5_hl.so.200 => /opt/lib/libhdf5_hl.so.200 (0x00007fb825eb3000)
\tlibcurl.so.4 => not found
\tlibgeos.so.3.10.2 => /opt/bin/../lib/libgeos.so.3.10.2 (0x00007fb825a50000)
\tlibkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007fb825841000)
\tlibkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007fb82563d000)
\tlibresolv.so.2 => /lib64/libresolv.so.2 (0x00007fb825427000)
\tlibselinux.so.1 => /lib64/libselinux.so.1 (0x00007fb825200000)
vincentsarago commented 2 years ago

Oh no I though I check this 😬

vincentsarago commented 2 years ago

🤔

ghcr.io/lambgeo/lambda-gdal:3.5-python3.9 image

$ docker run --entrypoint bash --platform=linux/amd64 -it --rm ghcr.io/lambgeo/lambda-gdal:3.5-python3.9
$ ldd /opt/bin/gdalinfo
        libgdal.so.31 => /opt/bin/../lib/libgdal.so.31 (0x0000004001a2b000)
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x0000004003236000)
        libm.so.6 => /lib64/libm.so.6 (0x00000040035b8000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00000040038f8000)
        libc.so.6 => /lib64/libc.so.6 (0x0000004003b0e000)
        libdl.so.2 => /lib64/libdl.so.2 (0x0000004003eb9000)
        libz.so.1 => /lib64/libz.so.1 (0x00000040040bf000)
        libcurl.so.4 => /lib64/libcurl.so.4 (0x00000040042d4000)
        libxml2.so.2 => /lib64/libxml2.so.2 (0x000000400456b000)
        libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00000040048c4000)
        libssl.so.10 => /lib64/libssl.so.10 (0x0000004004d1a000)
        libzstd.so.1 => /opt/bin/../lib/libzstd.so.1 (0x0000004004f89000)
        liblzma.so.5 => /var/lang/lib/liblzma.so.5 (0x0000004005213000)
        libdeflate.so.0 => /opt/bin/../lib/libdeflate.so.0 (0x0000004005438000)
        libblosc.so.1 => /opt/bin/../lib/libblosc.so.1 (0x0000004005648000)
        libproj.so.22 => /opt/bin/../lib/libproj.so.22 (0x000000400586e000)
        libexpat.so.1 => /opt/bin/../lib/libexpat.so.1 (0x0000004005d61000)
        libjpeg.so.62 => /opt/bin/../lib/libjpeg.so.62 (0x0000004005f91000)
        libtiff.so.5 => /opt/bin/../lib/libtiff.so.5 (0x000000400623a000)
        libgeotiff.so.5 => /opt/bin/../lib/libgeotiff.so.5 (0x00000040064ba000)
        libpng16.so.16 => /opt/bin/../lib/libpng16.so.16 (0x00000040066ee000)
        libLercLib.so => /opt/bin/../lib/libLercLib.so (0x000000400691f000)
        libnetcdf.so.18 => /opt/bin/../lib/libnetcdf.so.18 (0x0000004006b83000)
        libhdf5.so.200 => /opt/bin/../lib/libhdf5.so.200 (0x0000004006eb6000)
        libsz.so.2 => /opt/bin/../lib/libsz.so.2 (0x000000400758a000)
        libwebp.so.7 => /opt/bin/../lib/libwebp.so.7 (0x000000400779e000)
        libsqlite3.so.0 => /opt/bin/../lib/libsqlite3.so.0 (0x0000004007a08000)
        libpq.so.5 => /opt/bin/../lib/libpq.so.5 (0x0000004007d19000)
        libopenjp2.so.7 => /opt/bin/../lib/libopenjp2.so.7 (0x0000004007f62000)
        libpcre.so.1 => /lib64/libpcre.so.1 (0x00000040081b6000)
        libgeos_c.so.1 => /opt/bin/../lib/libgeos_c.so.1 (0x000000400841c000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x0000004008659000)
        libdf.so.0 => /opt/bin/../lib/libdf.so.0 (0x0000004008877000)
        libmfhdf.so.0 => /opt/bin/../lib/libmfhdf.so.0 (0x0000004008b28000)
        /lib64/ld-linux-x86-64.so.2 (0x0000004000000000)
        libnghttp2.so.14 => /lib64/libnghttp2.so.14 (0x0000004008d52000)
        libidn2.so.0 => /lib64/libidn2.so.0 (0x0000004008f7b000)
        libssh2.so.1 => /lib64/libssh2.so.1 (0x00000040091ca000)
        libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00000040093f2000)
        libkrb5.so.3 => /lib64/libkrb5.so.3 (0x000000400963e000)
        libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x0000004009922000)
        libcom_err.so.2 => /lib64/libcom_err.so.2 (0x0000004009b53000)
        libldap-2.4.so.2 => /lib64/libldap-2.4.so.2 (0x0000004009d59000)
        liblber-2.4.so.2 => /lib64/liblber-2.4.so.2 (0x0000004009fab000)
        libhdf5_hl.so.200 => /opt/lib/libhdf5_hl.so.200 (0x000000400a1ba000)
        libgeos.so.3.10.2 => /opt/bin/../lib/libgeos.so.3.10.2 (0x000000400a3dc000)
        libunistring.so.0 => /lib64/libunistring.so.0 (0x000000400a841000)
        libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x000000400ab59000)
        libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x000000400ad68000)
        libresolv.so.2 => /lib64/libresolv.so.2 (0x000000400af6c000)
        libsasl2.so.3 => /lib64/libsasl2.so.3 (0x000000400b182000)
        libssl3.so => /lib64/libssl3.so (0x000000400b39e000)
        libsmime3.so => /lib64/libsmime3.so (0x000000400b5ff000)
        libnss3.so => /lib64/libnss3.so (0x000000400b825000)
        libnssutil3.so => /lib64/libnssutil3.so (0x000000400bb52000)
        libplds4.so => /lib64/libplds4.so (0x000000400bd81000)
        libplc4.so => /lib64/libplc4.so (0x000000400bf85000)
        libnspr4.so => /lib64/libnspr4.so (0x000000400c18a000)
        libselinux.so.1 => /lib64/libselinux.so.1 (0x000000400c3c8000)
        libcrypt.so.1 => /var/lang/lib/libcrypt.so.1 (0x000000400c5ef000)
        librt.so.1 => /lib64/librt.so.1 (0x000000400c826000)

$ ls /lib64/ | grep "curl"
libcurl.so.4
libcurl.so.4.7.0
$ ls /lib64/ | grep "xml2"
libxml2.so.2
libxml2.so.2.9.1

public.ecr.aws/lambda/python:3.9

docker run --entrypoint bash --platform=linux/amd64 -it --rm public.ecr.aws/lambda/python:3.9
$  ls /lib64 | grep "curl"
libcurl.so.4
libcurl.so.4.7.0
$ ls /lib64 | grep "xml2"
libxml2.so.2
libxml2.so.2.9.1

@ThiagoDiasV ☝️ seems to tell the libraries a present 🤷

vincentsarago commented 2 years ago

😬 public.ecr.aws/lambda/python:3.9 do not match the AWS Lambda environment 🤬🤬🤬🤬🤬

ThiagoDiasV commented 2 years ago

What do you mean by public.ecr.aws/lambda/python:3.9 doesn't match the AWS lambda env?

vincentsarago commented 2 years ago

What do you mean by public.ecr.aws/lambda/python:3.9 doesn't match the AWS lambda env?

as shown in previous comment we see /lib64/libcurl.so.4 in public.ecr.aws/lambda/python:3.9 but when doing ls /lib64 | grep "lib curl" in AWS Lambda it returns empty

ThiagoDiasV commented 2 years ago

Thank you mister for the quick fix. I already tested and everything went fine.

But I think you forgot to update the layer versions in layer.json file. I deployed a new lambda successfully but using arn:aws:lambda:{region}:524387336408:layer:gdal35:3 instead of arn:aws:lambda:{region}:524387336408:layer:gdal35:2

vincentsarago commented 2 years ago

🙏

But I think you forgot to update the layer versions in layer.json

Yes I went to bed before the layer were published 😄, I'll do it now