Closed meyer1994 closed 4 years ago
vincentsarago
commented
4 years ago @meyer1994 thanks again for the issue. I should have check that, and you are also right about libsqlite3.
The only solution I can think of is to move or build those libraries at build time to /opt/lib
I don't have time this week to work on this sadly
vincentsarago
commented
4 years ago @meyer1994 in fact it goes down to understand what libraries are available in lambda runtime.
we are using lambci/lambda-base:build docker image to make sure we compile GDAL is the exact same environement as in lambda so if libsqlite3.so.0 is link to gdal.so it means it might be available in lambda ....
Thus we just need to copy in /opt/lib the libraries that are not in lambda (proj, goes ...)
I'll continue to investigate
vincentsarago
commented
4 years ago @meyer1994 just run ldd libgdal.so on lambda runtime directly and seems that everything is almost here
linux-vdso.so.1 => (0x00007ffee3f9b000)
libcrypto.so.10 => /var/lang/lib/libcrypto.so.10 (0x00007f6407733000)
libgeos_c.so.1 => /opt/lib/libgeos_c.so.1 (0x00007f6407500000)
libwebp.so.7 => /opt/lib/libwebp.so.7 (0x00007f6407293000)
libsqlite3.so.0 => /usr/lib64/libsqlite3.so.0 (0x00007f6406fe0000)
libexpat.so.1 => /lib64/libexpat.so.1 (0x00007f6406db7000)
libopenjp2.so.7 => /opt/lib/libopenjp2.so.7 (0x00007f6406b5d000)
libjpeg.so.62 => /opt/lib/libjpeg.so.62 (0x00007f64068cc000)
libpng16.so.16 => /opt/lib/libpng16.so.16 (0x00007f640669a000)
libzstd.so.1 => /opt/lib/libzstd.so.1 (0x00007f640641b000)
libproj.so.15 => /opt/lib/libproj.so.15 (0x00007f6405f8f000)
libz.so.1 => /lib64/libz.so.1 (0x00007f6405d79000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6405b5d000)
librt.so.1 => /lib64/librt.so.1 (0x00007f6405955000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f6405751000)
libcurl.so.4 => /usr/lib64/libcurl.so.4 (0x00007f64054ca000)
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f6405145000)
libm.so.6 => /lib64/libm.so.6 (0x00007f6404e43000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f6404c2d000)
libc.so.6 => /lib64/libc.so.6 (0x00007f6404860000)
/lib64/ld-linux-x86-64.so.2 (0x00007f6408db0000)
libgeos-3.7.2.so => /opt/lib/libgeos-3.7.2.so (0x00007f64044b0000)
libnghttp2.so.14 => /usr/lib64/libnghttp2.so.14 (0x00007f640428b000)
libidn2.so.0 => /usr/lib64/libidn2.so.0 (0x00007f640406a000)
libssh2.so.1 => /usr/lib64/libssh2.so.1 (0x00007f6403e42000)
libpsl.so.0 => /usr/lib64/libpsl.so.0 (0x00007f6403bcd000)
libssl.so.10 => /var/lang/lib/libssl.so.10 (0x00007f640395c000)
libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2 (0x00007f640370f000)
libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x00007f6403426000)
libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 (0x00007f640320b000)
libcom_err.so.2 => /usr/lib64/libcom_err.so.2 (0x00007f6403008000)
libldap-2.4.so.2 => /lib64/libldap-2.4.so.2 (0x00007f6402db5000)
liblber-2.4.so.2 => /lib64/liblber-2.4.so.2 (0x00007f6402ba6000)
libunistring.so.0 => /usr/lib64/libunistring.so.0 (0x00007f6402890000)
libicuuc.so.50 => /usr/lib64/libicuuc.so.50 (0x00007f640251a000)
libkrb5support.so.0 => /usr/lib64/libkrb5support.so.0 (0x00007f640230b000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f6402108000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f6401eef000)
libsasl2.so.2 => /usr/lib64/libsasl2.so.2 (0x00007f6401cd4000)
libssl3.so => /usr/lib64/libssl3.so (0x00007f6401a82000)
libsmime3.so => /usr/lib64/libsmime3.so (0x00007f640185b000)
libnss3.so => /usr/lib64/libnss3.so (0x00007f6401536000)
libnssutil3.so => /usr/lib64/libnssutil3.so (0x00007f6401307000)
libplds4.so => /lib64/libplds4.so (0x00007f6401103000)
libplc4.so => /lib64/libplc4.so (0x00007f6400efe000)
libnspr4.so => /lib64/libnspr4.so (0x00007f6400cc0000)
libicudata.so.50 => /usr/lib64/libicudata.so.50 (0x00007f63ff6ed000)
libselinux.so.1 => /usr/lib64/libselinux.so.1 (0x00007f63ff4cc000)
libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f63ff295000)
libfreebl3.so => /lib64/libfreebl3.so (0x00007f63ff093000)the only differences I see when running this in remotepixel/amazonlinux:gdal3.0-py3.7 are:
libpcre.so.0
libxml2.so.2
liblzma.so.5What we can see is first, we might be missing libxml in runtime and Xerces is definitely not in lambda
vincentsarago
commented
4 years ago @meyer1994 I cannot reproduce the error you mentioned, can you share more about how you created your layer or pacakge ?
meyer1994
commented
4 years ago I am trying to reproduce it, unsuccessfully. I will try to create a reproducible example in the next couple of days.
Weirdly, my stack is working based on waht I said. It wasn't working without copying the libexpat file to the layer...
Weirdly, again, I have not copied the libxml2.so.2. It seems that it would be necessairy, GML is XML based.
vincentsarago
commented
4 years ago after other checks it seems that libxml2 is present on lambda runtime env 🤔.
vincentsarago
commented
4 years ago @meyer1994 with the latest update I don't see how this could still be a problem. Can you test with the latest lambda layer or docker images and close here if everything works?
meyer1994
commented
4 years ago Weird. The error persists :(
Just tried with the latest layer:
arn:aws:lambda:eu-central-1:524387336408:layer:gdal30-py37-geo:5When trying to open .gml it returns None. So it fails right after. I couldn't reproduce the first bug though. The one with:
Xerces or Expat support not configured into GDAL/OGRSample lambda:
from osgeo import gdal, ogr
gdal.UseExceptions()
ogr.UseExceptions()
gdal.SetConfigOption('AWS_DEFAULT_REGION', 'eu-central-1')
gdal.SetConfigOption('AWS_REQUEST_PAYER', 'requester')
BUCKET = 'sentinel-s2-l1c'
KEY = 'tiles/22/J/GQ/2019/8/3/0/qi/MSK_CLOUDS_B00.gml'
def handler(event, context):
key = f'/vsis3/{BUCKET}/{KEY}'
mask = ogr.Open(key)
print('Layers:', mask.GetLayerCount())
layer = mask.GetLayer()
print('Features:', layer.GetFeatureCount())@meyer1994 thanks I'm not sure what's going on. When running the code on the docker image it works just file but on the lambda it fails. both libgdal.so are linked to the same libs so it should work there is maybe some environement variables that needs to be configured ?
meyer1994
commented
4 years ago You sure it works fine?
I am running it on remotepixel/amazonlinux:gdal3.0-py3.7 and it is returning None when opening the .gml.
vincentsarago
commented
4 years ago I've try with remotepixel/amazonlinux:gdal3.0-py3.7-geo
docker run \
--name dockerimage \
-p 8000:8000 \
--env AWS_ACCOUNT_ID=${AWS_ACCOUNT_ID} \
--env AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
--env AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
--env AWS_REGION=${AWS_REGION} \
--volume $(pwd)/:/local \
--rm -it remotepixel/amazonlinux:gdal3.0-py3.7-geo bash
bash-4.2# python
from osgeo import gdal, ogr
gdal.UseExceptions()
ogr.UseExceptions()
gdal.SetConfigOption('AWS_DEFAULT_REGION', 'eu-central-1')
gdal.SetConfigOption('AWS_REQUEST_PAYER', 'requester')
BUCKET = 'sentinel-s2-l1c'
KEY = 'tiles/22/J/GQ/2019/8/3/0/qi/MSK_CLOUDS_B00.gml'
key = f'/vsis3/{BUCKET}/{KEY}'
mask = ogr.Open(key)
print('Layers:', mask.GetLayerCount())
Layers: 1
layer = mask.GetLayer()
print('Features:', layer.GetFeatureCount())
Features: 1I tried somethings around, with no avail...
It seems to me that adding the libraries to the layer by hand, using:
zip -r9 --symlinks /tmp/${PACKAGE_NAME}.zip /lib64/libexpat.so*Is the best solution for now.
If you want you can close this issue and reopen it when it seems fit.
vincentsarago
commented
4 years ago 🤔 but it seems to me that /lib64/libexpat.so is already in the environment!
are you using python3.7 ?
meyer1994
commented
4 years ago I am using python3.7.
but it seems to me that /lib64/libexpat.so is already in the environment!
Yes, it is. But it is in the wrong place. When you add the file to the layer, it will be unzipped in /opt/lib or /opt/lib64 when the layer is deployed. I think your built version of gdal searches for it on /opt.
Using this command in the layer creation:
$ zip -r9 --symlinks /tmp/${PACKAGE_NAME}.zip /lib64/libexpat.so*After the layer is deployed into the lambda, the library will be in /opt/lib64.
vincentsarago
commented
4 years ago When you add the file to the layer
we don't ship libexpact with the layer.
I think your built version of gdal searches for it on /opt.
ldd libgdal.so shows it is linked to /lib64/libexpat.so so I'm not sure why it can't find it 🤔
meyer1994
commented
4 years ago we don't ship libexpact with the layer.
Yes yes. I was trying to say when someone adds the file to the layer (like I did). My bad.
ldd libgdal.soshows it is linked to/lib64/libexpat.soso I'm not sure why it can't find it :thinking:
I don't know what else to do. For now, my version is working with the copied libexpat. Feel free to close it if you think is not worth it.
meyer1994
commented
4 years ago Ok. I am ashamed.
The new layer works!!! The problem was with IAM roles... Adding permissions to access sentinel bucket made it work fine.
See serverless.yml:
service: layertest
provider:
name: aws
runtime: python3.7
stage: dev
versionFunctions: false
region: eu-central-1
environment:
PROJ_LIB: '/opt/share/proj'
GEOS_CONFIG: '/opt/bin/geos-config'
GDAL_DATA: '/opt/share/gdal'
GDAL_CONFIG: '/opt/bin/gdal-config'
GDAL_CACHEMAX: '512'
GDAL_HTTP_MERGE_CONSECUTIVE_RANGES: 'YES'
GDAL_HTTP_MULTIPLEX: 'YES'
GDAL_HTTP_VERSION: '2'
GDAL_DISABLE_READDIR_ON_OPEN: 'EMPTY_DIR'
VSI_CACHE: 'true'
VSI_CACHE_SIZE: '536870912'
CPL_TMPDIR: '/tmp'
CPL_VSIL_CURL_ALLOWED_EXTENSIONS: '.tif,.jp2,.gml'
iamRoleStatements:
- Effect: Allow
Action:
- s3:GetObject
Resource:
- arn:aws:s3:::sentinel-s2-l1c*
package:
include:
- handler.py
functions:
layer:
handler: handler.handler
layers:
- arn:aws:lambda:eu-central-1:524387336408:layer:gdal30-py37-geo:5And handler.py:
import glob
from osgeo import gdal, ogr
gdal.UseExceptions()
ogr.UseExceptions()
gdal.SetConfigOption('AWS_DEFAULT_REGION', 'eu-central-1')
gdal.SetConfigOption('AWS_REQUEST_PAYER', 'requester')
BUCKET = 'sentinel-s2-l1c'
KEY = 'tiles/22/J/GQ/2019/8/3/0/qi/MSK_CLOUDS_B00.gml'
def handler(event, context):
libs = glob.glob('/opt/libs*/*')
libs = sorted(libs)
print('\n'.join(libs))
key = f'/vsis3/{BUCKET}/{KEY}'
mask = ogr.Open(key)
print('Layers:', mask.GetLayerCount())
layer = mask.GetLayer()
print('Features:', layer.GetFeatureCount())After deploying:
$ sls invoke -f layer -l
null
--------------------------------------------------------------------
START RequestId: a4121a73-5241-40ab-b867-62f4c3e4f735 Version: $LATEST
Warning 1: HTTP response code on https://sentinel-s2-l1c.s3.amazonaws.com/tiles/22/J/GQ/2019/8/3/0/qi/MSK_CLOUDS_B00.resolved.gml: 403
Layers: 1
Features: 1
END RequestId: a4121a73-5241-40ab-b867-62f4c3e4f735
REPORT RequestId: a4121a73-5241-40ab-b867-62f4c3e4f735 Duration: 265.57 ms Billed Duration: 300 ms Memory Size: 1024 MB Max Memory Used: 111 MB Init Duration: 358.54 ms🎉 nothing to be ashamed, this happened to me so many time 😄 Glad you figured out and can't wait to see what you are building!
After PR #12 got merged, I tried to create a layer with the script located here. But I still got the same errors:
After some intense search I've stumbled upon this doc which led me to
ldd.From inside the docker image I ran
$ ldd /opt/lib/libgdal.so. Lo and behold, lots of info came out of it:It appear that the build process for gdal does not place every library inside the
/opt/libdir. So, when creating the layer, we should add the libraries by hand (unfortunately).The addition of the following line to the
create-lambda-layer.shfile fixed the error for me:However, I think there are more problems that may appear from this problem. For example,
libsqlite3.so.0not included by the script as well. I have not tested it. But it is entirely possible.I did not make a PR because I am not sure how to fix this in a more generic way. Maybe adding each lib by hand is the best option, but I am not sure.