search

RemotePixel / amazonlinux

Create Geospatial ready AWS lambda layer and docker image
MIT License
12 stars 3 forks source link

install newer sqlite version #17

Closed vincentsarago closed 4 years ago

vincentsarago commented 4 years ago

closes #16

as mentioned in https://github.com/OSGeo/PROJ/pull/1721 PROJ 6 needs SQLite > 3.11 to have good performances!

vincentsarago commented 4 years ago

This Branch

$ ldd /opt/lib/libgdal.so | grep "sql"        
libsqlite3.so.0 => /opt/lib/libsqlite3.so.0 (0x00007f1bc7b9b000)

$ ldd /opt/lib/libproj.so | grep "sql"
libsqlite3.so.0 => /opt/lib/libsqlite3.so.0 (0x00007fae78409000)

$ ldd /opt/lib/libsqlite3.so
linux-vdso.so.1 =>  (0x00007ffd311b4000)
libz.so.1 => /lib64/libz.so.1 (0x00007f93760fd000)
libm.so.6 => /lib64/libm.so.6 (0x00007f9375dfb000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f9375bf7000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f93759db000)
libc.so.6 => /lib64/libc.so.6 (0x00007f937560e000)
/lib64/ld-linux-x86-64.so.2 (0x00007f9376620000)

$ /opt/bin/sqlite3 --version
3.30.1 2019-10-10 20:19:45 18db032d058f1436ce3dea84081f4ee5a0f2259ad97301d43c426bc7f3df1b0b

 time echo "0 0" | cs2cs +proj=longlat +datum=WGS84 +to +init=epsg:2163
9473741.42      1181205.06 0.00
real    0m0.146s
user    0m0.130s
sys     0m0.000s

time GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR gdalwarp -q -te -11036283.891926888 10331840.239250703 -10958012.374962866 10410111.756214723 -t_srs EPSG:3857 -ts 256 256 -r bilinear -overwrite /vsicurl/https://s3.amazonaws.com/opendata.remotepixel.ca/bench_tiler/LC08_L1TP_040013_20191014_20191029_01_T1_B4.tif out.tif
real    0m3.969s
user    0m2.160s
sys     0m0.080s

Master

$ sqlite3 --version
3.7.17 2013-05-20 00:56:22 118a3b35693b134d56ebd780123b7fd6f1497668

$ time echo "0 0" | cs2cs +proj=longlat +datum=WGS84 +to +init=epsg:2163
9473741.42      1181205.06 0.00
real    0m5.262s
user    0m4.990s
sys     0m0.280s

$ time GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR gdalwarp -q -te -11036283.891926888 10331840.239250703 -10958012.374962866 10410111.756214723 -t_srs EPSG:3857 -ts 256 256 -r bilinear -overwrite /vsicurl/https://s3.amazonaws.com/opendata.remotepixel.ca/bench_tiler/LC08_L1TP_040013_20191014_20191029_01_T1_B4.tif out.tif
real    0m3.974s
user    0m1.970s
sys     0m0.120s

Updating SQLite to > 3.11 (3.30) did fix performances of PROJ (cs2cs command) but has no influence on gdalwarp.

There might be something else going on :-(

cc @rouault

rouault commented 4 years ago

but has no influence on gdalwarp.

what about ldd on gdalwarp / libgdal.so to check which sqlite you're linking against ?

vincentsarago commented 4 years ago

what about ldd on gdalwarp / libgdal.so to check which sqlite you're linking against ?

$ ldd /opt/bin/cs2cs | grep "sql"
        libsqlite3.so.0 => /opt/lib/libsqlite3.so.0 (0x00007f0cc6415000)
$ ldd /opt/lib/libgdal.so | grep "sqlite"
        libsqlite3.so.0 => /opt/lib/libsqlite3.so.0 (0x00007f16d615c000)
$ ldd /opt/bin/gdalwarp | grep "sqlite"
        libsqlite3.so.0 => /usr/lib64/libsqlite3.so.0 (0x00007fd24d448000)

So gdalwarp is linked to the old sqlite while libgdal and cs2cs are linked to the new one.

Right on @rouault, when I force /opt/lib to be first in LD_LIBRARY_PATH it then work fine. But still wondering why cs2cs found the good sqlite while gdalwarp doesn't.

export LD_LIBRARY_PATH=/opt/lib:$LD_LIBRARY_PATH
$ ldd /opt/bin/gdalwarp | grep "sqlite"
        libsqlite3.so.0 => /opt/lib/libsqlite3.so.0 (0x00007f86d7f84000)

$ time GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR gdalwarp -q -te -11036283.891926888 10331840.239250703 -10958012.374962866 10410111.756214723 -t_srs EPSG:3857 -ts 256 256 -r bilinear -overwrite /vsicurl/https://s3.amazonaws.com/opendata.remotepixel.ca/bench_tiler/LC08_L1TP_040013_20191014_20191029_01_T1_B4.tif out.tif
real    0m0.780s
user    0m0.120s
sys     0m0.040s
vincentsarago commented 4 years ago

Ok, seems that PROJ user rpath by default to its binaries while gdal doesn't. I think I know what to do next. thanks a lot @rouault

edit: ☝️ I think this is my fault. I set $ORIGIN in ligdal.so manually https://github.com/RemotePixel/amazonlinux/blob/master/base/gdal3.0/Dockerfile#L168-L170

$ readelf -d /opt/bin/cs2cs | grep "rpath"
 0x000000000000000f (RPATH)              Library rpath: [/opt/lib]

$ readelf -d /opt/bin/gdalwarp | grep "rpath"
empty

$ readelf -d /opt/lib/libgdal.so | grep "rpath"
 0x000000000000000f (RPATH)              Library rpath: [$ORIGIN]