GDAL not available

[yang162132]

When I try gdal_version,it seem not available.It's my code:

from osgeo import gdal
from pyrasterframes.utils import create_rf_spark_session,gdal_version
spark = create_rf_spark_session()

print("gdal:" + gdal.__version__)
print("---------------")
print("resterframes:" + gdal_version())

and the result is:

gdal:3.4.1
---------------
resterframes:not available

[pomadchin]

Hey @yang162132!

GDAL Support is exposed via JNI bindings. Most likely LD_LIBRARY_PATH is not set, this env variable should be set both on a driver node and on all executors pointing to a proper GDAL location.

It can be a Spark Session property config (spark.yarn.appMasterEnv.LD_LIBRARY_PATH, spark.executorEnv.LD_LIBRARY_PATH; see example and docs) as well as the ENV variable (Dokerfile setting example).

[yang162132]

But it's seem not useful

spark = create_rf_spark_session("local[*]",**{"spark.yarn.appMasterEnv.LD_LIBRARY_PATH": "/root/anaconda3/lib/:/usr/local/lib/:/usr/lib/:/usr/local/gdal",
                                 "spark.executorEnv.LD_LIBRARY_PATH": "/root/anaconda3/lib/:/usr/local/lib/:/usr/lib/:/usr/local/gdal/"})

[yang162132]

and my gdal is in LD_LIBRARY_PATH

[pomadchin]

@yang162132 oh, I think that's due to GDAL versions mismatch; since GDAL 3.x it is sensitive to gdal minor versions upgrades;

I suggest just to symlink the new .so to match the old one, should be smth like: ln -s /usr/local/lib/libgdal.28.dylib /usr/local/lib/libgdal.27.dylib (/usr/local/lib/libgdal.27.dylib is the expected filename)

[yang162132]

@yang162132哦，我想那是因为 GDAL 版本不匹配；从 GDAL 3.x 开始，它对 gdal 次要版本升级很敏感；

我建议只对新的 .so 进行符号链接以匹配旧的，应该像这样：（ln -s /usr/local/lib/libgdal.28.dylib /usr/local/lib/libgdal.27.dylib是/usr/local/lib/libgdal.27.dylib预期的文件名）

Oh, It's uesful!

[shendannan]

I have same question, with windows operating system. My code is same as yang162132, and the result is:

gdal:3.4.3
---------------
rasterframes:not available

since I use windows OS, how can I config Spark Session property?

[pomadchin]

@shendannan that's the bindings repo we're using: https://github.com/geotrellis/gdal-warp-bindings

Its GDAL 3.0.4 compatible only (mingw, windows), but you may try to symlink it as well.

[yang162132]

@shendannan I suggest you install a linux virtual machine for linux , it is really inconvenient to use GDAL on windows