search

locationtech / rasterframes

Geospatial Raster support for Spark DataFrames
http://rasterframes.io
Apache License 2.0
243 stars 46 forks source link

Unable to load local/remote rasters or catalogs in 0.8.4 #445

Closed SarvaPulla closed 3 years ago

SarvaPulla commented 4 years ago

RasterFrames 0.8.4 is unable to load rasters or a catalog after initialization. Version 0.8.2 is unable to find gdal, but the following code initializes a session and is able to load in rasters/catalogs just fine.

import pyrasterframes
spark_config = {'spark.driver.memory':'25G',
                'spark.executor.memory': '32G', #can be 32G
                'spark.driver.maxResultSize': '24G',
                'spark.executor.cores': '8',
                'spark.executor.instances' : '25',
                'spark.default.parallelism' : '500', # increase the partition number at cluster initialization
                'spark.dynamicAllocation.enabled' : True}

spark = pyrasterframes.get_spark_session(master='yarn', **spark_config)
gt_cat = spark.read.format('geotrellis-catalog').load('/mnt/catalog/').toPandas()

The following code does initialize a session with 0.8.4. But it's not able to load rasters or catalogs.

import pyrasterframes
from pyrasterframes.utils import create_rf_spark_session
spark_config = {'spark.driver.memory':'25G',
                'spark.executor.memory': '32G',
                'spark.driver.maxResultSize': '24G',
                'spark.executor.cores': '8',
                'spark.executor.instances' : '25',
                'spark.default.parallelism' : '500',
                'spark.dynamicAllocation.enabled' : True}
spark = create_rf_spark_session(master='yarn', **spark_config)
rf = spark.read.raster('https://s22s-test-geotiffs.s3.amazonaws.com/luray_snp/B02.tif')

Here's the build info: {'scalaVersion': '2.11.12', 'builtAtMillis': '1573482950714', 'sbtVersion': '1.2.8', 'name': 'core', 'rfSparkVersion': '2.4.4', 'moduleName': 'rasterframes', 'rfGeoMesaVersion': '2.2.1', 'GDAL': 'not available', 'rfGeoTrellisVersion': '2.3.3', 'version': '0.8.4', 'builtAtString': '2019-11-11 14:35:50.714'}

Full Traceback from reading the raster. I have even saved the raster locally and it's throwing the same error.

Py4JJavaError                             Traceback (most recent call last)
<ipython-input-5-84a92af0b321> in <module>
----> 1 rf = spark.read.raster('https://s22s-test-geotiffs.s3.amazonaws.com/luray_snp/B02.tif')

/opt/miniconda/envs/py3/lib/python3.6/site-packages/pyrasterframes/__init__.py in _raster_reader(df_reader, source, catalog_col_names, band_indexes, tile_dimensions, lazy_tiles, **options)
    221     return df_reader \
    222         .format("raster") \
--> 223         .load(path, **options)
    224 
    225 

/usr/lib/spark/python/pyspark/sql/readwriter.py in load(self, path, format, schema, **options)
    164         self.options(**options)
    165         if isinstance(path, basestring):
--> 166             return self._df(self._jreader.load(path))
    167         elif path is not None:
    168             if type(path) != list:

/usr/lib/spark/python/lib/py4j-src.zip/py4j/java_gateway.py in __call__(self, *args)
   1255         answer = self.gateway_client.send_command(command)
   1256         return_value = get_return_value(
-> 1257             answer, self.gateway_client, self.target_id, self.name)
   1258 
   1259         for temp_arg in temp_args:

/usr/lib/spark/python/pyspark/sql/utils.py in deco(*a, **kw)
     61     def deco(*a, **kw):
     62         try:
---> 63             return f(*a, **kw)
     64         except py4j.protocol.Py4JJavaError as e:
     65             s = e.java_exception.toString()

/usr/lib/spark/python/lib/py4j-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
    326                 raise Py4JJavaError(
    327                     "An error occurred while calling {0}{1}{2}.\n".
--> 328                     format(target_id, ".", name), value)
    329             else:
    330                 raise Py4JError(

Py4JJavaError: An error occurred while calling o175.load.
: java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformUp(Lscala/PartialFunction;)Lorg/apache/spark/sql/catalyst/plans/logical/LogicalPlan;
    at org.locationtech.rasterframes.rules.SpatialFilterPushdownRules$.apply(SpatialFilterPushdownRules.scala:35)
    at org.locationtech.rasterframes.rules.SpatialFilterPushdownRules$.apply(SpatialFilterPushdownRules.scala:33)
    at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:87)
    at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:84)
    at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:124)
    at scala.collection.immutable.List.foldLeft(List.scala:84)
    at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:84)
    at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:76)
    at scala.collection.immutable.List.foreach(List.scala:381)
    at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:76)
    at org.apache.spark.sql.execution.QueryExecution.optimizedPlan$lzycompute(QueryExecution.scala:66)
    at org.apache.spark.sql.execution.QueryExecution.optimizedPlan(QueryExecution.scala:66)
    at org.apache.spark.sql.execution.QueryExecution.sparkPlan$lzycompute(QueryExecution.scala:72)
    at org.apache.spark.sql.execution.QueryExecution.sparkPlan(QueryExecution.scala:68)
    at org.apache.spark.sql.execution.QueryExecution.executedPlan$lzycompute(QueryExecution.scala:77)
    at org.apache.spark.sql.execution.QueryExecution.executedPlan(QueryExecution.scala:77)
    at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3249)
    at org.apache.spark.sql.Dataset.head(Dataset.scala:2484)
    at org.apache.spark.sql.Dataset.take(Dataset.scala:2698)
    at org.apache.spark.sql.DataFrameReader.csv(DataFrameReader.scala:495)
    at org.locationtech.rasterframes.datasource.raster.RasterSourceDataSource$RasterSourceCatalog.registerAsTable(RasterSourceDataSource.scala:80)
    at org.locationtech.rasterframes.datasource.raster.RasterSourceDataSource$$anonfun$2.apply(RasterSourceDataSource.scala:43)
    at org.locationtech.rasterframes.datasource.raster.RasterSourceDataSource$$anonfun$2.apply(RasterSourceDataSource.scala:43)
    at scala.util.Either.fold(Either.scala:98)
    at org.locationtech.rasterframes.datasource.raster.RasterSourceDataSource.createRelation(RasterSourceDataSource.scala:43)
    at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:340)
    at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:239)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:227)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:174)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.lang.Thread.run(Thread.java:745)

The conda environment that I am using for reference

# packages in environment at /opt/miniconda/envs/py3:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
affine                    2.3.0                      py_0    conda-forge
aiohttp                   3.6.2            py36h516909a_0    conda-forge
arrow-cpp                 0.15.1           py36h982ac2c_5    conda-forge
async-timeout             3.0.1                   py_1000    conda-forge
attrs                     19.3.0                     py_0    conda-forge
awscli                    1.16.313                 py36_0    conda-forge
backcall                  0.1.0                      py_0    conda-forge
bleach                    3.1.0                      py_0    conda-forge
boost-cpp                 1.70.0               ha2d47e9_1    conda-forge
boto3                     1.10.49                    py_0    conda-forge
botocore                  1.13.49                    py_0    conda-forge
brotli                    1.0.7             he1b5a44_1000    conda-forge
bzip2                     1.0.8                h516909a_2    conda-forge
c-ares                    1.15.0            h516909a_1001    conda-forge
ca-certificates           2019.11.28           hecc5488_0    conda-forge
cachetools                3.1.1                      py_0    conda-forge
cairo                     1.16.0            h18b612c_1001    conda-forge
certifi                   2019.11.28               py36_0    conda-forge
cffi                      1.13.2           py36h8022711_0    conda-forge
cfitsio                   3.470                hb60a0a2_2    conda-forge
chardet                   3.0.4                 py36_1003    conda-forge
click                     7.0                        py_0    conda-forge
click-plugins             1.1.1                      py_0    conda-forge
cligj                     0.5.0                      py_0    conda-forge
colorama                  0.4.1                      py_0    conda-forge
cryptography              2.8              py36h72c5cf5_1    conda-forge
curl                      7.65.3               hf8cf82a_0    conda-forge
cycler                    0.10.0                     py_2    conda-forge
dbus                      1.13.6               he372182_0    conda-forge
decorator                 4.4.1                      py_0    conda-forge
defusedxml                0.6.0                      py_0    conda-forge
deprecated                1.2.7                      py_0    conda-forge
descartes                 1.1.0                      py_4    conda-forge
df2gspread                1.0.4                 py36_1000    conda-forge
docopt                    0.6.2                      py_1    conda-forge
docutils                  0.15.2                   py36_0    conda-forge
double-conversion         3.1.5                he1b5a44_2    conda-forge
entrypoints               0.3                   py36_1000    conda-forge
expat                     2.2.5             he1b5a44_1004    conda-forge
fastavro                  0.22.9           py36h516909a_0    conda-forge
fiona                     1.8.6            py36hf242f0b_3    conda-forge
fontconfig                2.13.1            he4413a7_1000    conda-forge
fpdf                      1.7.2                    pypi_0    pypi
freetype                  2.10.0               he983fc9_1    conda-forge
freexl                    1.0.5             h14c3975_1002    conda-forge
fsspec                    0.6.2                      py_0    conda-forge
gdal                      2.4.1           py36h5f563d9_10    conda-forge
geographiclib             1.50                       py_0    conda-forge
geohash2                  1.1                        py_0    conda-forge
geojson                   2.5.0                      py_0    conda-forge
geopandas                 0.6.2                      py_0    conda-forge
geopy                     1.20.0                     py_0    conda-forge
geos                      3.7.2                he1b5a44_2    conda-forge
geotiff                   1.4.3             hb6868eb_1001    conda-forge
gettext                   0.19.8.1          hc5be6a0_1002    conda-forge
gflags                    2.2.2             he1b5a44_1002    conda-forge
giflib                    5.1.7                h516909a_1    conda-forge
gitdb2                    2.0.6                      py_0    conda-forge
gitpython                 3.0.5                      py_0    conda-forge
glib                      2.58.3          py36h6f030ca_1002    conda-forge
glog                      0.4.0                he1b5a44_1    conda-forge
google-api-python-client  1.7.11                     py_0    conda-forge
google-auth               1.10.0                     py_0    conda-forge
google-auth-httplib2      0.0.3                      py_2    conda-forge
grpc-cpp                  1.25.0               h213be95_2    conda-forge
gspread                   3.1.0                      py_0    conda-forge
gst-plugins-base          1.14.5               h0935bb2_0    conda-forge
gstreamer                 1.14.5               h36ae1b5_0    conda-forge
hdf4                      4.2.13            hf30be14_1003    conda-forge
hdf5                      1.10.4          nompi_h3c11f04_1106    conda-forge
httplib2                  0.15.0                   py36_0    conda-forge
icu                       58.2              hf484d3e_1000    conda-forge
idna                      2.8                   py36_1000    conda-forge
idna_ssl                  1.1.0                 py36_1000    conda-forge
importlib_metadata        1.3.0                    py36_0    conda-forge
ipykernel                 5.1.3            py36h5ca1d4c_0    conda-forge
ipython                   7.11.1           py36h5ca1d4c_0    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
jedi                      0.15.2                   py36_0    conda-forge
jinja2                    2.10.3                     py_0    conda-forge
jmespath                  0.9.4                      py_0    conda-forge
joblib                    0.14.1                     py_0    conda-forge
jpeg                      9c                h14c3975_1001    conda-forge
json-c                    0.13.1            h14c3975_1001    conda-forge
jsonschema                3.2.0                    py36_0    conda-forge
jupyter_client            5.3.4                    py36_0    conda-forge
jupyter_core              4.6.1                    py36_0    conda-forge
kealib                    1.4.10            h1978553_1003    conda-forge
kiwisolver                1.1.0            py36hc9558a2_0    conda-forge
krb5                      1.16.4               h2fd8d38_0    conda-forge
libblas                   3.8.0               14_openblas    conda-forge
libcblas                  3.8.0               14_openblas    conda-forge
libcurl                   7.65.3               hda55be3_0    conda-forge
libdap4                   3.20.4               hd3bb157_0    conda-forge
libedit                   3.1.20170329      hf8c457e_1001    conda-forge
libevent                  2.1.10               h72c5cf5_0    conda-forge
libffi                    3.2.1             he1b5a44_1006    conda-forge
libgcc-ng                 9.2.0                h24d8f2e_1    conda-forge
libgdal                   2.4.1               hc4f5fd6_10    conda-forge
libgfortran-ng            7.3.0                hdf63c60_3    conda-forge
libgomp                   9.2.0                h24d8f2e_1    conda-forge
libiconv                  1.15              h516909a_1005    conda-forge
libkml                    1.3.0             h4fcabce_1010    conda-forge
liblapack                 3.8.0               14_openblas    conda-forge
libnetcdf                 4.6.2             hbdf4f91_1001    conda-forge
libopenblas               0.3.7                h5ec1e0e_6    conda-forge
libpng                    1.6.37               hed695b0_0    conda-forge
libpq                     11.5                 hd9ab2ff_1    conda-forge
libprotobuf               3.11.0               h8b12597_0    conda-forge
libsodium                 1.0.17               h516909a_0    conda-forge
libspatialindex           1.9.3                he1b5a44_1    conda-forge
libspatialite             4.3.0a            h79dc798_1030    conda-forge
libssh2                   1.8.2                h22169c7_2    conda-forge
libstdcxx-ng              9.2.0                hdf63c60_1    conda-forge
libtiff                   4.1.0                hc3755c2_3    conda-forge
libuuid                   2.32.1            h14c3975_1000    conda-forge
libxcb                    1.13              h14c3975_1002    conda-forge
libxml2                   2.9.9                h13577e0_2    conda-forge
lz4-c                     1.8.3             he1b5a44_1001    conda-forge
markupsafe                1.1.1            py36h516909a_0    conda-forge
matplotlib                3.0.3                    py36_1    conda-forge
matplotlib-base           3.0.3            py36h5f35d83_1    conda-forge
mistune                   0.8.4           py36h516909a_1000    conda-forge
more-itertools            8.0.2                      py_0    conda-forge
multidict                 4.7.3            py36h516909a_0    conda-forge
munch                     2.5.0                      py_0    conda-forge
nb_conda                  2.2.1                    py36_2    conda-forge
nb_conda_kernels          2.2.2                    py36_0    conda-forge
nbconvert                 5.6.1                    py36_0    conda-forge
nbformat                  5.0.3                      py_0    conda-forge
ncurses                   6.1               hf484d3e_1002    conda-forge
notebook                  6.0.1                    py36_0    conda-forge
numpy                     1.17.3           py36h95a1406_0    conda-forge
oauth2client              4.1.3                      py_0    conda-forge
olefile                   0.46                       py_0    conda-forge
openjpeg                  2.3.1                h981e76c_3    conda-forge
openmp_impl               4.5                       0_gnu    conda-forge
openssl                   1.1.1d               h516909a_0    conda-forge
palettable                3.3.0                      py_0    conda-forge
pandas                    0.25.3           py36hb3f55d8_0    conda-forge
pandoc                    2.9.1.1                       0    conda-forge
pandocfilters             1.4.2                      py_1    conda-forge
parquet-cpp               1.5.1                         2    conda-forge
parso                     0.5.2                      py_0    conda-forge
patsy                     0.5.1                      py_0    conda-forge
pcre                      8.43                 he1b5a44_0    conda-forge
pdf2image                 1.11.0                   pypi_0    pypi
pexpect                   4.7.0                    py36_0    conda-forge
pickleshare               0.7.5                 py36_1000    conda-forge
pillow                    7.0.0            py36hefe7db6_0    conda-forge
pip                       19.3.1                   py36_0    conda-forge
pixman                    0.38.0            h516909a_1003    conda-forge
poppler                   0.67.0               h14e79db_8    conda-forge
poppler-data              0.4.9                         1    conda-forge
postgresql                11.5                 hc63931a_1    conda-forge
proj4                     5.2.0             he1b5a44_1006    conda-forge
prometheus_client         0.7.1                      py_0    conda-forge
prompt_toolkit            3.0.2                      py_0    conda-forge
psycopg2                  2.8.4            py36h72c5cf5_0    conda-forge
pthread-stubs             0.4               h14c3975_1001    conda-forge
ptyprocess                0.6.0                   py_1001    conda-forge
py4j                      0.10.7                     py_1    conda-forge
pyarrow                   0.15.1           py36h8b68381_1    conda-forge
pyasn1                    0.4.8                      py_0    conda-forge
pyasn1-modules            0.2.7                      py_0    conda-forge
pycairo                   1.18.2           py36h438ddbb_0    conda-forge
pycparser                 2.19                     py36_1    conda-forge
pycrypto                  2.6.1           py36h516909a_1003    conda-forge
pygeotools                0.6.0                    pypi_0    pypi
pygments                  2.5.2                      py_0    conda-forge
pyopenssl                 19.1.0                   py36_0    conda-forge
pyparsing                 2.4.6                      py_0    conda-forge
pyproj                    1.9.6           py36h516909a_1002    conda-forge
pyqt                      5.9.2            py36hcca6a23_4    conda-forge
pyrasterframes            0.8.4                    pypi_0    pypi
pyrsistent                0.15.7           py36h516909a_0    conda-forge
pysal                     2.1.0                      py_0    conda-forge
pyshp                     2.1.0                      py_0    conda-forge
pysocks                   1.7.1                    py36_0    conda-forge
pyspark                   2.4.4                      py_0    conda-forge
python                    3.6.7             h357f687_1006    conda-forge
python-dateutil           2.8.1                      py_0    conda-forge
python-domino             0.3.5                    pypi_0    pypi
pytz                      2019.3                     py_0    conda-forge
pyyaml                    5.2              py36h516909a_0    conda-forge
pyzmq                     18.1.1           py36h1768529_0    conda-forge
qt                        5.9.7                h52cfd70_2    conda-forge
rasterio                  1.0.25           py36hdff7cfa_0    conda-forge
re2                       2020.01.01           he1b5a44_0    conda-forge
readline                  8.0                  hf8c457e_0    conda-forge
reportlab                 3.5.32           py36h7d98c4e_0    conda-forge
requests                  2.22.0                   py36_1    conda-forge
rsa                       3.4.2                      py_1    conda-forge
rtree                     0.9.3            py36h7b0cdae_0    conda-forge
s3fs                      0.4.0                      py_0    conda-forge
s3transfer                0.2.1                    py36_0    conda-forge
scikit-learn              0.22.1           py36hcdab131_1    conda-forge
scipy                     1.4.1            py36h921218d_0    conda-forge
seaborn                   0.9.0                      py_2    conda-forge
send2trash                1.5.0                      py_0    conda-forge
setuptools                44.0.0                   py36_0    conda-forge
shapely                   1.6.4           py36hec07ddf_1006    conda-forge
simplejson                3.17.0           py36h516909a_0    conda-forge
sip                       4.19.8          py36hf484d3e_1000    conda-forge
six                       1.13.0                   py36_0    conda-forge
smmap2                    2.0.5                      py_0    conda-forge
snappy                    1.1.7             he1b5a44_1003    conda-forge
snuggs                    1.4.7                      py_0    conda-forge
sqlite                    3.30.1               hcee41ef_0    conda-forge
statsmodels               0.10.2           py36hc1659b7_0    conda-forge
terminado                 0.8.3                    py36_0    conda-forge
testpath                  0.4.4                      py_0    conda-forge
thrift-cpp                0.12.0            hf3afdfd_1004    conda-forge
tk                        8.6.10               hed695b0_0    conda-forge
tornado                   6.0.3            py36h516909a_0    conda-forge
tqdm                      4.41.1                     py_0    conda-forge
traitlets                 4.3.3                    py36_0    conda-forge
typing_extensions         3.7.4.1                  py36_0    conda-forge
tzcode                    2019a             h516909a_1002    conda-forge
ujson                     1.35            py36h516909a_1001    conda-forge
uriparser                 0.9.3                he1b5a44_1    conda-forge
uritemplate               3.0.1                      py_0    conda-forge
urllib3                   1.25.7                   py36_0    conda-forge
utm                       0.5.0                      py_0    conda-forge
wcwidth                   0.1.8                      py_0    conda-forge
webencodings              0.5.1                      py_1    conda-forge
wheel                     0.33.6                   py36_0    conda-forge
wrapt                     1.11.2           py36h516909a_0    conda-forge
xarray                    0.14.1                     py_1    conda-forge
xerces-c                  3.2.2             hea5cb30_1003    conda-forge
xorg-kbproto              1.0.7             h14c3975_1002    conda-forge
xorg-libice               1.0.10               h516909a_0    conda-forge
xorg-libsm                1.2.3             h84519dc_1000    conda-forge
xorg-libx11               1.6.9                h516909a_0    conda-forge
xorg-libxau               1.0.9                h14c3975_0    conda-forge
xorg-libxdmcp             1.1.3                h516909a_0    conda-forge
xorg-libxext              1.3.4                h516909a_0    conda-forge
xorg-libxrender           0.9.10            h516909a_1002    conda-forge
xorg-renderproto          0.11.1            h14c3975_1002    conda-forge
xorg-xextproto            7.3.0             h14c3975_1002    conda-forge
xorg-xproto               7.0.31            h14c3975_1007    conda-forge
xz                        5.2.4             h14c3975_1001    conda-forge
yaml                      0.2.2                h516909a_1    conda-forge
yarl                      1.3.0           py36h516909a_1000    conda-forge
zeromq                    4.3.2                he1b5a44_2    conda-forge
zipp                      0.6.0                      py_0    conda-forge
zlib                      1.2.11            h516909a_1006    conda-forge
zstd                      1.4.4                h3b9ef0a_1    conda-forge
vpipkt commented 4 years ago

https://gitter.im/locationtech/rasterframes?at=5e173223810a1042cb8f59c2

metasim commented 3 years ago

Closing due to dormancy.