search

locationtech / rasterframes

Geospatial Raster support for Spark DataFrames
http://rasterframes.io
Apache License 2.0
240 stars 46 forks source link

How to resolve "CPLE_OpenFailed(4) "Open failed.""? #512

Closed JenniferYingyiWu2020 closed 3 years ago

JenniferYingyiWu2020 commented 3 years ago

Dear All, I have written "unsupervised machine learning.py" using the codes from the website "https://rasterframes.io/unsupervised-learning.html". However, I encountered the errors when I run the above codes. The error stack is below. First and foremost, it says "FAILURE(3) CPLE_HttpResponse(11) "HTTP response." HTTP response code: 302 - Failed to connect to raw.githubusercontent.com port 443: Connection refused [2 of 1000] FAILURE(3) CPLE_OpenFailed(4) "Open failed." `/vsicurl/https://github.com/locationtech/rasterframes/raw/develop/core/src/test/resources/L8-B1-Elkton-VA.tiff' not recognized as a supported file format. "

I am so confused for the above messages, because when I change the image set from github to local images, the errros are similar: "FAILURE(3) CPLE_OpenFailed(4) "Open failed." /home/jenniferwu/Raster_Data_Set/samples/L8-B1-Elkton-VA.tiff: No such file or directory [2 of 1000] FAILURE(3) CPLE_OpenFailed(4) "Open failed." /home/jenniferwu/Raster_Data_Set/samples/L8-B1-Elkton-VA.tiff: No such file or directory". 16

[Stage 1:======================================================>(199 + 1) / 200][1 of 1000] FAILURE(3) CPLE_HttpResponse(11) "HTTP response." HTTP response code: 302 - Failed to connect to raw.githubusercontent.com port 443: Connection refused [2 of 1000] FAILURE(3) CPLE_OpenFailed(4) "Open failed." `/vsicurl/https://github.com/locationtech/rasterframes/raw/develop/core/src/test/resources/L8-B1-Elkton-VA.tiff' not recognized as a supported file format. 20/10/15 01:56:01 WARN BlockManager: Putting block rdd_23_37 failed due to exception java.lang.IllegalArgumentException: Error fetching data for one of: GDALRasterSource(https://github.com/locationtech/rasterframes/raw/develop/core/src/test/resources/L8-B1-Elkton-VA.tiff), GDALRasterSource(https://github.com/locationtech/rasterframes/raw/develop/core/src/test/resources/L8-B2-Elkton-VA.tiff), GDALRasterSource(https://github.com/locationtech/rasterframes/raw/develop/core/src/test/resources/L8-B3-Elkton-VA.tiff), GDALRasterSource(https://github.com/locationtech/rasterframes/raw/develop/core/src/test/resources/L8-B4-Elkton-VA.tiff), GDALRasterSource(https://github.com/locationtech/rasterframes/raw/develop/core/src/test/resources/L8-B5-Elkton-VA.tiff), GDALRasterSource(https://github.com/locationtech/rasterframes/raw/develop/core/src/test/resources/L8-B6-Elkton-VA.tiff), GDALRasterSource(https://github.com/locationtech/rasterframes/raw/develop/core/src/test/resources/L8-B7-Elkton-VA.tiff). 20/10/15 01:56:01 WARN BlockManager: Block rdd_23_37 could not be removed as it was not found on disk or in memory 20/10/15 01:56:01 ERROR Executor: Exception in task 37.0 in stage 1.0 (TID 207) java.lang.IllegalArgumentException: Error fetching data for one of: GDALRasterSource(https://github.com/locationtech/rasterframes/raw/develop/core/src/test/resources/L8-B1-Elkton-VA.tiff), GDALRasterSource(https://github.com/locationtech/rasterframes/raw/develop/core/src/test/resources/L8-B2-Elkton-VA.tiff), GDALRasterSource(https://github.com/locationtech/rasterframes/raw/develop/core/src/test/resources/L8-B3-Elkton-VA.tiff), GDALRasterSource(https://github.com/locationtech/rasterframes/raw/develop/core/src/test/resources/L8-B4-Elkton-VA.tiff), GDALRasterSource(https://github.com/locationtech/rasterframes/raw/develop/core/src/test/resources/L8-B5-Elkton-VA.tiff), GDALRasterSource(https://github.com/locationtech/rasterframes/raw/develop/core/src/test/resources/L8-B6-Elkton-VA.tiff), GDALRasterSource(https://github.com/locationtech/rasterframes/raw/develop/core/src/test/resources/L8-B7-Elkton-VA.tiff) at org.locationtech.rasterframes.expressions.generators.RasterSourceToRasterRefs.eval(RasterSourceToRasterRefs.scala:81) at org.apache.spark.sql.execution.GenerateExec$$anonfun$1$$anonfun$3.apply(GenerateExec.scala:95) at org.apache.spark.sql.execution.GenerateExec$$anonfun$1$$anonfun$3.apply(GenerateExec.scala:92) at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:435) at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:441) at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:212) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409) at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440) at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:212) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source) at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409) at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221) at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:299) at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1165) at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1156) at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1091) at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1156) at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:882) at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:357) at org.apache.spark.rdd.RDD.iterator(RDD.scala:308) at org.apache.spark.rdd.ZippedPartitionsRDD2.compute(ZippedPartitionsRDD.scala:89) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346) at org.apache.spark.rdd.RDD.iterator(RDD.scala:310) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346) at org.apache.spark.rdd.RDD.iterator(RDD.scala:310) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at org.apache.spark.scheduler.Task.run(Task.scala:123) at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: geotrellis.raster.gdal.MalformedDataException: Unable to construct a RasterExtent from the Transformation given. GDAL Error Code: 4 at geotrellis.raster.gdal.GDALDataset$.rasterExtent$extension1(GDALDataset.scala:143) at geotrellis.raster.gdal.GDALRasterSource.gridExtent$lzycompute(GDALRasterSource.scala:93) at geotrellis.raster.gdal.GDALRasterSource.gridExtent(GDALRasterSource.scala:93) at geotrellis.raster.RasterMetadata$class.cols(RasterMetadata.scala:55) at geotrellis.raster.RasterSource.cols(RasterSource.scala:44) at org.locationtech.rasterframes.ref.SimpleRasterInfo$.apply(SimpleRasterInfo.scala:71) at org.locationtech.rasterframes.ref.GDALRasterSource$$anonfun$tiffInfo$1.apply(GDALRasterSource.scala:53) at org.locationtech.rasterframes.ref.GDALRasterSource$$anonfun$tiffInfo$1.apply(GDALRasterSource.scala:53) at scala.compat.java8.functionConverterImpls.AsJavaFunction.apply(FunctionConverters.scala:262) at com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$0(LocalCache.java:139) at com.github.benmanes.caffeine.cache.UnboundedLocalCache.lambda$computeIfAbsent$2(UnboundedLocalCache.java:238) at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660) at com.github.benmanes.caffeine.cache.UnboundedLocalCache.computeIfAbsent(UnboundedLocalCache.java:234) at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:108) at com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62) at com.github.blemale.scaffeine.Cache.get(Cache.scala:40) at org.locationtech.rasterframes.ref.SimpleRasterInfo$.apply(SimpleRasterInfo.scala:49) at org.locationtech.rasterframes.ref.GDALRasterSource.tiffInfo(GDALRasterSource.scala:53) at org.locationtech.rasterframes.ref.GDALRasterSource.extent(GDALRasterSource.scala:57) at org.locationtech.rasterframes.ref.RFRasterSource.rasterExtent(RFRasterSource.scala:71) at org.locationtech.rasterframes.expressions.generators.RasterSourceToRasterRefs$$anonfun$1.apply(RasterSourceToRasterRefs.scala:65) at org.locationtech.rasterframes.expressions.generators.RasterSourceToRasterRefs$$anonfun$1.apply(RasterSourceToRasterRefs.scala:63) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.mutable.ArraySeq.foreach(ArraySeq.scala:74) at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at scala.collection.AbstractTraversable.map(Traversable.scala:104) at org.locationtech.rasterframes.expressions.generators.RasterSourceToRasterRefs.eval(RasterSourceToRasterRefs.scala:63) ... 44 more

JenniferYingyiWu2020 commented 3 years ago

Moreover, the file does exist. 17

JenniferYingyiWu2020 commented 3 years ago

21

JenniferYingyiWu2020 commented 3 years ago

I have also run the code "unsupervised machine learning" from Jupyter, but when the code got the line "model = pipeline.fit(df)", the following errors display. Could you pls help to give me some suggestions? Thanks!

Py4JJavaError Traceback (most recent call last) in ----> 1 model = pipeline.fit(df)

/usr/local/spark/python/pyspark/ml/base.py in fit(self, dataset, params) 130 return self.copy(params)._fit(dataset) 131 else: --> 132 return self._fit(dataset) 133 else: 134 raise ValueError("Params must be either a param map or a list/tuple of param maps, "

/usr/local/spark/python/pyspark/ml/pipeline.py in _fit(self, dataset) 107 dataset = stage.transform(dataset) 108 else: # must be an Estimator --> 109 model = stage.fit(dataset) 110 transformers.append(model) 111 if i < indexOfLastEstimator:

/usr/local/spark/python/pyspark/ml/base.py in fit(self, dataset, params) 130 return self.copy(params)._fit(dataset) 131 else: --> 132 return self._fit(dataset) 133 else: 134 raise ValueError("Params must be either a param map or a list/tuple of param maps, "

/usr/local/spark/python/pyspark/ml/wrapper.py in _fit(self, dataset) 293 294 def _fit(self, dataset): --> 295 java_model = self._fit_java(dataset) 296 model = self._create_model(java_model) 297 return self._copyValues(model)

/usr/local/spark/python/pyspark/ml/wrapper.py in _fit_java(self, dataset) 290 """ 291 self._transfer_params_to_java() --> 292 return self._java_obj.fit(dataset._jdf) 293 294 def _fit(self, dataset):

/usr/local/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py in call(self, *args) 1255 answer = self.gateway_client.send_command(command) 1256 return_value = get_return_value( -> 1257 answer, self.gateway_client, self.target_id, self.name) 1258 1259 for temp_arg in temp_args:

/usr/local/spark/python/pyspark/sql/utils.py in deco(*a, kw) 61 def deco(*a, *kw): 62 try: ---> 63 return f(a, kw) 64 except py4j.protocol.Py4JJavaError as e: 65 s = e.java_exception.toString()

/usr/local/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name) 326 raise Py4JJavaError( 327 "An error occurred while calling {0}{1}{2}.\n". --> 328 format(target_id, ".", name), value) 329 else: 330 raise Py4JError(

Py4JJavaError: An error occurred while calling o100.fit. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 37 in stage 1.0 failed 1 times, most recent failure: Lost task 37.0 in stage 1.0 (TID 207, localhost, executor driver): org.apache.spark.SparkException: Failed to execute user defined function($anonfun$4: (structb1:double,b2:double,b3:double,b4:double,b5:double,b6:double,b7:double) => struct<type:tinyint,size:int,indices:array,values:array>) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source) at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409) at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221) at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:299) at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1165) at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1156) at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1091) at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1156) at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:882) at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:357) at org.apache.spark.rdd.RDD.iterator(RDD.scala:308) at org.apache.spark.rdd.ZippedPartitionsRDD2.compute(ZippedPartitionsRDD.scala:89) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346) at org.apache.spark.rdd.RDD.iterator(RDD.scala:310) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346) at org.apache.spark.rdd.RDD.iterator(RDD.scala:310) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at org.apache.spark.scheduler.Task.run(Task.scala:123) at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.spark.SparkException: Encountered NaN while assembling a row with handleInvalid = "error". Consider removing NaNs from dataset or using handleInvalid = "keep" or "skip". at org.apache.spark.ml.feature.VectorAssembler$$anonfun$assemble$1.apply(VectorAssembler.scala:258) at org.apache.spark.ml.feature.VectorAssembler$$anonfun$assemble$1.apply(VectorAssembler.scala:255) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35) at org.apache.spark.ml.feature.VectorAssembler$.assemble(VectorAssembler.scala:255) at org.apache.spark.ml.feature.VectorAssembler$$anonfun$4.apply(VectorAssembler.scala:144) at org.apache.spark.ml.feature.VectorAssembler$$anonfun$4.apply(VectorAssembler.scala:143) ... 29 more

22

23

vpipkt commented 3 years ago

In the Jupyter example, it does seem you are having some success reading the raster data. I think the most important part of the stack trace there is 

Caused by: org.apache.spark.SparkException: Encountered NaN while assembling a row with handleInvalid = "error". Consider
removing NaNs from dataset or using handleInvalid = "keep" or "skip".

Please check out the use of pyrasterframes.rf_types.NoDataFilter which is shown and explained in the Supervised ML page

The code shown in this doc page is built nightly in our CI and is passing as of now. There may be a version mismatch or other configuration issue. Rasterframes 0.9.0 works with Spark 2.4.x, we suggest pinning that version to 2.4.5

vpipkt commented 3 years ago

In the case of the local file, can you share the code you are using to read? You may need to try spark.read.raster('file:///home/....'). Please let us know what the result of that is.

Unfortunately there is a lot that can go wrong and diagnosing root cause of accessing the file over HTTP is very complex. You did the right thing trying to copy to local.

JenniferYingyiWu2020 commented 3 years ago

Hi, I have used "spark.read.raster('file:///home/....')" in my programs to read a GeoTiff. Firstly, I read a local geotiff image, then I read a geotiff image from "https://github.com/locationtech/rasterframes/raw/develop/core/src/test/resources/". The result is normal and successful. 1 2 3 4

JenniferYingyiWu2020 commented 3 years ago

Hi, However, I have to say the "CPLE_OpenFailed(4)" error still exist when the the program execute at the line: "model = pipeline.fit(df)". "# Fitting the pipeline actually executes exploding the tiles, assembling the features vectors, and fitting the K-means clustering model. model = pipeline.fit(df)"

    The image data set "L8-B{}-Elkton-VA.tiff" do exist and can be read using the above codes, but when it run to "pipeline.fit(df)", the following exception has been shown:

java.lang.IllegalArgumentException: Error fetching data for one of: GDALRasterSource(file:///home/jenniferwu/notebooks/samples/resources/L8-B1-Elkton-VA.tiff), GDALRasterSource(file:///home/jenniferwu/notebooks/samples/resources/L8-B2-Elkton-VA.tiff), GDALRasterSource(file:///home/jenniferwu/notebooks/samples/resources/L8-B3-Elkton-VA.tiff), GDALRasterSource(file:///home/jenniferwu/notebooks/samples/resources/L8-B4-Elkton-VA.tiff), GDALRasterSource(file:///home/jenniferwu/notebooks/samples/resources/L8-B5-Elkton-VA.tiff), GDALRasterSource(file:///home/jenniferwu/notebooks/samples/resources/L8-B6-Elkton-VA.tiff), GDALRasterSource(file:///home/jenniferwu/notebooks/samples/resources/L8-B7-Elkton-VA.tiff) at org.locationtech.rasterframes.expressions.generators.RasterSourceToRasterRefs.eval(RasterSourceToRasterRefs.scala:81) at org.apache.spark.sql.execution.GenerateExec$$anonfun$1$$anonfun$3.apply(GenerateExec.scala:95) at org.apache.spark.sql.execution.GenerateExec$$anonfun$1$$anonfun$3.apply(GenerateExec.scala:92)

Caused by: geotrellis.raster.gdal.MalformedDataException: Unable to construct a RasterExtent from the Transformation given. GDAL Error Code: 4 at geotrellis.raster.gdal.GDALDataset$.rasterExtent$extension1(GDALDataset.scala:143) at geotrellis.raster.gdal.GDALRasterSource.gridExtent$lzycompute(GDALRasterSource.scala:93) at geotrellis.raster.gdal.GDALRasterSource.gridExtent(GDALRasterSource.scala:93)

    So, could you pls help to give me some suggestions? Thanks!

1

3

4

gdal

By the way, my local GDAL version is: GDAL 2.4.4

root |-- b1_path: string (nullable = false) |-- b2_path: string (nullable = false) |-- b3_path: string (nullable = false) |-- b4_path: string (nullable = false) |-- b5_path: string (nullable = false) |-- b6_path: string (nullable = false) |-- b7_path: string (nullable = false) |-- b1: struct (nullable = true) | |-- tile_context: struct (nullable = true) | | |-- extent: struct (nullable = false) | | | |-- xmin: double (nullable = false) | | | |-- ymin: double (nullable = false) | | | |-- xmax: double (nullable = false) | | | |-- ymax: double (nullable = false) | | |-- crs: struct (nullable = false) | | | |-- crsProj4: string (nullable = false) | |-- tile: tile (nullable = false) |-- b2: struct (nullable = true) | |-- tile_context: struct (nullable = true) | | |-- extent: struct (nullable = false) | | | |-- xmin: double (nullable = false) | | | |-- ymin: double (nullable = false) | | | |-- xmax: double (nullable = false) | | | |-- ymax: double (nullable = false) | | |-- crs: struct (nullable = false) | | | |-- crsProj4: string (nullable = false) | |-- tile: tile (nullable = false) |-- b3: struct (nullable = true) | |-- tile_context: struct (nullable = true) | | |-- extent: struct (nullable = false) | | | |-- xmin: double (nullable = false) | | | |-- ymin: double (nullable = false) | | | |-- xmax: double (nullable = false) | | | |-- ymax: double (nullable = false) | | |-- crs: struct (nullable = false) | | | |-- crsProj4: string (nullable = false) | |-- tile: tile (nullable = false) |-- b4: struct (nullable = true) | |-- tile_context: struct (nullable = true) | | |-- extent: struct (nullable = false) | | | |-- xmin: double (nullable = false) | | | |-- ymin: double (nullable = false) | | | |-- xmax: double (nullable = false) | | | |-- ymax: double (nullable = false) | | |-- crs: struct (nullable = false) | | | |-- crsProj4: string (nullable = false) | |-- tile: tile (nullable = false) |-- b5: struct (nullable = true) | |-- tile_context: struct (nullable = true) | | |-- extent: struct (nullable = false) | | | |-- xmin: double (nullable = false) | | | |-- ymin: double (nullable = false) | | | |-- xmax: double (nullable = false) | | | |-- ymax: double (nullable = false) | | |-- crs: struct (nullable = false) | | | |-- crsProj4: string (nullable = false) | |-- tile: tile (nullable = false) |-- b6: struct (nullable = true) | |-- tile_context: struct (nullable = true) | | |-- extent: struct (nullable = false) | | | |-- xmin: double (nullable = false) | | | |-- ymin: double (nullable = false) | | | |-- xmax: double (nullable = false) | | | |-- ymax: double (nullable = false) | | |-- crs: struct (nullable = false) | | | |-- crsProj4: string (nullable = false) | |-- tile: tile (nullable = false) |-- b7: struct (nullable = true) | |-- tile_context: struct (nullable = true) | | |-- extent: struct (nullable = false) | | | |-- xmin: double (nullable = false) | | | |-- ymin: double (nullable = false) | | | |-- xmax: double (nullable = false) | | | |-- ymax: double (nullable = false) | | |-- crs: struct (nullable = false) | | | |-- crsProj4: string (nullable = false) | |-- tile: tile (nullable = false) |-- crs: struct (nullable = true) | |-- crsProj4: string (nullable = false) |-- extent: struct (nullable = true) | |-- xmin: double (nullable = false) | |-- ymin: double (nullable = false) | |-- xmax: double (nullable = false) | |-- ymax: double (nullable = false)

JenniferYingyiWu2020 commented 3 years ago

Hi, I used the local GeoTiff image set, and I encountered the error, which is caused by: org.apache.spark.SparkException: Encountered NaN while assembling a row with handleInvalid = "error". Consider removing NaNs from dataset or using handleInvalid = "keep" or "skip".

    So, what do you think about it? Whether the errors display because of the version of GDAL? What is the proper version of GDAL used in the pyrasterframes?

111

vpipkt commented 3 years ago

@JenniferYingyiWu2020

  1. spark.read.raster will always correctly print "lazy" operations like printSchema as long as the parameters passed in are syntactically correct. The lazy operations do not read any data or do any checks of existence of resources, etc.

  2. Did you check carefully the file permssions and path on the resource mentioned in this log line??

  3. Regarding org.apache.spark.SparkException: Encountered NaN while assembling a row: I commented earlier about the use of NoDataFilter, did you try that in your pipeline? .... The parameter handleInvalid is perhaps a reference to the VectorAssembler? If you have this in your pipeline, consider changing the option as described in the error.

Related, can you perhaps share a complete source file or files for these errors? The screenshots are somewhat helpful but are giving us limited context to understand what is causing the errors.

Can you confirm which version of Spark you are using?

GDAL version 2.4.4 is compatible with RasterFrames 0.9.0.

JenniferYingyiWu2020 commented 3 years ago

Hi, The permission and path of the above local file is OK, and the error in this log line has been resolved. Moreover, regarding "org.apache.spark.SparkException: Encountered NaN while assembling a row", I skipped using "NoDataFilter" in pipeline, I have used "handleInvalid" in VectorAssembler instead. So, now the "unsupervised machine learning.py" can be executed normally and successfully! Moreover, my local GDAL version is "3.0.2", and pyrasterframes version is "0.9.0". Because the python program of "unsupervised machine learning" was fine and no error several months ago, "org.apache.spark.SparkException: Encountered NaN while assembling a row" just happened last week. Although I have resolved the above errors, I am just worried why this error appeared these days, and was rare before?

JenniferYingyiWu2020 commented 3 years ago

Furthermore, my local Spark version is "2.4.6".

3

vpipkt commented 3 years ago

@JenniferYingyiWu2020 currently RasterFrames 0.9.x series is compatible with GDAL 2.4.4 but we have an open issue to upgrade to GDAL 3.x.

I am not sure why the error had not been happening before but is happening now. It seems plausible that the underlying GDAL version may have changed with some unexpected results.

@JenniferYingyiWu2020 Do you think this issue can be considered resolved and closed?

JenniferYingyiWu2020 commented 3 years ago

Hi @vpipkt, Thank you for your help! I consider the issue have been resolved and can be closed now.