search

locationtech / rasterframes

Geospatial Raster support for Spark DataFrames
http://rasterframes.io
Apache License 2.0
240 stars 46 forks source link

How to reslove "java.lang.IllegalArgumentException: Error fetching data for one of: HadoopGeoTiffRasterSource(hdfs://192.168.101.201:9000/.../split_20200613clip/SCL.tif), ...)" #552

Closed JenniferYingyiWu2020 closed 3 years ago

JenniferYingyiWu2020 commented 3 years ago

1 2 3 4 5 6 7 8 10

So, could you pls help to give me some suggestions on how to resolve the above errors? Thanks!
JenniferYingyiWu2020 commented 3 years ago 
Moreover, I have tried 3 ways to install GDAL under three different conda environments. The strategy is just for testing. However, the "java.lang.IllegalArgumentException: Error fetching data for one of: HadoopGeoTiffRasterSource" has always been encoutered.

So, could you pls give me some suggestions? Thanks! The first method is installing the latest GDAL using conda: 1 2 3 4 5 6 Secondly, I change to install GDAL from source code: 21 22 23 24 25 Lastly, under my local hadoop user environment, pyrasterframes has been installed using "pip" and GDAL has been installed using "conda": 31 32 33 34 35 36

lepitrust commented 3 years ago

anyone resolve this issue? i have the same... The only one things that i can't made is recompile because i trying to configure it on docker env

JenniferYingyiWu2020 commented 3 years ago

Hi All, I have resolved the issue by myself. So, after that, I will close the issue. Thanks!

lepitrust commented 3 years ago

Hi All, I have resolved the issue by myself. So, after that, I will close the issue. Thanks!

Can you tell me how do you resolve? Thanks

EDIT: I understood why it didn't work. On every node with spark and hadoop must configure core-site.xml. Set the property fs.default.name and fs.defaultFS to your hdfs:// address (for me hdfs://namenode:8020/)

JenniferYingyiWu2020 commented 3 years ago

Yes, it is exactly that.