search

biggis-project / biggis-landuse

Land use update detection
http://biggis-project.eu/biggis-docs/demos/landuse/
11 stars 3 forks source link

ERROR org.apache.spark.executor.Executor ... scala.MatchError: Some() at geotrellis.raster.io.geotiff.reader.GeoTiffCSParser.getEllipsoidInfo #2

Closed aklink closed 7 years ago

aklink commented 8 years ago

When I try:

java -cp target/biggis-landuse-0.0.1-SNAPSHOT.jar \ biggis.landuse.spark.examples.GeotiffToPyramid \ ./data/DOP_RGBI_T2.tif \ new_layer \ ./data/pyramid

I get the following error message:

15:58:50 ERROR org.apache.spark.executor.Executor - Exception in task 0.0 in stage 0.0 (TID 0) scala.MatchError: Some() (of class scala.Some) at geotrellis.raster.io.geotiff.reader.GeoTiffCSParser.getEllipsoidInfo(GeoTiffCSParser.scala:570) at geotrellis.raster.io.geotiff.reader.GeoTiffCSParser.createGeoTiffCSParameters(GeoTiffCSParser.scala:172) at geotrellis.raster.io.geotiff.reader.GeoTiffCSParser.geoTiffCSParameters$lzycompute(GeoTiffCSParser.scala:78) at geotrellis.raster.io.geotiff.reader.GeoTiffCSParser.geoTiffCSParameters(GeoTiffCSParser.scala:78) at geotrellis.raster.io.geotiff.reader.GeoTiffCSParser.model(GeoTiffCSParser.scala:80) at geotrellis.raster.io.geotiff.tags.TiffTags.crs$lzycompute(TiffTags.scala:207) at geotrellis.raster.io.geotiff.tags.TiffTags.crs(TiffTags.scala:205) at geotrellis.raster.io.geotiff.reader.GeoTiffReader$.readGeoTiffInfo(GeoTiffReader.scala:315) at geotrellis.raster.io.geotiff.reader.GeoTiffReader$.readSingleband(GeoTiffReader.scala:67) at geotrellis.raster.io.geotiff.reader.GeoTiffReader$.readSingleband(GeoTiffReader.scala:61) at geotrellis.raster.io.geotiff.SinglebandGeoTiff$.apply(SinglebandGeoTiff.scala:40) at geotrellis.spark.io.hadoop.formats.GeotiffInputFormat.read(GeotiffInputFormat.scala:28) at geotrellis.spark.io.hadoop.formats.BinaryFileInputFormat$$anonfun$createRecordReader$1.apply(BinaryFileInputFormat.scala:34) at geotrellis.spark.io.hadoop.formats.BinaryFileInputFormat$$anonfun$createRecordReader$1.apply(BinaryFileInputFormat.scala:34) at geotrellis.spark.io.hadoop.formats.BinaryFileRecordReader.initialize(BinaryFileInputFormat.scala:18) at org.apache.spark.rdd.NewHadoopRDD$$anon$1.(NewHadoopRDD.scala:158) at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:129) at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:64) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) at org.apache.spark.scheduler.Task.run(Task.scala:89) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

aklink commented 8 years ago

Issue is specific to used image. Trying a landsat image (created with ERDAS Imagine) is ok (no issue).

Trying a different aerila image leads to different error:

15:05:03 ERROR org.apache.spark.executor.Executor - Exception in task 0.0 in stage 3.0 (TID 3) geotrellis.raster.GeoAttrsError: invalid cols: 0 at geotrellis.raster.RasterExtent.(RasterExtent.scala:72) at geotrellis.raster.GridDefinition$class.toRasterExtent(GridDefinition.scala:36) at geotrellis.raster.GridExtent.toRasterExtent(GridExtent.scala:9) at geotrellis.raster.reproject.ReprojectRasterExtent$.apply(ReprojectRasterExtent.scala:105) at geotrellis.spark.reproject.TileRDDReproject$$anonfun$6$$anonfun$apply$2.apply(TileRDDReproject.scala:185) at geotrellis.spark.reproject.TileRDDReproject$$anonfun$6$$anonfun$apply$2.apply(TileRDDReproject.scala:182) at scala.collection.Iterator$$anon$11.next(Iterator.scala:409) at scala.collection.Iterator$$anon$11.next(Iterator.scala:409) at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434) at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440) at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:126) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) at org.apache.spark.scheduler.Task.run(Task.scala:89) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)

aklink commented 8 years ago

The first issue with "geotrellis.raster.io.geotiff.reader.GeoTiffCSParser.getEllipsoidInfo" is caused by an older SRS (Gauss-Krüger Zone 3, used in Baden-Württemberg) in GeoTiff file, can be fixed by applying "gdalwarp -t_srs EPSG:25832 srcfile targetfile" to file to use a newer SRS (ETRS89 UTM32N). Geotrellis seems not to support older SRS.

The second issue with "geotrellis.raster.RasterExtent.(RasterExtent.scala:72)" is caused by a missing SRS in GeoTiff file, can be fixed by applying "gdal_translate -a_srs EPSG:25832 srcfile targetfile" in advance to add a valid SRS (on test file, not in sourcecode).

aklink commented 8 years ago

Maybe related to this issues?

aklink commented 8 years ago

Problematic CRS is EPSG:31467 in following GDAL representation:

PROJCS["DHDN_3_Degree_Gauss_Zone_3",
    GEOGCS["GCS_Deutsches_Hauptdreiecksnetz",
        DATUM["Deutsches_Hauptdreiecksnetz",
            SPHEROID["Bessel_1841",6377397.155,299.1528128,
                AUTHORITY["EPSG","7004"]],
            AUTHORITY["EPSG","6314"]],
        PRIMEM["Greenwich",0],
        UNIT["degree",0.0174532925199433]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["latitude_of_origin",0],
    PARAMETER["central_meridian",9],
    PARAMETER["scale_factor",1],
    PARAMETER["false_easting",3500000],
    PARAMETER["false_northing",0],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]]]
vsimko commented 8 years ago

It works for my TIF file in EPSG_31467:

$ gdalinfo morning_EPSG_31467.tif
Driver: GTiff/GeoTIFF
Files: morning_EPSG_31467.tif
Size is 5161, 7911
Coordinate System is:
PROJCS["DHDN / 3-degree Gauss-Kruger zone 3",
    GEOGCS["DHDN",
        DATUM["Deutsches_Hauptdreiecksnetz",
            SPHEROID["Bessel 1841",6377397.155,299.1528128000008,
                AUTHORITY["EPSG","7004"]],
            TOWGS84[598.1,73.7,418.2,0.202,0.045,-2.455,6.7],
            AUTHORITY["EPSG","6314"]],
        PRIMEM["Greenwich",0],
        UNIT["degree",0.0174532925199433],
        AUTHORITY["EPSG","4314"]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["latitude_of_origin",0],
    PARAMETER["central_meridian",9],
    PARAMETER["scale_factor",1],
    PARAMETER["false_easting",3500000],
    PARAMETER["false_northing",0],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    AUTHORITY["EPSG","31467"]]
Origin = (3445385.870000000111759,5447456.839999999850988)
Pixel Size = (5.000000000000000,-5.000000000000000)
Metadata:
  AREA_OR_POINT=Area
  TIFFTAG_IMAGEDESCRIPTION=
  TIFFTAG_RESOLUTIONUNIT=3 (pixels/cm)
  TIFFTAG_XRESOLUTION=100
  TIFFTAG_YRESOLUTION=100
Image Structure Metadata:
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  ( 3445385.870, 5447456.840) (  8d15' 3.88"E, 49d 9'46.43"N)
Lower Left  ( 3445385.870, 5407901.840) (  8d15'23.00"E, 48d48'25.95"N)
Upper Right ( 3471190.870, 5447456.840) (  8d36'17.74"E, 49d 9'52.74"N)
Lower Right ( 3471190.870, 5407901.840) (  8d36'27.83"E, 48d48'32.18"N)
Center      ( 3458288.370, 5427679.340) (  8d25'48.14"E, 48d59' 9.82"N)
Band 1 Block=5161x1 Type=Byte, ColorInterp=Palette
  Color Table (RGB with 256 entries)
...
aklink commented 8 years ago

Seems like the major difference between the two CRS representations is missing AUTHORITY["EPSG","31467"] as well as TOWGS84[598.1,73.7,418.2,0.202,0.045,-2.455,6.7] in non-working variant (unfortunatelly all tiles in BW are using this non-working representation)

pomadchin commented 8 years ago

Is this still an issue? Can you throw us a non working example, or (as i can understand) the problem is in corrupted (or specific) geotiff metadata? Or it is https://github.com/geotrellis/geotrellis/issues/1415 ?

aklink commented 8 years ago

I am not sure if this exactly issue https://github.com/geotrellis/geotrellis/issues/1415 ...

However it is specific to SRS MetaData of DHDN_3_Degree_Gauss_Zone_3.

When looking on: http://spatialreference.org/ref/epsg/31467/ It is neither exactly http://spatialreference.org/ref/epsg/31467/ogcwkt/ nor http://spatialreference.org/ref/epsg/31467/esriwkt/ due to a slighty different string representation.

Please find attached file using DHDN_3_Degree_Gauss_Zone_3. DOP_RGBI_T2_400x400_epsg31467.zip

pomadchin commented 8 years ago

@aklink thx for attached file, so yes, i believe it is unsupported projection, good info, think it would be fixed (or we'll try to fix it :D) in terms of https://github.com/geotrellis/geotrellis/issues/1415 looks like a bug in a proj4j or in tiff metadata.

aklink commented 8 years ago

@pomadchin Thanks for looking into this issue. I really appreciate it and I am looking forward to have this projection supported by Geotrellis.

pomadchin commented 7 years ago

Yo guyz, can you try this PR https://github.com/locationtech/geotrellis/pull/2403 if it's not too late? :D basically it was a minor bug, sry that it took us half a year to fix it.

aklink commented 7 years ago

@pomadchin: Thanks for the update. I will have a look into it. Anyhow, I have trouble building it using ./scripts/publish-local.sh in Win10. 

Downloading sbt launcher for 0.13.16:
  From  http://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/0.13.16/sbt-launch.jar
    To  /home/ak/.sbt/launchers/0.13.16/sbt-launch.jar
Invalid maximum heap size: -Xmx2g
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

Is there any Maven repository with an already built test version? Or another simple option how to test it ontop of an "older" Geotrellis version?

pomadchin commented 7 years ago

@aklink sure, we have a location tech snapshot repo: 

resolvers ++= Seq(
  "LocationTech GeoTrellis Snapshots" at "https://repo.locationtech.org/content/repositories/geotrellis-snapshots"
)

After that just use 1.2.0-SNAPSHOT as a GeoTrellis version in your deps.

aklink commented 7 years ago

@pomadchin Sorry for my late reply. Thanks! I have tested it with 1.2.0-RC1 and it seems to work fine. I did not check the georeferenced position yet if it is on the proper location, but at least the error is gone. Layer is created properly.