Open jdries opened 1 year ago
EmileSonneveld
commented
1 year ago In the breakMap object there are some evidence parameters that cause an issue. They get created somewhere here: https://github.com/locationtech-labs/geopyspark/blob/master/geopyspark/geotrellis/color.py#L174-L176
Maybe this can be fixed by using the same kind of hack as here: https://github.com/Open-EO/openeo-geotrellis-extensions/issues/108
EmileSonneveld
commented
1 year ago This snippet will run correctly when launched from Scala, but will throw when launched from Python:
def test(sc: SparkContext): Unit = {
// val sc = SparkContext.getOrCreate
val mCopy = "0.0:aec7e8ff;1.0:d62728ff;2.0:f7b6d2ff;3.0:dbdb8dff;4.0:c7c7c7ff".split(";").map(x => {
val l = x.split(":");
// parseUnsignedInt, because there is no minus sign in the hexadecimal representation.
// When casting an unsigned int to an int, it will correctly overflow
Tuple2(l(0).toDouble, Integer.parseUnsignedInt(l(1), 16))
}).toMap
val colorMap = new DoubleColorMap(mCopy)
val formatOptions = new GTiffOptions()
formatOptions.setColorMap(colorMap)
val spire = new DoubleAlgebra()
val data = sc.parallelize(Seq(1, 2, 3))
val result = data.map(x => {
println(spire)
println(formatOptions)
x
})
result.collect()
print("test done")
}I guess this is because of different objects/libraries available in the runtime environment, messing with the serialising process
To run from python:
from openeogeotrellis.utils import get_jvm
sc = get_jvm().org.apache.spark.SparkContext.getOrCreate()
get_jvm().org.openeo.geotrellis.geotiff.package.test(sc)
jdries
commented
1 year ago Aha, so this type: type org.locationtech.geopyspark.shaded.cats.kernel.Order seems wrong, because when I look in the code, it should be of type spire.algebra.Order But we also see that some 'shading' has happened, where this 'cats.kernel.Order' type got relocated to a different package. It's quite likely that this causes some kind of mixup.
jdries
commented
1 year ago Note that nowadays, spark has a dependency on cats-kernel_2.12-2.1.1 Perhaps we can remove this shading rule from pom.xml and it might work?
EmileSonneveld
commented
1 year ago Commenting out the cats.kernel org.locationtech.geopyspark.shaded.cats.kernel part from pom.xml changes the namespace of Order:
Caused by: java.lang.ClassCastException: cannot assign instance of spire.std.DoubleAlgebra to field geotrellis.raster.render.BreakMap.evidence$1 of type spire.algebra.Order in instance of geotrellis.raster.render.BreakMap$mcDI$sp
jdries
commented
1 year ago I now removed geotrellis entirely from assembly jar: https://artifactory.vgt.vito.be/webapp/#/artifacts/browse/tree/General/auxdata-public/openeo/geotrellis-backend-assembly-0.4.8-openeo_2.12.jar Can you try if that works/helps?
EmileSonneveld
commented
1 year ago It still gives: ClassCastException: cannot assign instance of spire.std.DoubleAlgebra to field geotrellis.raster.render.BreakMap.evidence$1 of type org.locationtech.geopyspark.shaded.cats.kernel.Order in instance of geotrellis.raster.render.BreakMap$mcDI$sp
I comitted the scala test function on develop, for easier debugging: https://github.com/Open-EO/openeo-geotrellis-extensions/commit/eac875794ce143cd13cbb5569730e43c4f612e79
jdries
commented
11 months ago need to retry running test from python, assembly jar got removed
EmileSonneveld
commented
11 months ago I tried running the snippet again from python, but got the same error. https://github.com/Open-EO/openeo-geotrellis-extensions/blob/develop/openeo-geotrellis/src/main/scala/org/openeo/geotrellis/geotiff/package.scala#L739
The scala code seems still to refer to geotrelis code in a jar: /home/emile/.m2/repository/org/locationtech/geotrellis/geotrellis-raster_2.12/3.6.0/geotrellis-raster_2.12-3.6.0-sources.jar!/geotrellis/raster/render/ColorMap.scala
We still have an error related to serialization of the geotiff colormap: