ClassCastException in geopyspark.query on Accumulo backend

[jdries]

When doing a simple query on my Accumulo backend, I get this exception: (Solution below) py4j.protocol.Py4JJavaError: An error occurred while calling o188.query. : java.lang.ClassCastException: geotrellis.spark.io.LayerHeader$LayeHeaderFormat$$anon$1 cannot be cast to geotrellis.spark.io.accumulo.AccumuloLayerHeader at geotrellis.spark.io.accumulo.AccumuloLayerReader.read(AccumuloLayerReader.scala:43) at geotrellis.spark.io.accumulo.AccumuloLayerReader.read(AccumuloLayerReader.scala:31) at geotrellis.spark.io.FilteringLayerReader.read(FilteringLayerReader.scala:57) at geopyspark.geotrellis.io.LayerReaderWrapper.query(LayerReaderWrapper.scala:113) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:280) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:214) at java.lang.Thread.run(Thread.java:748) This is caused by caching of layer attributes in the attribute store. On line 49 of geopyspark.geotrellis.io.LayerReaderWrapper, the Layer header is already read into a different type of object: val header = attributeStore.readHeaderLayerHeader I worked around this by clearing the cache immediately afterwards: attributeStore.clearCache(id) Not sure if this is the proper way to fix this?

[jbouffard]

Hello @jdries! Could you tell me what version of GPS you're using? Or the last sha of the master branch if you cloned from GitHub? As there doesn't appear to be anything on line 49 in LayerReaderWrapper in the current master branch (though the line that creates header is still there).

[jdries]

Hi,

I checked out the code at tag v0.3.0, which might explain the shift. (It looks quite similar on master.)

best regards,

Jeroen

Jeroen Dries TAP - Centre for Remote Sensing and Earth Observation Processes VITO NV | Boeretang 200 | 2400 Mol tel. +32 14 33 55 11 | jeroen.dries@vito.behttps://outlook.vito.be/OWA/redir.aspx?C=bkbvq29Z1EO1hvVK0QJMbACOitzQ7c8Ip7CWFVkhZV75OTezDjCIkdDzkCq1rif2MTOFHY6bxtg.&URL=mailto%3avoornaam.naam%40vito.be [cid:image001.jpg@01CDCE3A.4E197A50]https://outlook.vito.be/OWA/redir.aspx?C=bkbvq29Z1EO1hvVK0QJMbACOitzQ7c8Ip7CWFVkhZV75OTezDjCIkdDzkCq1rif2MTOFHY6bxtg.&URL=http%3a%2f%2fwww.vito.be%2f[cid:image001.jpg@01CD3F0D.B57AA190]https://outlook.vito.be/OWA/redir.aspx?C=bkbvq29Z1EO1hvVK0QJMbACOitzQ7c8Ip7CWFVkhZV75OTezDjCIkdDzkCq1rif2MTOFHY6bxtg.&URL=http%3a%2f%2fwww.linkedin.com%2fcompany%2f17947[cid:image003.jpg@01CD3F0D.B57AA190]https://outlook.vito.be/OWA/redir.aspx?C=bkbvq29Z1EO1hvVK0QJMbACOitzQ7c8Ip7CWFVkhZV75OTezDjCIkdDzkCq1rif2MTOFHY6bxtg.&URL=http%3a%2f%2ftwitter.com%2f%23%21%2fvitobelgium[cid:image002.jpg@01CD3F0D.B57AA190]https://outlook.vito.be/OWA/redir.aspx?C=bkbvq29Z1EO1hvVK0QJMbACOitzQ7c8Ip7CWFVkhZV75OTezDjCIkdDzkCq1rif2MTOFHY6bxtg.&URL=https%3a%2f%2fwww.facebook.com%2fVITObelgium

---

Van: Jacob Bouffard notifications@github.com Verzonden: donderdag 18 januari 2018 16:13 Aan: locationtech-labs/geopyspark CC: Dries Jeroen; Mention Onderwerp: Re: [locationtech-labs/geopyspark] ClassCastException in geopyspark.query on Accumulo backend (#618)

Hello @jdrieshttps://github.com/jdries! Could you tell me what version of GPS you're using? Or the last sha of the master branch if you cloned from GitHub? As there doesn't appear to be anything on line 49https://github.com/locationtech-labs/geopyspark/blob/master/geopyspark-backend/geotrellis/src/main/scala/geopyspark/geotrellis/io/LayerReaderWrapper.scala?utf8=%E2%9C%93#L49 in LayerReaderWrapper in the current master branch (though the line that creates header is still there).

- You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/locationtech-labs/geopyspark/issues/618#issuecomment-358676395, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AFqXyL4Yrcyj4fm9kjmNwjKOYLxiQMbhks5tL19-gaJpZM4Ri3se.

Indien u VITO Mol bezoekt, hou aub er dan rekening mee dat de hoofdingang voortaan enkel bereikbaar is vanuit de richting Dessel-Retie, niet vanuit richting Mol, zie vito.be/route.http://www.vito.be/route If you plan to visit VITO at Mol, then please note that the main entrance can only be reached coming from Dessel-Retie and no longer coming from Mol, see vito.be/en/contact/locations.http://www.vito.be/en/contact/locations VITO Disclaimer: http://www.vito.be/e-maildisclaimer

[jbouffard]

Hey, @jdries . Yeah, those two versions are pretty similar, but I wanted to be sure.

Just so I understand what's happening: you're caching attributes with an AttributeStore for a backend that's not Accumulo and then when you try query a layer using Accumulo with that AttributeStore you get an error. Is that correct?

[jdries]

Hi Jacob,

it's actually more simple, I just do a query like this:

gps.query(uri="accumulo://epod6.vgt.vito.be:2181/hdp-accumulo-instance",layer_name="accumulo_layer")

and then I already get the exception. I don't interfere with the caching or the attributestore at all.

Jeroen Dries TAP - Centre for Remote Sensing and Earth Observation Processes VITO NV | Boeretang 200 | 2400 Mol tel. +32 14 33 55 11 | jeroen.dries@vito.behttps://outlook.vito.be/OWA/redir.aspx?C=bkbvq29Z1EO1hvVK0QJMbACOitzQ7c8Ip7CWFVkhZV75OTezDjCIkdDzkCq1rif2MTOFHY6bxtg.&URL=mailto%3avoornaam.naam%40vito.be [cid:image001.jpg@01CDCE3A.4E197A50]https://outlook.vito.be/OWA/redir.aspx?C=bkbvq29Z1EO1hvVK0QJMbACOitzQ7c8Ip7CWFVkhZV75OTezDjCIkdDzkCq1rif2MTOFHY6bxtg.&URL=http%3a%2f%2fwww.vito.be%2f[cid:image001.jpg@01CD3F0D.B57AA190]https://outlook.vito.be/OWA/redir.aspx?C=bkbvq29Z1EO1hvVK0QJMbACOitzQ7c8Ip7CWFVkhZV75OTezDjCIkdDzkCq1rif2MTOFHY6bxtg.&URL=http%3a%2f%2fwww.linkedin.com%2fcompany%2f17947[cid:image003.jpg@01CD3F0D.B57AA190]https://outlook.vito.be/OWA/redir.aspx?C=bkbvq29Z1EO1hvVK0QJMbACOitzQ7c8Ip7CWFVkhZV75OTezDjCIkdDzkCq1rif2MTOFHY6bxtg.&URL=http%3a%2f%2ftwitter.com%2f%23%21%2fvitobelgium[cid:image002.jpg@01CD3F0D.B57AA190]https://outlook.vito.be/OWA/redir.aspx?C=bkbvq29Z1EO1hvVK0QJMbACOitzQ7c8Ip7CWFVkhZV75OTezDjCIkdDzkCq1rif2MTOFHY6bxtg.&URL=https%3a%2f%2fwww.facebook.com%2fVITObelgium

---

Van: Jacob Bouffard notifications@github.com Verzonden: donderdag 18 januari 2018 17:33 Aan: locationtech-labs/geopyspark CC: Dries Jeroen; Mention Onderwerp: Re: [locationtech-labs/geopyspark] ClassCastException in geopyspark.query on Accumulo backend (#618)

Hey, @jdrieshttps://github.com/jdries . Yeah, those two versions are pretty similar, but I wanted to be sure.

Just so I understand what's happening: you're caching attributes with an AttributeStore for a backend that's not Accumulo and then when you try query a layer using Accumulo with that AttributeStore you get an error. Is that correct?

- You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/locationtech-labs/geopyspark/issues/618#issuecomment-358702534, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AFqXyMEmf0EJX5hAEO8QA76QssoIvTCWks5tL3JBgaJpZM4Ri3se.

Indien u VITO Mol bezoekt, hou aub er dan rekening mee dat de hoofdingang voortaan enkel bereikbaar is vanuit de richting Dessel-Retie, niet vanuit richting Mol, zie vito.be/route.http://www.vito.be/route If you plan to visit VITO at Mol, then please note that the main entrance can only be reached coming from Dessel-Retie and no longer coming from Mol, see vito.be/en/contact/locations.http://www.vito.be/en/contact/locations VITO Disclaimer: http://www.vito.be/e-maildisclaimer

[jbouffard]

@jdries Ah, I see. There is definitely an issue then. Assuming your URI and layer_name are correct, then that should work. I'll look into this issue more and will hopefully have a solution within the next few days. Thanks for bringing this problem to our attention!

[jbouffard]

Hey @jdries Sorry for taking so long to get back to you on this issue. After spending some time looking at it, we're not really sure why you're getting this error. Everything looks correct.

We have found a work around which is in #621, though. That solution will change the API a bit, but I don't think the impacts of that change should be too severe.