Issues with Writing Spatial Data in WKB Format to SBDF Files Using export_data Function

[bschwartzjetrock]

Description:

When I attempt to write spatial data (in WKB format) to a Spotfire Binary Data Format (SBDF) file using the export_data function provided in the spotfire Python package, the geometry data does not appear correctly when viewing the SBDF file in Spotfire. Instead of expected Geometry objects, I am seeing polygons.

Here's an example of how the spatial data is represented when viewed in Spotfire: POLYGON ((-101.63958257 33.51659818, -101.63958695 33.50205422, -101.65510673 33.50201273, -101.65696424 33.50202114, -101.65694942 33.51653987, -101.63958257 33.51659818))

I'm curious if there's a specific format or method for correctly processing spatial data when writing to an SBDF. Alternatively, this could be a bug within the package.

Environment: Python version: 3.11.3 spotfire package version: 2.0.0

[bbassett-tibco]

Hi, @bschwartzjetrock! Do you happen to have a sample data frame or a reproducible Python sample that demonstrates the problem? I don't have enough information to be able to diagnose the specific problem you're having.

Generally, export_data will only process geometry objects when exporting the "geometry" column of a geopandas GeoDataFrame. I'm not sure why your geometry is appearing in Spotfire as the WKT representation, but I suspect your geometry objects are being converted to Spotfire "String" type (given that Shapely geometry objects convert to WKT when passed through the Python str() function). Again, a concrete example would help to identify the problem.

[bschwartzjetrock]

Hi, @bbassett-tibco as it seems you are aware I am also in contact with TIBCO support and I have attached set of items that display the issue. Let me know if you want me to provide any files here.

I have made a lot of improvements I discovered as you said "_exportdata will only process geometry objects when exporting the "geometry" column of a geopandas GeoDataFrame" by digging a little through the codebase. With this information I was able to produce sbdf files that represent geometry objects and I can view them on the map in Spotfire. The issue I am running into now is when I zoom in on the map the objects begin to disappear. This is a behavior I have never seen when using Spotfire, but also the first time producing my own sbdfs with spatial data.

What may be the issue is the geometry column in my dataset contains both polygons and multipolygons. The library seems to not like having multiple geometry types in one column. I found this error checking within your script that raises an error if there are multiple geometry types and it seems to not be able to handle multipolygons.

(In sbdf.pyx, Line 1012)

Table Metadata

table_metadata["MapChart.IsGeocodingTable"] = [True]
if not table_metadata.get("MapChart.IsGeocodingEnabled"):
    table_metadata["MapChart.IsGeocodingEnabled"] = [True]
# Geometry types
if geometry.geom_type.nunique() != 1:
    raise SBDFError("cannot convert mixed geometry types")
geom_type = geometry[0].geom_type
if geom_type == "Point":
    table_metadata["MapChart.GeometryType"] = ["Point"]
elif geom_type == "LineString" or geom_type == "LinearRing":
    table_metadata["MapChart.GeometryType"] = ["Line"]
elif geom_type == "MultiLineString":
    table_metadata["MapChart.GeometryType"] = ["PolyLine"]
elif geom_type == "Polygon":
    table_metadata["MapChart.GeometryType"] = ["Polygon"]
else:
    raise SBDFError("cannot convert collections of Shapely objects")

Even with this error raised the sbdf file is produced and the only issue is the zooming issue described above. Even the multipolygons appear correctly until zoomed in.

[bbassett-tibco]

Thanks for the additional info. As I see it, there are actually two issues at play here:

1. The exception being raised by the table metadata code you highlighted is being swallowed by the Cython function call of _export_obj_geodataframe_geometry.
2. Our conditions for converting GeoDataFrames to DataFrames appear to be more restrictive than is required for importing into Spotfire. I suspect this may have been confusion between geographic data and geocoding tables on the part of the original author (who is no longer with the team); I'll follow up with the Spotfire Geoanalytics Engineering team to get a better handle on what is actually required.

We'll target a future release of the spotfire package for fixes.

[bschwartzjetrock]

Amazing thank you Brian!

Is there a way for me to be notified when this change is made?

On Thu, Jun 15, 2023 at 3:04 PM Brian Bassett @.***> wrote:

Thanks for the additional info. As I see it, there are actually two issues at play here:

1. The exception being raised by the table metadata code you highlighted is being swallowed by the Cython function call of _export_obj_geodataframe_geometry.
2. Our conditions for converting GeoDataFrames to DataFrames appear to be more restrictive than is required for importing into Spotfire. I suspect this may have been confusion between geographic data and geocoding tables on the part of the original author (who is no longer with the team); I'll follow up with the Spotfire Geoanalytics Engineering team to get a better handle on what is actually required.

We'll target a future release of the spotfire package for fixes.

— Reply to this email directly, view it on GitHub https://github.com/TIBCOSoftware/spotfire-python/issues/44#issuecomment-1593658340, or unsubscribe https://github.com/notifications/unsubscribe-auth/A7WH3KU4FIM7ODD7C5DFO3TXLNTEJANCNFSM6AAAAAAY3KWSOA . You are receiving this because you were mentioned.Message ID: @.***>

[bschwartzjetrock]

Additionally, do you believe that this could relate to the issue where objects disappear as I zoom in on the map?

Thanks!

On Thu, Jun 15, 2023 at 3:08 PM Benjamin Schwartz @.***> wrote:

Amazing thank you Brian!

Is there a way for me to be notified when this change is made?

On Thu, Jun 15, 2023 at 3:04 PM Brian Bassett @.***> wrote:

Thanks for the additional info. As I see it, there are actually two issues at play here:

1. The exception being raised by the table metadata code you highlighted is being swallowed by the Cython function call of _export_obj_geodataframe_geometry.
2. Our conditions for converting GeoDataFrames to DataFrames appear to be more restrictive than is required for importing into Spotfire. I suspect this may have been confusion between geographic data and geocoding tables on the part of the original author (who is no longer with the team); I'll follow up with the Spotfire Geoanalytics Engineering team to get a better handle on what is actually required.

We'll target a future release of the spotfire package for fixes.

— Reply to this email directly, view it on GitHub https://github.com/TIBCOSoftware/spotfire-python/issues/44#issuecomment-1593658340, or unsubscribe https://github.com/notifications/unsubscribe-auth/A7WH3KU4FIM7ODD7C5DFO3TXLNTEJANCNFSM6AAAAAAY3KWSOA . You are receiving this because you were mentioned.Message ID: @.***>

[bbassett-tibco]

You should get notified when this issue is closed. As for when we release the change to PyPI, you should be able to watch this repository (at the top of any of our GitHub pages); you can limit it to just releases by using a custom watch notification.

I don't think that there is any relationship to your disappearing objects under zoom conditions issue. This package only deals with the Python side of things (which means that the SBDF file is the boundary involved here). Unless we're putting malformed data of some kind (and it would be only slightly malformed, since it is being imported back into Spotfire successfully, which discounts gross structural issues), the issue is entirely on Spotfire and the map chart. Please continue working with TIBCO support to remedy that issue.

[bbassett-tibco]

Yep, the requirements for geographic data are much less than for geocoding tables (specifically, only the geometry and the CRS are required to get Spotfire to pick up geographic data). As a result, I've moved the geocoding table logic from sbdf_export into a new public function set_geocoding_table for users who require the other features (the additional columns and metadata properties).