Errors when detecting the SBDF column export type when column has all missing values

[bbassett-tibco]

When exporting SBDF from Python (usually encountered as part of writing the outputs for a data function), some column types like String can't be inferred if the entire column is empty. The error is informative, but doesn't really indicate how one should handle that case.
spotfire.sbdf.SBDFError: cannot determine type for column 'EmptyString'; all values are missing

In TERR/R's SBDF import/export we handle the roundtrip data type declarations by attaching an attribute to the column containing the original Spotfire data type on import and using that, if defined, to declare a specific data type on Export. We probably should implement something similar for Python SBDF. See https://docs.tibco.com/pub/enterprise-runtime-for-R/6.0.0/doc/html/Language_Reference/SpotfireData/setSBDFExportTypes.html for how the TERR/R one works.

Issue migrated from TIBCO Software JIRA [PYSRV-227] created by jorobert

[bbassett-tibco]

I've added the Known Issue to all previously-released versions, back to 1.0.0. 

• https://docs.tibco.com/pub/sf-pysrv/1.0.0/TIB_sf-pysrv_1.0.0_relnotes.pdf?id=0
• https://docs.tibco.com/pub/sf-pysrv/1.0.1/TIB_sf-pysrv_1.0.1_relnotes.pdf?id=0
• https://docs.tibco.com/pub/sf-pysrv/1.0.2/TIB_sf-pysrv_1.0.2_relnotes.pdf?id=0
• https://docs.tibco.com/pub/sf-pysrv/1.1.0/TIB_sf-pysrv_1.1.0_relnotes.pdf?id=0
• https://docs.tibco.com/pub/sf-pysrv/1.2.0/TIB_sf-pysrv_1.2.0_relnotes.pdf?id=0

I've updated 1.3.0, but I'm sitting on pushing out the publication until after we're cleared to release.  This KI will carry forward for (as-yet-uncreated) versions 1.3.1, 1.4.0, and 1.0.3 unless it's fixed. 

Comment migrated from TIBCO Software JIRA originally by kroan

[jaybazile]

I have this same issue in Python when trying to export using 1.8.0

[bbassett-tibco]

Thanks to @jorobert for his work on the feature/set-column-types branch, which provided a very good starting point for what was checked in. It basically works thusly:

• On import_data, each column has a spotfire_type attribute set (via DataFrame.attrs) with the Spotfire UI name (i.e., "Real" not "double") of the type of the column.
• On export_data, for DataFrames and Series, the spotfire_type attribute is checked. If this attribute exists and exactly matches the UI name of a type, that column will be exported as that type. Otherwise, the current algorithm of inferring the type from the Pandas dtype of the column or Python object type (for the "object" dtype) of the column data is used. When exporting non-Series/non-DataFrame objects, the current algorithm remains the only method of determining the column type, since the only thing that has Pandas-style attributes is Pandas.
• To help users manipulate the spotfire_type attributes, the utility functions get_spotfire_types and set_spotfire_types have been added to the top-level spotfire module (where we have placed other user-visible utility functions). These functions check that they are passed the right type as the first parameter, and will raise a TypeError if it is not a DataFrame. set_spotfire_types expects a dictionary that maps from column names to Spotfire type strings; invalid column names and invalid type names will result in warnings being emitted.