search

agrc / palletjack

A library for updating AGOL data from various external sources
https://agrc.github.io/palletjack/palletjack/
MIT License
12 stars 0 forks source link

Pre-check and/or fill nans before append #26

Closed jacobdadams closed 1 year ago

jacobdadams commented 1 year ago

FeatureLayer.append apparently really doesn't like nans/Nones in it's source dataframe (or maybe it's the conversion to geojson via feature set). It will throw a weird Exception: Object reference not set to an instance of an object. (Error Code: 400) error. Removing any nans (filling to 0s or empty strings) seems to fix this.

Palletjack should at the very least check for nans or fill them with appropriate data (0s -999, or something).

Or, investigate further to see if it's something in the geojson conversion that can more gracefully handle nans.

Is it something to do with certain fields that aren't set to nullable in AGOL?

jacobdadams commented 1 year ago

This error occurs when we try to set a field that doesn't allow null values and doesn't have a default value to a null value. If a default value is specified, it will use that instead and not throw the error.

It also occurs when we exceed a field's length value.

Write checks for both of these conditions.

jacobdadams commented 1 year ago

Also: int columns with nulls get converted to floats with np.nan for missing values. Convert to nullable ints.

jacobdadams commented 1 year ago

Extra stuff that seems to belong here: arcgis 2.1 handles conversion of null dataframe values to null values in featureset/geojson. However, if we convert a datetime column to the python nullable string type, it converts <NA> to an empty string, which apparently gets uploaded/converted to <> in AGOL, which isn't a valid date string and thus throws an error. This should be caught by the datetime checks.. but may not be.

Similarly, nullable Int64 type <NA>s get switched to 0s (the docs mention similar behavior in to_featurelayer() being caused by shapely conventions). 6cde768d249acb2dc2d94bc211eb4898bbf1a991 and 59d9593027b41a642cb41aa7c1e946c56479f6f8 added a check and warning for this behavior.