Esri / arcgis-python-api

Documentation and samples for ArcGIS API for Python
https://developers.arcgis.com/python/
Apache License 2.0
1.89k stars 1.1k forks source link

Spatially Enabled DataFrames convert null values to the string "None" #1643

Closed skykasko closed 1 year ago

skykasko commented 1 year ago

Describe the bug When a database table is loaded into a Spatially Enabled DataFrame using pandas.DataFrame.spatial.from_featureclass(), null values in the database are incorrectly converted to the string "None", when they should be converted to the object None.

I have reproduced this issue with an Oracle and Postges database. When using a file geodatabase, this issue does not occur, and null values are correctly loaded into the DataFrame as None.

To Reproduce Suppose you have a database table my_table with a string column notes that has a null value in the first row:

import pandas as pd
from arcgis.features import GeoAccessor

df = pd.DataFrame.spatial.from_featureclass("my_table")
print(repr(df.notes[0]))
# 'None'
print(type(df.notes[0]))
# <class 'str'>

Expected behavior I would expect that null values in text columns of Oracle, Postgres, and other database tables would be handled the same way that they currently are in file geodatabase tables. In the example above, the output should be:

print(repr(df.notes[0]))
# None
print(type(df.notes[0]))
# <class 'NoneType'>

Platform

Additional context The current behavior has caused errors in my AGOL update process, which involves loading a database table into a Spatially Enabled DataFrame and calling an AGOL feature layer's edit_features() method to apply the needed updates. When a value that was null in the database and has been converted to "None" in the DataFrame is inserted in a text field with a maximum of fewer than 4 characters, an error occurs and the update is rejected.

I expect there are other situations where it would be necessary to distinguish whether a cell stores a null value or the string "None", and this is currently not possible.

nanaeaubry commented 1 year ago

@skykasko First try to update your version of the python API to version 2.1.0.3

We made fixes for this at that version I believe and even more for the upcoming version (2.2.0) that will be released in September

skykasko commented 1 year ago

Thanks for the suggestion! I updated to ArcGIS Pro version 3.1.3, which comes with version 2.1.0.2 of the ArcGIS API for Python, and my issue was resolved. Now null values in string columns are consistently converted to pandas.NA regardless of whether the DataFrame is loaded from a file geodatabase, an Oracle or Postgres database, or an AGOL feature layer.