Esri / arcgis-python-api

Documentation and samples for ArcGIS API for Python
https://developers.arcgis.com/python/
Apache License 2.0
1.87k stars 1.1k forks source link

pandas.DataFrame.spatial.from_table/.to_table reads/writes an incomplete number of rows in 1.9.1 #1234

Closed jf-geo closed 2 years ago

jf-geo commented 2 years ago

Describe the bug When writing an SEDF/DF to a FileGDB attribute table using pandas.DataFrame.spatial.to_table, an incomplete number of rows are written. The same goes for reading an attribute table using pandas.DataFrame.spatial.from_table, an incomplete number of rows are read.

For a single dataframe, the number of incomplete rows written is consistent for each attempt. The number of incomplete rows written is inconsistent across different dataframes. The same goes for reading from attribute tables. The number of rows read & written from the same table is inconsistent too.

To Reproduce Steps to reproduce the behavior:

import pandas as pd

from arcgis.features import GeoAccessor, GeoSeriesAccessor

csv = "/some_table.csv"

location= "/some_file_geodatabase.gdb/some_table"

df = pd.read_csv(csv)

_ = df.spatial.to_table(location)

## Visually confirm in ArcPro that table was/wasn't written correctly

df_2 = pd.DataFrame.spatial.from_table(location)

## Confirm df & df_2 contain the same number of rows
print(len(df.index) == len(df_2.index))

error: No error code.

Expected behavior When writing an SEDF/DF to a FileGDB attribute table using pandas.DataFrame.spatial.to_table all the rows are written. The same goes for reading an attribute table from a FileGDB using pandas.DataFrame.spatial.from_table.

Platform (please complete the following information):

Additional context My organisation's IT restrictions mean that we're stuck with base conda environment and can only have one version of ArcPro installed at any one time (currently 2.9.0). This bug breaks one of our python toolboxes.

nanaeaubry commented 2 years ago

Thank you for reporting this, we will take a look and post further updates

achapkowski commented 2 years ago

Can you post a sample CSV file?

jf-geo commented 2 years ago

Thanks,

Ironically I can't seem to reproduce the writing-to-a-table issue with the CSVs I was having issues with. I'll try a few things and get back to you.

Using some test data the reading-from-a-table issue is present for rows with Null values in the attached CSV.

test_data.csv

nanaeaubry commented 2 years ago

@jf-geo The null values are being skipped because in the from_table method there is a parameter called: skip_nulls and the default is set to True.

See the documentation here: https://developers.arcgis.com/python/api-reference/arcgis.features.toc.html?highlight=from_table#arcgis.features.GeoAccessor.from_table

If you set the value to False you will see all the rows returned.