search

Esri / arcgis-python-api

Documentation and samples for ArcGIS API for Python
https://developers.arcgis.com/python/
Apache License 2.0
1.89k stars 1.1k forks source link

Certain characters in SDF sets off UTF8 decoding error which causes to_featurelayer to fail #982

Closed williehk closed 3 years ago

williehk commented 3 years ago

Describe the bug Certain characters in SDF column sets off UTF8 decoding error which causes to_featurelayer to fail

To Reproduce Export from SDF to feature layer a SDF with a column containing the UTF8 character hex code point 23FB7 http://www.ltg.ed.ac.uk/~richard/utf-8.cgi?input=23FB7&mode=hex

https://services8.arcgis.com/PXQv9PaDJHzt8rp0/arcgis/rest/services/Merge_Display_0227_test_view/FeatureServer/0 Row entry OBJECTID 46448 https://services8.arcgis.com/PXQv9PaDJHzt8rp0/arcgis/rest/services/Merge_Display_0227_test_view/FeatureServer/0/query?where=&objectIds=46448&time=&geometry=&geometryType=esriGeometryEnvelope&inSR=&spatialRel=esriSpatialRelIntersects&resultType=standard&distance=0.0&units=esriSRUnit_Meter&returnGeodetic=false&outFields=*&returnGeometry=true&featureEncoding=esriDefault&multipatchOption=xyFootprint&maxAllowableOffset=&geometryPrecision=&outSR=&datumTransformation=&applyVCSProjection=false&returnIdsOnly=false&returnUniqueIdsOnly=false&returnCountOnly=false&returnExtentOnly=false&returnQueryGeometry=false&returnDistinctValues=false&cacheHint=false&orderByFields=&groupByFieldsForStatistics=&outStatistics=&having=&resultOffset=&resultRecordCount=&returnZ=false&returnM=false&returnExceededLimitFeatures=true&quantizationParameters=&sqlFormat=none&f=html&token=

大廈名單: | 柏𣾷6座

<insert your code here>

error:

---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

The above exception was the direct cause of the following exception:

SystemError                               Traceback (most recent call last)
<ipython-input-58-966598729207> in <module>
----> 1 layer_to_be_updated = sdf.iloc[14251:14255].spatial.to_featurelayer(title=title,gis=gis,tags=["python","automation","covid19"])

C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\features\geo\_accessor.py in to_featurelayer(self, title, gis, tags, folder)
  2180                 raise ValueError("GIS object must be provided")
  2181         content = gis.content
-> 2182         return content.import_data(self._data, folder=folder, title=title, tags=tags)
  2183     # ----------------------------------------------------------------------
  2184     @staticmethod

C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\__init__.py in import_data(self, df, address_fields, folder, item_id, **kwargs)
  5276                     ds =\
  5277                         df.spatial.to_featureclass(location=os.path.join(fgdb,
-> 5278                                                                          os.path.basename(temp_dir)))
  5279
  5280                 zip_fgdb = zipws(path=fgdb, outfile=temp_zip, keep=True)

C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\features\geo\_accessor.py in to_featureclass(self, location, overwrite, has_z, has_m, sanitize_columns)
  2120                                has_z=has_z,
  2121                                sanitize_columns=sanitize_columns,
-> 2122                                has_m=has_m)
  2123     #----------------------------------------------------------------------
  2124     def to_table(self, location, overwrite=True):

C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\features\geo\_io\fileops.py in to_featureclass(geo, location, overwrite, validate, sanitize_columns, has_m, has_z)
   728             q = df[geo._name].isna()
   729             df.loc[q, 'SHAPE'] = null_geom # set null values to proper JSON
--> 730             np.apply_along_axis(_insert_row, 1, df[dfcols].values)
   731             df.loc[q, 'SHAPE'] = None # reset null values
   732         df.set_index(old_idx)

<__array_function__ internals> in apply_along_axis(*args, **kwargs)

~\AppData\Roaming\Python\Python37\site-packages\numpy\lib\shape_base.py in apply_along_axis(func1d, axis, arr, *args, **kwargs)
   377             'Cannot apply_along_axis when any iteration dimensions are 0'
   378         ) from None
--> 379     res = asanyarray(func1d(inarr_view[ind0], *args, **kwargs))
   380
   381     # build a buffer for storing evaluations of func1d.

C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\features\geo\_io\fileops.py in _insert_row(row)
   725                     if isinstance(row[idx], type(pd.NaT)):
   726                         row[idx] = None
--> 727                 irows.insertRow(row)
   728             q = df[geo._name].isna()
   729             df.loc[q, 'SHAPE'] = null_geom # set null values to proper JSON

SystemError: <method 'insertRow' of 'da.InsertCursor' objects> returned a result with an error set

Expected behavior Export a feature layer to AGOL

Platform (please complete the following information):

scdub commented 3 years ago

@williehk Thank you for the report. This issue is in the .da cursors of ArcPy, and will be addressed in Pro 2.8.

achapkowski commented 3 years ago

@williehk I am going to close this issue since it is not an ArcGIS API for Python issue. Thank you for reporting it!