knoopum
commented
1 month ago The underlying issue here may be that to_featurelayer is not cleaning up after itself as expected. It is leaving its temporary geodatabase behind on the local file system in a locked state.
The expectation is that if one uses to_featurelayer, and then deletes the published feature layer (and related geodatabase) from ArcGIS Online, then one should be able to run to_featurelayer with the same parameters again.
If one tries, then you get an error message:
FileExistsError: [WinError 183] Cannot create a file when that file already exists...If the second attempt to publish occurs in the same script/notebook session, then the presence of the lock files in the geodatabase prevent one from manually deleting the temporary geodatabase as a workaround to continue.
The code below is another way to reproduce this issue. It was run in the standalone Jupyter notebook environment that comes with ArcGIS Pro 3.3.2, and includes version 2.3.0 of the ArcGIS Python API.
import arcgis
from arcgis import GIS
import arcpy
# ArcGIS API for Python version:
print('ArcGIS API for Python version:', arcgis.__version__)
# Connect to default GIS.
gis = GIS("home")
# Prepare some test data as a dictionary for a feature set
d = {
'features': [
{
'geometry': {
'x': -4243783.810391866,
'y': 4640950.609361529,
'spatialReference': {
'wkid': 102100,
'latestWkid': 3857
}
},
'attributes': {
'OBJECTID': 1, 'Eight_Character': 'test'
}
}
],
'objectIdFieldName': 'OBJECTID',
'globalIdFieldName': '',
'spatialReference': {
'wkid': 102100,
'latestWkid': 3857
},
'geometryType': 'esriGeometryPoint',
'fields': [
{
'name': 'OBJECTID',
'type': 'esriFieldTypeOID',
'alias': 'OBJECTID',
'sqlType': 'sqlTypeOther',
'domain': None,
'defaultValue': None
},
{
'name': 'Eight_Character',
'type': 'esriFieldTypeString',
'alias': 'Eight_Character',
'sqlType': 'sqlTypeOther',
'length': 8,
'domain': None,
'defaultValue': None
}
]
}
# Convert dictionary to feature set
fs = arcgis.features.FeatureSet.from_dict(d)
# Convert feature set to spatial data frame
sdf = fs.sdf
# Publish feature set as spatial data frame to feature layer.
print("\nPublishing...")
item = sdf.spatial.to_featurelayer(
title = 'Bug to_featurelayer not cleaning up',
gis = gis,
service_name = 'Bug_to_featurelayer_not_cleaning_up'
)
# Delete published feature layer and file geodatabase
print("\nDeleting...")
fgdb_item = item.related_items(
rel_type = 'Service2Data'
)[0]
print("Delete item:", item.delete(permanent = True))
print("Delete item:", fgdb_item.delete(permanent = True))
# Once again, publish feature set as spatial data frame to feature layer.
print("\nPublishing again...")
item = sdf.spatial.to_featurelayer(
title = 'Bug to_featurelayer not cleaning up',
gis = gis,
service_name = 'Bug_to_featurelayer_not_cleaning_up'
)The resulting output in the notebook:
Publishing...
Deleting...
Delete item: True
Delete item: True
Publishing again...
---------------------------------------------------------------------------
FileExistsError Traceback (most recent call last)
~\AppData\Local\ESRI\conda\envs\work\Lib\site-packages\arcgis\gis\_impl\_content_manager\_import_data.py in _create_file_item(gis, df, file_type, **kwargs)
84 # set up temporary zip to be used in directory
---> 85 os.makedirs(temp_dir)
86 temp_zip = os.path.join(temp_dir, "%s.zip" % ("a" + uuid4().hex[:5]))
~\AppData\Local\ESRI\conda\envs\work\Lib\os.py in makedirs(name, mode, exist_ok)
FileExistsError: [WinError 183] Cannot create a file when that file already exists: 'C:\\Users\\knoop\\AppData\\Local\\Temp\\2\\Bug_to_featurelayer_not_cleaning_up'
During handling of the above exception, another exception occurred:
UnboundLocalError Traceback (most recent call last)
~\AppData\Local\Temp\2\ipykernel_95840\262642015.py in <cell line: 0>()
78 # Once again, publish feature set as spatial data frame to feature layer.
79 print("\nPublishing again...")
---> 80 item = sdf.spatial.to_featurelayer(
81 title = 'Bug to_featurelayer not cleaning up',
82 gis = gis,
~\AppData\Local\ESRI\conda\envs\work\Lib\site-packages\arcgis\features\geo\_accessor.py in to_featurelayer(self, title, gis, tags, folder, sanitize_columns, service_name, **kwargs)
2910 )
2911
-> 2912 result = content.import_data(
2913 self._data,
2914 folder=folder,
~\AppData\Local\ESRI\conda\envs\work\Lib\site-packages\arcgis\gis\__init__.py in import_data(self, df, address_fields, folder, item_id, **kwargs)
8589 if _is_geoenabled(df) or (overwrite or insert):
8590 # Item Workflow
-> 8591 return _cm_helper.import_as_item(self._gis, df, **kwargs)
8592 else:
8593 # Feature Collection Workflow
~\AppData\Local\ESRI\conda\envs\work\Lib\site-packages\arcgis\gis\_impl\_content_manager\_import_data.py in import_as_item(gis, df, **kwargs)
245
246 # Create the file item, new item published from the file item, and the publish parameters
--> 247 file_item, new_item = _create_file_item(gis, df, file_type, **kwargs)
248
249 # If not overwrite or insert, return the new item
~\AppData\Local\ESRI\conda\envs\work\Lib\site-packages\arcgis\gis\_impl\_content_manager\_import_data.py in _create_file_item(gis, df, file_type, **kwargs)
153 shutil.rmtree(temp_dir, ignore_errors=True)
154
--> 155 if os.path.exists(file):
156 os.remove(file)
157 return file_item, new_item
UnboundLocalError: cannot access local variable 'file' where it is not associated with a value
Describe the bug When using spatial.to_featurelayer on a moderately large (62,000 rows, 93 columns) spatially enabled data frame, I'm getting the following error, and the feature layer is never created.
To Reproduce Run the python script atteched below (file extension changed to .txt to allow uploading). I'd rather not post the data file here publicly, so if someone from ESRI takes a look at this, feel free to contact me and I'll email it. I've tried making a dummy file to post here instead of the actual one, but nothing I've generated so far has been able to cause this error. Also note that the file originally had 220k+ lines; I'm able to reduce the error with a subset of this file that's 62k lines or more, but once I start making files smaller than that, the error doesn't occur. This is independent of which portion of the records I use to create the smaller files. to_featurelayer_bug_demo.txt
Expected behavior I expect to_featurelayer() to create my feature layer without throwing an error.
Platform (please complete the following information):
Additional context This error started occurring after Python API version 2.0.1. (Until I recently upgraded to ArcGIS Pro 3.3.1, I was specifically installing Python API version 2.0.1 instead of more recent versions because it would successfully run my script without throwing this error.)