search

Esri / arcgis-osm-editor

ArcGIS Editor for OpenStreetMap is a toolset for GIS users to access and contribute to OpenStreetMap through their Desktop or Server environment.
Apache License 2.0
395 stars 129 forks source link

OSM File Loader (Load Only) fails on Netherlands extract #175

Closed mboeringa closed 5 years ago

mboeringa commented 7 years ago

Hi Thomas,

I don't know what changed in my setup, besides the regular Windows updates, but I now get a failure to load the Netherlands extract of Geofabrik. I have loaded this extract multiple times over the past years, and never got this kind of failure.

I attempted it with both 64 bit background geoprocessing enabled, and disabled, both fail at the same point, when loading of ways needs to start. It appears the Python processes, which seem to have already been initiated, still wait for some response from ArcMap before they can continue. The processes simply do not continue beyond this stage, even after waiting hours. If I kill ArcMap using the Task Manager, the intiated Python processes DO start to run. See the attached images explaining the situation.

geoprocessing dialog

task_manager_arcmap_process

task_manager_python_processes

explorer_gdbs

explorer_osm_files

task_manager_python_processes_after_arcmap_kill

ThomasEmge commented 7 years ago

Marco, not sure what or why it would happen. I have used the tools over the past two weeks on a non-developer machine without any issues. From your screen captures it looks like the order of processing is working as expected. I don't expect any issues coming from the OS updates, as the python modules seem to be starting just fine. I am tending towards some locking issues as there seems to be a dependence on ArcMap. How many ArcMap/ArcCatalog instances do you have running when this happens?

mboeringa commented 7 years ago

@ThomasEmge , could this potentially be caused by some firewall restriction on Python, where the Python processes can't communicate with ArcMap, while they can obviously be launched as the processes are visible in the Task Manager?

I am asking because I have seen Windows putting restrictions on Python automatically and warning about this. That never seemed to affect the running of the import process though, but I don't know if some recent Windows update set some new restriction.

Or is this unlikely or impossible considering the particular point of failure?

mboeringa commented 7 years ago

I am tending towards some locking issues as there seems to be a dependence on ArcMap. How many ArcMap/ArcCatalog instances do you have running when this happens?

My last runs were with just a single ArcMap session open to prevent such possible locking issues. I also made sure all other Python and ArcMap session were killed using the Task Manager before attempting another import, or rebooted.

ThomasEmge commented 7 years ago

Hmm, there goes the locking idea. I don't think it is a firewall issue as Python is only asking for a file on disk. Nothing really leaves the computer or establishes a connection with an outside service. Even if i:\ drive is on a network, it worked for the points/nodes. Have you tried to write the points, lines, and polygons into different (separate) file geodatabases? I wondering if there are some other resources being exhausted.

mboeringa commented 7 years ago

@ThomasEmge , I also see this issue now with the import of a small dataset. There's plenty of free RAM and disk space.

I have now tested the import of the small Bosnia Geofabrik extract again that also failed, and it again got stuck. I do see one lock file inside the File Geodatabase. If I look in Windows Source Control as opened from the Task Manager, I see this single lock file having the same Process ID (PID) as the ArcMap session I used to import the data. So only the ArcMap session that initiated the import, seems to hold a lock on the File Geodatabase during the import and the particular stage it is failing.

I guess this lock should be there and is entirely normal, as ArcMap is using the File Geodatabase, so there doesn't seem anything wrong here... No other "lock" files are in the File Geodatabase folder.

mboeringa commented 7 years ago

By the way, if instead of killing the ArcMap process, I kill the spawned Python processes, the progress dialog in ArcMap happily reports success for the step, and continues with the next step (that again gets stuck once the Python processes appear in the Task Manager). If I keep killing each spawned Python process, I eventually end up with a "Succeeded" message at the end of the tool run, with no apparent errors being displayed...

ThomasEmge commented 5 years ago

Please reopen, if the issue still exists.