Create Network Dataset hung on 'Populating Turn Features from OSM Relations' (ArcGIS 10.2.2 and other versions)

[borloa]

From about one month I am checking the possibility of creating large networks dataset. My goal is the creation of the graph for the whole Europe. To test the feasibility I started with small dataset on which I managed to get the result . In the last two weeks i worked on the elaboration of osm file between 8 and 10GB. I always get the same result : the network creation tool remains hung for hours on "Populating Turn Features from OSM Relations". Checking writing to disk the only file that appears to be written is the .lock file. After a few hours of waiting so I always decided to cancel the process (I got to let it run for 15 hours though apparently it was writing only on the lock file). I made a dozen attempts always getting the same result. I'm starting to think it is impossible to process so big files.

This is my procedure (I'm using foreground processing. 'Enable' Background Processing unflagged):

1. download the osm file from geoabrik (http://download.geofabrik.de/europe.html)
2. Create a new file gdb
3. Load OSM File tool
4. Create OSM Network Datataset tool (using DriveGeneric config file)

This is my actual enviroment (but i also tried 10.3 and 10.4 without 64 Bit Background Geoprocessing):

1. Windows 7 Professional 64bit (Intel Core 2.7GHz)
2. 500GB SSD disk
3. ArcGIS 10.2.2
4. ArcGIS_BackgroundGP_for_Desktop_1022_140426
5. ArcGISEditor10_2_1_x64

This is the last attempt log (Create OSM Network Datataset):

_Executing: OSMGPCreateNetworkDataset C:\data\fca_germany.gdb\ger "C:\Program Files (x86)\ArcGIS\Desktop10.2\ArcToolbox\Toolboxes\ND_ConfigFiles\DriveGeneric.xml" C:\data\fca_germany.gdb\ger\ger_nd Start Time: Thu Jun 09 12:15:22 2016 Extracting Edge Features OSMGPAttributeSelector_osmtools Executing: OSMGPAttributeSelector C:\data\fca_germany.gdb\ger\ger_osm_ln access;surface;smoothness;construct;proposed;maxheight;maxspeed;oneway;name Start Time: Thu Jun 09 12:15:22 2016 Adding field osm_access based on tag key access. Adding field osm_surface based on tag key surface. Adding field osm_smoothness based on tag key smoothness. Adding field osm_construct based on tag key construct. Adding field osm_proposed based on tag key proposed. Adding field osm_maxheight based on tag key maxheight. Adding field osm_maxspeed based on tag key maxspeed. Adding field osm_oneway based on tag key oneway. Adding field osm_name based on tag key name. useUpdateCursor: True Succeeded at Thu Jun 09 12:34:09 2016 (Elapsed Time: 18 minutes 46 seconds) Select_analysis Executing: Select C:\data\fca_germany.gdb\ger\ger_osm_ln C:\data\fca_germany.gdb\ger\ger_nd_roads "(highway IS NOT NULL) AND (LOWER(highway) in ('motorway','motorway_link','trunk','trunk_link','primary','primary_link','secondary','secondary_link','tertiary','tertiary_link','living_street','residential','unclassified','road'))" Start Time: Thu Jun 09 12:34:09 2016 Succeeded at Thu Jun 09 12:35:40 2016 (Elapsed Time: 1 minutes 30 seconds) Extracting Junction Features OSMGPAttributeSelector_osmtools Executing: OSMGPAttributeSelector C:\data\fca_germany.gdb\ger\ger_osm_pt access Start Time: Thu Jun 09 12:35:40 2016 Adding field osm_access based on tag key access. useUpdateCursor: True Succeeded at Thu Jun 09 17:49:41 2016 (Elapsed Time: 5 hours 14 minutes 0 seconds) Select_analysis Executing: Select C:\data\fca_germany.gdb\ger\ger_osm_pt C:\data\fca_germany.gdb\ger\ger_nd_barriers "(barrier IS NOT NULL) AND (LOWER(barrier) in ('block','bollard','chain','debris','jersey_barrier','lift_gate','log','spikes','swing_gate'))" Start Time: Thu Jun 09 17:49:41 2016 Succeeded at Thu Jun 09 17:58:21 2016 (Elapsed Time: 8 minutes 39 seconds) Assigning Network Connectivity Groups Extracting Turn Restrictions CreateTurnFeatureClass_na Executing: CreateTurnFeatureClass C:\data\fca_germany.gdb\ger ger_nd_turns 2 # # # # 0 0 0 DISABLED Start Time: Thu Jun 09 18:01:33 2016 Succeeded at Thu Jun 09 18:01:33 2016 (Elapsed Time: 0,21 seconds) AddField_management Executing: AddField C:\data\fca_germany.gdb\ger\ger_nd_turns RestrictionType TEXT # # 30 # NULLABLE NON_REQUIRED # Start Time: Thu Jun 09 18:01:33 2016 Adding RestrictionType to C:\data\fca_germany.gdb\ger\ger_ndturns... Succeeded at Thu Jun 09 18:01:33 2016 (Elapsed Time: 0,04 seconds) Populating Turn Features from OSM Relations

PS: I am not clear if I'm really using ArcGISEditor10_2_1_x64 . I only installed the 64bit version but I noticed that the sw is present at both of these locations:

• C : \ Program Files \ ESRI \ OSMEditor \ Bin
• C : \ Program Files ( x86 ) \ ESRI \ OSMEditor \ Bin

PS2: The 64bit version is only used with background processing enabled ?

I am going crazy! Any help is appreciated Alessandro

[GravitaZ]

From about one month I am checking the possibility of creating large networks dataset. My goal is the creation of the graph for the whole Europe. To test the feasibility I started with small dataset on which I managed to get the result . In the last two weeks i worked on the elaboration of osm file between 8 and 10GB. I always get the same result : the network creation tool remains hung for hours on "Populating Turn Features from OSM Relations". Checking writing to disk the only file that appears to be written is the .lock file. After a few hours of waiting so I always decided to cancel the process (I got to let it run for 15 hours though apparently it was writing only on the lock file). I made a dozen attempts always getting the same result. I'm starting to think it is impossible to process so big files.

This is my procedure (I'm using foreground processing. 'Enable' Background Processing unflagged):

1. download the osm file from geoabrik (http://download.geofabrik.de/europe.html)
2. Create a new file gdb
3. Load OSM File tool
4. Create OSM Network Datataset tool (using DriveGeneric config file)

This is my actual enviroment (but i also tried 10.3 and 10.4 without 64 Bit Background Geoprocessing):

1. Windows 7 Professional 64bit (Intel Core 2.7GHz)
2. 500GB SSD disk
3. ArcGIS 10.2.2
4. ArcGIS_BackgroundGP_for_Desktop_1022_140426
5. ArcGISEditor10_2_1_x64

This is the last attempt log (Create OSM Network Datataset):

_Executing: OSMGPCreateNetworkDataset C:\data\fca_germany.gdb\ger "C:\Program Files (x86)\ArcGIS\Desktop10.2\ArcToolbox\Toolboxes\ND_ConfigFiles\DriveGeneric.xml" C:\data\fca_germany.gdb\ger\ger_nd Start Time: Thu Jun 09 12:15:22 2016 Extracting Edge Features OSMGPAttributeSelector_osmtools Executing: OSMGPAttributeSelector C:\data\fca_germany.gdb\ger\ger_osm_ln access;surface;smoothness;construct;proposed;maxheight;maxspeed;oneway;name Start Time: Thu Jun 09 12:15:22 2016 Adding field osm_access based on tag key access. Adding field osm_surface based on tag key surface. Adding field osm_smoothness based on tag key smoothness. Adding field osm_construct based on tag key construct. Adding field osm_proposed based on tag key proposed. Adding field osm_maxheight based on tag key maxheight. Adding field osm_maxspeed based on tag key maxspeed. Adding field osm_oneway based on tag key oneway. Adding field osm_name based on tag key name. useUpdateCursor: True Succeeded at Thu Jun 09 12:34:09 2016 (Elapsed Time: 18 minutes 46 seconds) Select_analysis Executing: Select C:\data\fca_germany.gdb\ger\ger_osm_ln C:\data\fca_germany.gdb\ger\ger_nd_roads "(highway IS NOT NULL) AND (LOWER(highway) in ('motorway','motorway_link','trunk','trunk_link','primary','primary_link','secondary','secondary_link','tertiary','tertiary_link','living_street','residential','unclassified','road'))" Start Time: Thu Jun 09 12:34:09 2016 Succeeded at Thu Jun 09 12:35:40 2016 (Elapsed Time: 1 minutes 30 seconds) Extracting Junction Features OSMGPAttributeSelector_osmtools Executing: OSMGPAttributeSelector C:\data\fca_germany.gdb\ger\ger_osm_pt access Start Time: Thu Jun 09 12:35:40 2016 Adding field osm_access based on tag key access. useUpdateCursor: True Succeeded at Thu Jun 09 17:49:41 2016 (Elapsed Time: 5 hours 14 minutes 0 seconds) Select_analysis Executing: Select C:\data\fca_germany.gdb\ger\ger_osm_pt C:\data\fca_germany.gdb\ger\ger_nd_barriers "(barrier IS NOT NULL) AND (LOWER(barrier) in ('block','bollard','chain','debris','jersey_barrier','lift_gate','log','spikes','swing_gate'))" Start Time: Thu Jun 09 17:49:41 2016 Succeeded at Thu Jun 09 17:58:21 2016 (Elapsed Time: 8 minutes 39 seconds) Assigning Network Connectivity Groups Extracting Turn Restrictions CreateTurnFeatureClass_na Executing: CreateTurnFeatureClass C:\data\fca_germany.gdb\ger ger_nd_turns 2 # # # # 0 0 0 DISABLED Start Time: Thu Jun 09 18:01:33 2016 Succeeded at Thu Jun 09 18:01:33 2016 (Elapsed Time: 0,21 seconds) AddField_management Executing: AddField C:\data\fca_germany.gdb\ger\ger_nd_turns RestrictionType TEXT # # 30 # NULLABLE NON_REQUIRED # Start Time: Thu Jun 09 18:01:33 2016 Adding RestrictionType to C:\data\fca_germany.gdb\ger\ger_ndturns... Succeeded at Thu Jun 09 18:01:33 2016 (Elapsed Time: 0,04 seconds) Populating Turn Features from OSM Relations

PS: I am not clear if I'm really using ArcGISEditor10_2_1_x64 . I only installed the 64bit version but I noticed that the sw is present at both of these locations:

• C : \ Program Files \ ESRI \ OSMEditor \ Bin
• C : \ Program Files ( x86 ) \ ESRI \ OSMEditor \ Bin

PS2: The 64bit version is only used with background processing enabled ?

I am going crazy! Any help is appreciated Alessandro

Hello, I know this is old, but I'm having this issue too, it is only going up like 1% every two days, this is for about 6 states in the Eastern US. Did you ever find a resolution?

[robblis]

Hi GravitaZ,

Here's some steps to try to help the performance.

1. Test on a smaller area first to see if the toolsets themselves are in working order. If successful, then for the larger set do the following

2. Extract the required attributes first - based on step 1 one should be able to figure out the required attributes.

3. Build indices on these attributes.

4. Then run the create network tools