OSM Create Network Dataset: Populating Turn Features from OSM Relations The row contains a bad value. [RestrictionType]

[chenna997]

Dear all, I am trying to execute "Create OSM Network Dataset" in ArcGIS 10.5. Most steps Succeeded until "Populating Turn Features from OSM Relations". It would be highly appreciated if any help is provided. I am attaching the processing information as follows:

Executing: OSMGPCreateNetworkDataset "C:\Users\chennn\OneDrive\Research\Green Umbrella\GIS\OSM.gdb\OSM_OKI2" "C:\Program Files (x86)\ArcGIS\Desktop10.5\ArcToolbox\Toolboxes\ND_ConfigFiles\CycleGeneric.xml" "C:\Users\chennn\OneDrive\Research\Green Umbrella\GIS\OSM.gdb\OSM_OKI2\OSM_OKI2_nd" Start Time: Mon Apr 16 14:33:35 2018 Extracting Edge Features Select_analysis Executing: Select "C:\Users\chennn\OneDrive\Research\Green Umbrella\GIS\OSM.gdb\OSM_OKI2\OSM_OKI2_osm_ln" "C:\Users\chennn\OneDrive\Research\Green Umbrella\GIS\OSM.gdb\OSM_OKI2\OSM_OKI2_nd_roads" "(highway IS NOT NULL) AND (LOWER(highway) in ('primary', 'primary_link', 'secondary', 'secondary_link', 'tertiary', 'tertiary_link', 'unclassified', 'residential', 'living_street', 'service', 'track', 'cycleway', 'footway', 'bridleway', 'pedestrian', 'path'))" Start Time: Mon Apr 16 14:33:36 2018 Succeeded at Mon Apr 16 14:35:05 2018 (Elapsed Time: 1 minutes 28 seconds) Extracting Junction Features Select_analysis Executing: Select "C:\Users\chennn\OneDrive\Research\Green Umbrella\GIS\OSM.gdb\OSM_OKI2\OSM_OKI2_osm_pt" "C:\Users\chennn\OneDrive\Research\Green Umbrella\GIS\OSM.gdb\OSM_OKI2\OSM_OKI2_nd_barriers" "(barrier IS NOT NULL) AND (LOWER(barrier) in ('block','bollard','chain','debris','jersey_barrier','lift_gate','log','spikes','swing_gate'))" Start Time: Mon Apr 16 14:35:05 2018 Succeeded at Mon Apr 16 14:39:32 2018 (Elapsed Time: 4 minutes 27 seconds) Assigning Network Connectivity Groups Extracting Turn Restrictions CreateTurnFeatureClass_na Executing: CreateTurnFeatureClass "C:\Users\chennn\OneDrive\Research\Green Umbrella\GIS\OSM.gdb\OSM_OKI2" OSM_OKI2_nd_turns 2 # # # # 0 0 0 DISABLED Start Time: Mon Apr 16 14:44:44 2018 Succeeded at Mon Apr 16 14:44:45 2018 (Elapsed Time: 0.58 seconds) AddField_management Executing: AddField "C:\Users\chennn\OneDrive\Research\Green Umbrella\GIS\OSM.gdb\OSM_OKI2\OSM_OKI2_nd_turns" RestrictionType TEXT # # 30 # NULLABLE NON_REQUIRED # Start Time: Mon Apr 16 14:44:45 2018 Adding RestrictionType to C:\Users\chennn\OneDrive\Research\Green Umbrella\GIS\OSM.gdb\OSM_OKI2\OSM_OKI2_nd_turns... Succeeded at Mon Apr 16 14:44:45 2018 (Elapsed Time: 0.19 seconds) Populating Turn Features from OSM Relations The row contains a bad value. [RestrictionType] at ESRI.ArcGIS.Geodatabase.IFeature.Store() at ESRI.ArcGIS.OSM.GeoProcessing.NetworkTurns.CreateTurnFeature_NO(TurnFeatureClassWrapper turnFCW, OSMTurnInfo osmTurn) at ESRI.ArcGIS.OSM.GeoProcessing.NetworkTurns.PopulateTurnsFromRelations() at ESRI.ArcGIS.OSM.GeoProcessing.RunTaskManager.ExecuteTask(String messageName, Action task) at ESRI.ArcGIS.OSM.GeoProcessing.NetworkTurns.ExtractTurnRestrictions() at ESRI.ArcGIS.OSM.GeoProcessing.NetworkDataset.ExtractTurnRestrictions() at ESRI.ArcGIS.OSM.GeoProcessing.RunTaskManager.ExecuteTask(String messageName, Action task) at ESRI.ArcGIS.OSM.GeoProcessing.NetworkDataset.CreateNetworkDataset() at ESRI.ArcGIS.OSM.GeoProcessing.OSMGPCreateNetworkDataset.Execute(IArray paramvalues, ITrackCancel TrackCancel, IGPEnvironmentManager envMgr, IGPMessages message) Failed to execute (OSMGPCreateNetworkDataset). Failed at Mon Apr 16 15:48:47 2018 (Elapsed Time: 1 hours 15 minutes 11 seconds)

I tried to delete this relation table, but still could not solve the issue. Thank you so much for your attention!

--Na

[Chaz6]

I am experiencing a very similar issue:-

Executing: OSMGPCreateNetworkDataset C:\Users\Chris.Hills\Documents\ArcGIS\Default.gdb\NetworkDataset2 "C:\Program Files (x86)\ArcGIS\Desktop10.5\ArcToolbox\Toolboxes\ND_ConfigFiles\DriveGeneric.xml" C:\Users\Chris.Hills\Documents\ArcGIS\Default.gdb\NetworkDataset2\NetworkDataset2_nd
Start Time: Wed Apr 25 12:14:42 2018
Extracting Edge Features
OSMGPAttributeSelector_osmtools
Executing: OSMGPAttributeSelector C:\Users\Chris.Hills\Documents\ArcGIS\Default.gdb\NetworkDataset2\NetworkDataset2_osm_ln access;surface;smoothness;construct;proposed;maxheight;maxspeed;oneway;name
Start Time: Wed Apr 25 12:14:43 2018
Adding field osm_access based on tag key access.
Adding field osm_surface based on tag key surface.
Adding field osm_smoothness based on tag key smoothness.
Adding field osm_construct based on tag key construct.
Adding field osm_proposed based on tag key proposed.
Adding field osm_maxheight based on tag key maxheight.
Adding field osm_maxspeed based on tag key maxspeed.
Adding field osm_oneway based on tag key oneway.
Adding field osm_name based on tag key name.
useUpdateCursor: False
Succeeded at Wed Apr 25 12:14:48 2018 (Elapsed Time: 4.61 seconds)
Select_analysis
Executing: Select C:\Users\Chris.Hills\Documents\ArcGIS\Default.gdb\NetworkDataset2\NetworkDataset2_osm_ln C:\Users\Chris.Hills\Documents\ArcGIS\Default.gdb\NetworkDataset2\NetworkDataset2_nd_roads "(highway IS NOT NULL) AND (LOWER(highway) in ('motorway','motorway_link','trunk','trunk_link','primary','primary_link','secondary','secondary_link','tertiary','tertiary_link','living_street','residential','unclassified','road'))"
Start Time: Wed Apr 25 12:14:48 2018
Succeeded at Wed Apr 25 12:14:53 2018 (Elapsed Time: 5.03 seconds)
Extracting Junction Features
OSMGPAttributeSelector_osmtools
Executing: OSMGPAttributeSelector C:\Users\Chris.Hills\Documents\ArcGIS\Default.gdb\NetworkDataset2\NetworkDataset2_osm_pt access
Start Time: Wed Apr 25 12:14:53 2018
Adding field osm_access based on tag key access.
useUpdateCursor: False
Succeeded at Wed Apr 25 12:14:57 2018 (Elapsed Time: 3.66 seconds)
Select_analysis
Executing: Select C:\Users\Chris.Hills\Documents\ArcGIS\Default.gdb\NetworkDataset2\NetworkDataset2_osm_pt C:\Users\Chris.Hills\Documents\ArcGIS\Default.gdb\NetworkDataset2\NetworkDataset2_nd_barriers "(barrier IS NOT NULL) AND (LOWER(barrier) in ('block','bollard','chain','debris','jersey_barrier','lift_gate','log','spikes','swing_gate'))"
Start Time: Wed Apr 25 12:14:57 2018
Succeeded at Wed Apr 25 12:15:02 2018 (Elapsed Time: 4.92 seconds)
Assigning Network Connectivity Groups
Extracting Turn Restrictions
Exception from HRESULT: 0x80040353
   at ESRI.ArcGIS.Geodatabase.IDatasetContainer2.CreateDataset(IDEDataset DataElement)
   at ESRI.ArcGIS.OSM.GeoProcessing.NetworkDataset.CreateTempNDS()
   at ESRI.ArcGIS.OSM.GeoProcessing.NetworkDataset.ExtractTurnRestrictions()
   at ESRI.ArcGIS.OSM.GeoProcessing.RunTaskManager.ExecuteTask(String messageName, Action task)
   at ESRI.ArcGIS.OSM.GeoProcessing.NetworkDataset.CreateNetworkDataset()
   at ESRI.ArcGIS.OSM.GeoProcessing.OSMGPCreateNetworkDataset.Execute(IArray paramvalues, ITrackCancel TrackCancel, IGPEnvironmentManager envMgr, IGPMessages message)
Failed to execute (OSMGPCreateNetworkDataset).
Failed at Wed Apr 25 12:15:03 2018 (Elapsed Time: 21.04 seconds)

[Chaz6]

The first time I tried using the tool "Download, Extract and Symbolize OSM Data". The second time I used "Download OSM Data" and it works. Perhaps the two tools behave slightly differently with regards to the routing network.

[chenna997]

Thanks for your response, Chris! I just tried "Download OSM Data" based on your information, and it shows that I requested too many nodes and asked to reduce the area size. I am trying to get the OSM data for a county. Did you have any experience for this issue? Thanks!

[mboeringa]

@chenna997

For big extracts, like a county/state/country, I really recommend you to use the Geofabrik "Download" option (http://www.geofabrik.de/ for main page and direct link to download is here: http://download.geofabrik.de/).

The Geofabrik company offers free extracts for many parts of the globe, which contain the full data if downloaded as either .pbf or .xml files. They also offer paid shapefile data, but that data is processed in their custom schema, and you thus loose flexibility.

BIG TIP: when on the download page, you initially only see the big continent extracts listed, however, these are offered as hyperlinks. Once you click a hyperlink, you drill down to a lower administrative level, e.g. clicking "North America" will open a new list with all American states, and so on if there are more levels for a specific region.

I therefor recommend you to download the XML files. Note that you need the XML files, as PBF (which is a more efficient / compact storage method for exactly the same data) is not supported by the import tools of the Editor.

So what to do with the XML?: you can simply import those with the Editor's import tools. There are two that support the *.xml files:

• Load OSM File will load all data, including all OSM relation information necessary if you intend to make edits to any part of the data and intend to post this back to OSM using the tool included for that in the toolbox (Upload OSM Data). Do take note though, that if you really intend to make edits, it is NOT recommended to first download an entire country! Just load a small amount of data (for which the Geofabrik extracts have no use, as they are big). However, if you just need to access the data, loading an entire country is fine.

• The preferred way though, to load an entire country, is using the new OSM File Loader (Load Only) tool. This tool again takes XML files as input, however, in contrast to the above tool, doesn't load data necessary to make edits. You do get all geographic data though, so don't worry! This tool has the advantage of supporting the ArcGIS Parallel Processing environment setting (http://desktop.arcgis.com/en/arcmap/latest/tools/environments/parallel-processing-factor.htm), meaning it can load data in parallel on multiple CPU cores, and the data load will thus be about 3 times faster than with the Load OSM File tool.

[chenna997]

Thanks so much for the detailed information. I was traveling in the past weeks and will try this you recommended.

Best,

-Na Chen

On Sat, Apr 28, 2018 at 5:32 PM mboeringa notifications@github.com wrote:

@chenna997 https://github.com/chenna997

For big extracts, like a county/state/country, I really recommend you to use the Geofabrik "Download" option (http://www.geofabrik.de/ for main page and direct link to download is here: http://download.geofabrik.de/).

The Geofabrik company offers free extracts for many parts of the globe, which contain the full data if downloaded as either .pbf or .xml files. They also offer paid shapefile data, but that data is processed in their custom schema, and you thus loose flexibility.

BIG TIP: when on the download page, you initially only see the big continent extracts listed, however, these are offered as hyperlinks. Once you click a hyperlink, you drill down to a lower administrative level, e.g. clicking "North America" will open a new list with all American states, and so on if there are more levels for a specific region.

I therefor recommend you to download the XML files. Note that you need the XML files, as PBF (which is a more efficient / compact storage method for exactly the same data) is not supported by the import tools of the Editor.

So what to do with the XML?: you can simply import those with the Editor's import tools. There are two that support the *.xml files:

-

Load OSM File will load all data, including all OSM relation information necessary if you intend to make edits to any part of the data and intend to post this back to OSM using the tool included for that in the toolbox (Upload OSM Data). Do take note though, that if you really intend to make edits, it is NOT recommended to first download an entire country! Just load a small amount of data (for which the Geofabrik extracts have no use, as they are big). However, if you just need to access the data, loading an entire country is fine.

The preferred way though, to load an entire country, is using the new OSM File Loader (Load Only) tool. This tool again takes XML files as input, however, in contrast to the above tool, doesn't load data necessary to make edits. You do get all geographic data though, so don't worry! This tool has the advantage of supporting the ArcGIS Parallel Processing environment setting ( http://desktop.arcgis.com/en/arcmap/latest/tools/environments/parallel-processing-factor.htm), meaning it can load data in parallel on multiple CPU cores, and the data load will thus be about 3 times faster than with the Load OSM File tool.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/Esri/arcgis-osm-editor/issues/192#issuecomment-385158477, or mute the thread https://github.com/notifications/unsubscribe-auth/AkqMakGowb-5-BfEIM69ux-csAf4enAmks5ttDclgaJpZM4TXPSi .

[mboeringa]

@chenna997 : just one extra note:

In case you intend to create a Network Dataset, you must use the Load OSM File tool and not the OSM File Loader (Load Only), as the creation of Network Datasets requires some of the extra relationship info loaded by the former tool, but not the latter.

However, if you sole purpose is display / styling / rendering of the data, then the OSM File Loader (Load Only) is the preferred tool to load.

[Chaz6]

I tried using the OSM File Loader (Load Only) as I want to build a routing network, but it failed after 2 hours (on a 2m point/300k way osm file):-

Updating remaining references...
The given key was not present in the dictionary.
   at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
   at ESRI.ArcGIS.OSM.GeoProcessing.OSMGPFileLoader.Execute(IArray paramvalues, ITrackCancel TrackCancel, IGPEnvironmentManager envMgr, IGPMessages message)
Failed to execute (OSMGPFileLoader).

[mboeringa]

@Chaz6 , as I wrote in my last post, for things like building a Network Dataset and routing networks, using the Load OSM File tool is necessary, as the faster parallel processing tool OSM File Loader (Load Only) does not load all required relation information, so you must switch tools.

As to the specific error during import regarding the OSM File Loader (Load Only) tool, I have no real clue. What was the source of this dataset? Did you download a full Geofabrik extract, or create some custom *.osm XML file, that may be lacking some info, with another non-ArcGIS open source tool?

Anyway, I am just a user of this toolbox like you. Maybe @ThomasEmge has something more specific to say about the exact error you posted.

[Chaz6]

Thanks I must have gotten myself confused, and I will give that a try. I used an country level file from Geofabrik which i clipped to a region using a *.poly file and osmconvert (but as far as I am aware, it retains all the node/way/relationship data as I did not do any other operation on it).

[Chaz6]

I double-checked and I was using the Load OSM File tool. I tried again, this time using background geoprocessing, and this time it worked!

[Chaz6]

I got as far as building the route network, and I ran into the same issue as before. I changed the config file to DriveMeters.xml which let it get further than before, but then ArcMap crashed with this message...

[Window Title]
ArcMap

[Main Instruction]
ArcMap has stopped working

[Content]
A problem caused the program to stop working correctly. Please close the program.

[Close the program]

[doggett-sarah]

i have the same problem

[MatthewHudnall]

Same issue here. I am trying to hunt down the issue, but not having any luck so far. The RestrictionTypes are contained within a Blob entry (probably XML or JSON encoded) within the osmTags field in the YourDBName_osm_ln shapefile. The field is hidden though by default, so you have to unhide it by selecting "Turn All Field On" in the attribute table. Anyway, this field is not editable in ArcMap due to the Blob data. So I have been removing restrictions in OSM prior to import. In other cases I have been deleting features in the YourDBName_osm_ln shapefile to remove segments with restrictions. This is NOT ideal though as you lose roadways doing that. Based on the error, "The row contains a bad value. [RestrictionType]", I am trying to hunt down the restriction types that do not work. Ideally though the tool needs to be modified to just skip erroneous RestrictionTypes. The data loaded from the hidden osmTags blob field in YourDBName_osm_ln is decoded and loaded into YourDBName_nd_trurns in the RestrictionType field. The tool is dying when it tries to load an invalid value from the osmTags blob into the RestrictionType field in YourDBName_nd_trurns. The field is of type string with a length of 30 (nulls allowed), so the likely issue is that a RestrictionType that exceeds 30 characters is trying to come in from the OSM data. Time to find that bad value in OSM, or fix the tool to handle the exception...

[MatthewHudnall]

My solution is to cap TurnType (which is where RestrictionType gets stored) in NetworkTurns.cs:

private string _turnType; public string TurnType { get { return this._turnType; } set { this._turnType = value.Substring(0, Math.Min(value.Length, 30)); } }

I also did a try/catch around the store command:

        try
        {
            turn.Store();
        }
        catch (Exception ex)
        {
            System.Diagnostics.Debug.WriteLine(ex.Message);
            System.Diagnostics.Debug.WriteLine(ex.StackTrace);
        }

Anything that gets stored with the above into TurnType will then automatically get truncated to 30 characters. New code works so far, but I have been at 100% and "Populating turn features from OSM relations" for like 15 hours now...

[ThomasEmge]

What is the extent of your geographical area?

[MatthewHudnall]

36.96,-91.63 (upper left) to 24.54,-80.03 (lower right), which is Tennessee, Mississippi, Alabama, Georgia, and Florida.

[chenna997]

@chenna997 : just one extra note:

In case you intend to create a Network Dataset, you must use the Load OSM File tool and not the OSM File Loader (Load Only), as the creation of Network Datasets requires some of the extra relationship info loaded by the former tool, but not the latter.

However, if you sole purpose is display / styling / rendering of the data, then the OSM File Loader (Load Only) is the preferred tool to load.

Hi, after couple of months, I am picking up this task again to try to create the Network Dataset using the OSM data. Following your suggestion, I have successfully downloaded and loaded the whole state OSM data. However, when I tried to create Network Dataset using the loaded data, I got the same error: "Populating Turn Features from OSM Relations", The row contains a bad value. [RestrictionType]. Is there any solution for this error? Thanks so much!

[ThomasEmge]

Fixed with #201 .

[Chaz6]

Thank you.

[GravitaZ]

@ThomasEmge Has this bug been updated in all the installers. I am still having issues with 10.3