No edges

[bjtaylor1]

Hi, Just got the latest code and latest OSM data to update my routing servers. Seems to run ok on my own home machine, which is a laptop running ubuntu 14.

On my laptop osrm-prepare outputs:

ben@ben-Ideapad-Z560:~/Project-OSRM$ ./osrm-prepare -p auk.lua great-britain-latest.osrm
[info] Input file: great-britain-latest.osrm
[info] Restrictions file: great-britain-latest.osrm.restrictions
[info] Profile: auk.lua
[info] Threads: 4
[info] Importing n = 12223401 nodes 
[info]  and 12768379 edges 
[info] Graph loaded ok and has 12768193 edges

..etc

But on my amazon server which is also ubuntu 14 osrm-prepare outputs:

ubuntu@ip-172-31-34-241:~/Project-OSRM$ ./osrm-prepare -p auk.lua great-britain-latest.osrm
[info] Input file: great-britain-latest.osrm
[info] Restrictions file: great-britain-latest.osrm.restrictions
[info] Profile: auk.lua
[info] Threads: 2
[info] Importing n = 12223401 nodes 
[info]  and 0 edges 
[info] Graph loaded ok and has 0 edges
[warn] The input data is empty, exiting.

(the last line is on stderr in red, and the exit code is 1). I presume the fact that it managed to find 0 edges as opposed to 12768379 is part of the problem?

Why could this be?

The great-britain-latest.osrm that they are using is not the same, on the server (which fails) it is 513,642,494 bytes in size but on my laptop (which is ok) it is slightly bigger at 565,857,571.

The only issue I did have updating the server was that it complained that it couldn't find something like "libboost_filesystem_1.46.so" so I got the latest boost 1.56, built it from source, installed it with "./bootstrap && ./b2 install", rebuilt OSRM, and then that error went away. I think my laptop is on boost 1.55 but still built from source.

Any ideas as to how I can track down this issue? The lua profile I am using is below in case it matters. But can't think it's an issue with that - otherwise it would surely not work on my laptop?

-- Begin of globals
require("lib/access")
require("lib/whitelist")
require("lib/blacklist")

barrier_whitelist = { ["cattle_grid"] = true, ["border_control"] = true, ["toll_booth"] = true, ["sally_port"] = true, ["gate"] = true, ["no"] = true, ["entrance"] = true}
access_tag_whitelist = { ["yes"] = true, ["motorcar"] = true, ["motor_vehicle"] = true, ["vehicle"] = true, ["permissive"] = true, ["designated"] = true  }
access_tag_blacklist = { ["no"] = true, ["private"] = true, ["agricultural"] = true, ["forestry"] = true, ["emergency"] = true }
access_tag_restricted = { ["destination"] = true, ["delivery"] = true }
access_tags = { "motorcar", "motor_vehicle", "vehicle" }
access_tags_hierachy = { "motorcar", "motor_vehicle", "vehicle", "access" }
service_tag_restricted = { ["parking_aisle"] = true }
ignore_in_grid = { ["ferry"] = true }
restriction_exception_tags = { "motorcar", "motor_vehicle", "vehicle" }
default_speed = 17
speed_trunk = default_speed 
speed_primary = default_speed
speed_secondary = default_speed
speed_tertiary = default_speed
speed_unclassified = default_speed 
speed_residential = default_speed 
speed_service = default_speed

myspeed_profile = {
  ["trunk"] = speed_trunk,
  ["trunk_link"] = speed_trunk,
  ["primary"] = speed_primary,
  ["primary_link"] = speed_primary,
  ["secondary"] = speed_secondary,
  ["secondary_link"] = speed_secondary,
  ["tertiary"] = speed_tertiary,
  ["tertiary_link"] = speed_tertiary,
  ["unclassified"] = speed_unclassified,
  ["residential"] = speed_residential,
  ["living_street"] = speed_residential,
  ["service"] = speed_service,
--  ["track"] = 5,
  ["ferry"] = 5,
  ["shuttle_train"] = 10,
  ["default"] = default_speed
}

take_minimum_of_speeds  = false
obey_oneway                   = true
obey_bollards           =  false
use_restrictions            = true
ignore_areas                  = true -- future feature
traffic_signal_penalty  = 60
u_turn_penalty              = 20

-- End of globals

function get_exceptions(vector)
    for i,v in ipairs(restriction_exception_tags) do
        vector:Add(v)
    end
end

local function parse_maxspeed(source)
    if source == nil then
        return 0
    end
    local n = tonumber(source:match("%d*"))
    if n == nil then
        n = 0
    end
    if string.match(source, "mph") or string.match(source, "mp/h") then
        n = (n*1609)/1000;
    end
    return math.abs(n)
end

function node_function (node)
  local barrier = node.tags:Find("barrier")
  local access = Access.find_access_tag(node, access_tags_hierachy)
  local traffic_signal = node.tags:Find("highway")

  --flag node if it carries a traffic light

  if traffic_signal == "traffic_signals" then
    node.traffic_light = true;
  end

    -- parse access and barrier tags
    if access  and access ~= "" then
        if access_tag_blacklist[access] then
            node.bollard = true
        end
    elseif barrier and barrier ~= "" then
        if barrier_whitelist[barrier] then
            return
        else
            node.bollard = true
        end
    end
end

function way_function (way)
  local thespeed = -1

--special ways override everything else and we can go max (unclassified) speed on them
  if(Whitelist.whitelist_ways_by_id[way.id]) then
    way.special = true
    way.specialtext = "whitelist"
    way.speed = speed_unclassified
    way.type = 1
    return
  end

  if(Blacklist.blacklist_ways_by_id[way.id]) then
    way.special = true
    way.specialtext = "blacklist"
    return
  end

  -- we dont route over areas
  local area = way.tags:Find("area")
  if ignore_areas and ("yes" == area) then
    return
  end

  -- check if oneway tag is unsupported
  local oneway = way.tags:Find("oneway")
  if "reversible" == oneway then
    return
  end

  local impassable = way.tags:Find("impassable")
  if "yes" == impassable then
    return
  end

  local status = way.tags:Find("status")
  if "impassable" == status then
    return
  end

  -- Check if we are allowed to access the way
  local access = Access.find_access_tag(way, access_tags_hierachy)
  if access_tag_blacklist[access] then
    return
  end

  -- Second, parse the way according to these properties
  local highway = way.tags:Find("highway")

  if "motorway" == highway then
    return
  end

  if "motorway_junction" == highway then
    return
  end
  local name = way.tags:Find("name")
  local ref = way.tags:Find("ref")
  local junction = way.tags:Find("junction")
  local route = way.tags:Find("route")
  local maxspeed = math.max(parse_maxspeed(way.tags:Find ( "maxspeed") ), default_speed)
  local maxspeed_forward = math.max(parse_maxspeed(way.tags:Find( "maxspeed:forward")), default_speed)
  local maxspeed_backward = parse_maxspeed(way.tags:Find( "maxspeed:backward"))
  local barrier = way.tags:Find("barrier")
  local cycleway = way.tags:Find("cycleway")
  local duration  = way.tags:Find("duration")
  local service  = way.tags:Find("service")

  if maxspeed >= 112 and "roundabout" ~= junction then --112 = 70 mph
    return
  end

  -- Set the name that will be used for instructions
    if "" ~= ref then
      way.name = ref
    elseif "" ~= name then
      way.name = name
--  else
--      way.name = highway      -- if no name exists, use way type
    end

    if "roundabout" == junction then
      way.roundabout = true;
    end

  -- Handling ferries and piers
  if (myspeed_profile[route] ~= nil and myspeed_profile[route] > 0) then
    if durationIsValid(duration) then
      way.duration = math.max( parseDuration(duration), 1 );
    end
    way.direction = Way.bidirectional
    if myspeed_profile[route] ~= nil then
      highway = route;
    end
    if tonumber(way.duration) < 0 then
      thespeed = myspeed_profile[highway]
    end
  end

  -- Set the avg speed on the way if it is accessible by road class
  if (myspeed_profile[highway] ~= nil and thespeed == -1 ) then
      thespeed = myspeed_profile[highway]
  end

  -- Set the avg speed on ways that are marked accessible
  if "" ~= highway and access_tag_whitelist[access] and thespeed == -1 then
    if 0 == maxspeed then
      maxspeed = math.huge
    end
    thespeed = math.min(myspeed_profile["default"], maxspeed)
  end

  -- Set access restriction flag if access is allowed under certain restrictions only
  if access ~= "" and access_tag_restricted[access] then
    way.is_access_restricted = true
  end

  -- Set access restriction flag if service is allowed under certain restrictions only
  if service ~= "" and service_tag_restricted[service] then
      way.is_access_restricted = true
  end

  -- Set direction according to tags on way
  way.direction = Way.bidirectional
  if obey_oneway  then
      if oneway == "-1" then
        way.direction = Way.opposite
    elseif oneway == "yes" or
      oneway == "1" or
      oneway == "true" or
      junction == "roundabout" or
      (highway == "motorway_link" and oneway ~="no") or
      (highway == "motorway" and oneway ~= "no")
      then
         way.direction = Way.oneway
    end
  end

  -- Override speed settings if explicit forward/backward maxspeeds are given
  if thespeed > 0 and maxspeed_forward ~= nil and maxspeed_forward > 0 then
    if Way.bidirectional == way.direction then
      way.backward_speed = thespeed
    end
    thespeed = math.max(maxspeed_forward, default_speed)
  end
  if maxspeed_backward ~= nil and maxspeed_backward > 0 then
    way.backward_speed = math.max(maxspeed_backward, default_speed)
  end

  -- Override general direction settings of there is a specific one for our mode of travel
  if ignore_in_grid[highway] ~= nil and ignore_in_grid[highway] then
        way.ignore_in_grid = true
    end
    way.type = 1
  way.speed = thespeed
  return
end

-- These are wrappers to parse vectors of nodes and ways and thus to speed up any tracing JIT

function node_vector_function(vector)
 for v in vector.nodes do
  node_function(v)
 end
end

function turn_function (angle)
  return 5
end

[bjtaylor1]

The output of ./osrm-extract on the server was:

[info] Input file: great-britain-latest.osm.pbf
[info] Profile: auk.lua
[info] Threads: 2
[info] Using script auk.lua
[STXXL-MSG] STXXL v1.3.1 (release)
[STXXL-MSG] 1 disks are allocated, total space: 1000 MiB
[info] Using turn restrictions
[info] Found 3 exceptions to turn restrictions:
[info]   motorcar
[info]   motor_vehicle
[info]   vehicle
[info] Parsing in progress..
[info] Parsing finished after 605.008 seconds
[extractor] Sorting used nodes        ... ok, after 5.446s
[extractor] Erasing duplicate nodes   ... ok, after 1.155s
[extractor] Sorting all nodes         ... ok, after 50.023s
[extractor] Sorting used ways         ... ok, after 3.088s
[extractor] Sorting restrictions. by from... ok, after 0.143s
[extractor] Fixing restriction starts ... ok, after 0.431s
[extractor] Sorting restrictions. by to  ... ok, after 0.147s
[extractor] Fixing restriction ends   ... ok, after 0.056s
[info] usable restrictions: 5421
[extractor] Confirming/Writing used nodes     ... ok, after 44.506s
[extractor] setting number of nodes   ... ok
[extractor] Sorting edges by start    ... ok, after 38.712s
[extractor] Setting start coords      ... ok, after 69.102s
[extractor] Sorting edges by target   ... ok, after 34.208s
[extractor] Setting target coords     ... ok, after 85.628s
[extractor] setting number of edges   ... ok
[extractor] writing street name index ... ok, after 0.056s
[info] Processed 12223401 nodes and 12768379 edges
[info] extraction finished after 938.011s
[info] To prepare the data for routing, run: ./osrm-prepare great-britain-latest.osrm

so I think it is an issue with prepare not reading them, rather than extract not creating them, as extract claims to have written them...

[DennisOSRM]

Looks like you ran out of disk space. Make sure /tmp has sufficient space. Alternatively use the latest code from develop branch which does not use /tmp anymore.

[bjtaylor1]

ok i'll give that a go. cheers

[bjtaylor1]

So I presume "develop" is the bleeding edge but less stable then? At what point do you tend to merge "develop" into main ? Any reasons why I might not want to use develop?

[DennisOSRM]

Develop is mostly stable. We are using it to power the demo site. As such there is quite some effort to make it as stable possible. It is merged into master branch once it has accumulated enough new features to justify a new release number.

For experiments and personal use develop branch is pretty alright. For production sites master is more tested.

[bjtaylor1]

ah. I seem to run into https://github.com/Project-OSRM/osrm-backend/issues/694 using develop. resolution seems to be to make sure profile.lua is a symlink to profiles/car.lua but still does it with that.

[bjtaylor1]

btw the build assembles the binaries into "Project-OSRM/build", I then copy them back one into "Project-OSRM" by doing "cp build/* ./", and run from there - is that the right way to do it?

[DennisOSRM]

I'd just symlink:

ln -s build/osrm-*