itinero / routing

The routing core of itinero.
Apache License 2.0
221 stars 69 forks source link

The index out of range exception has been thrown when I try to build the whole US router db. #214

Closed Casper-ThinkGeo closed 5 years ago

Casper-ThinkGeo commented 6 years ago

Hi, I'm working on building the whole US router db. I try to get the router db file with two approaches but it works fail:

  1. Use the whole US data with one North America .pbf (7.8G). I have download the North America data from https://download.geofabrik.de/ . I try to build the router db with following statements: `var usPbfFile = @"C:\Test\north-america-latest.osm.pbf"; using (var stream = File.OpenRead(usPbfFile)) { var osmStream = new OsmSharp.Streams.PBFOsmStreamSource(stream); var routerDb = new RouterDb(); routerDb.LoadOsmData(osmStream);

    if (!routerDb.HasContractedFor(vehicle.Fastest())) { Itinero.Logging.Logger.Log("RouterDbBuilder", Itinero.Logging.TraceEventType.Information, "No contracted graph found for the 'car' profile, building now..."); routerDb.AddContracted(vehicle.Fastest(), true); }

    using (var writeStream = File.Open(routerDbFileName, FileMode.Create)) { routerDb.Serialize(writeStream); } } ` It works failed and throws an index out of range exception when the router db calculating progress is 97% as the attached img_23072018_125337_0_north_america

  2. Use the divided state .pbf files. I have downloaded all of .pbf files in America, and build them into one router db file as following statements: ` public static RouterDb Build(string sourceDbFolder, string targetDbFilename, Vehicle vehicle) { RouterDb routerDb = null; var routerDbFileName = targetDbFilename; if (File.Exists(routerDbFileName)) { try { using (var stream = File.OpenRead(routerDbFileName)) { routerDb = RouterDb.Deserialize(stream); } } catch { routerDb = null; } }

        if (routerDb == null)
        {
            // check if OSM pbf file is there.
            var files = Directory.GetFiles(sourceDbFolder);
            if (files == null || files.Length <= 0)
            { // check if OSM file is there, otherwise attempt to download from overpass.
                throw new Exception("The .pbf file is not exist.");
            }
    
            // build routerdb.
            Itinero.Logging.Logger.Log("RouterDbBuilder", Itinero.Logging.TraceEventType.Information, "No existing RouterDb file found, creating now.");
    
            var osmFiles = new List<OsmStreamSource>();
            var fileStreams = new List<Stream>();
    
            foreach (var file in files)
            {
                OsmStreamSource osmStream = null;
                var stream = File.OpenRead(file);
                fileStreams.Add(stream);
                if (file.EndsWith(".osm.pbf"))
                {
                    osmStream = new OsmSharp.Streams.PBFOsmStreamSource(stream);
                }
                else
                {
                    osmStream = new OsmSharp.Streams.XmlOsmStreamSource(stream);
                }
    
                osmFiles.Add(osmStream);
            }
    
            routerDb = new RouterDb();
            routerDb.LoadOsmData(osmFiles.ToArray(), vehicle);
    
            foreach (var stream in fileStreams)
            {
                stream.Close();
                stream.Dispose();
            }
            Itinero.Logging.Logger.Log("RouterDbBuilder", Itinero.Logging.TraceEventType.Information, "RouterDb file created.");
    
            if (!routerDb.HasContractedFor(vehicle.Fastest()))
            {
                Itinero.Logging.Logger.Log("RouterDbBuilder", Itinero.Logging.TraceEventType.Information, "No contracted graph found for the 'car' profile, building now...");
                routerDb.AddContracted(vehicle.Fastest(), true);
            }
    
            using (var stream = File.Open(routerDbFileName, FileMode.Create))
            {
                routerDb.Serialize(stream);
            }
        }
        return routerDb;
    }

    ` It works failed and throws an index out of range exception when the router db is being written with a result data as the attached file: image-writting

I don't know how to fix this issue. I'm very appreciate for your any help. My computer configuration: RAM: 32G, CPU: Core i7 4790 OS: Win10 X64 Application: 64 bit.

xivk commented 6 years ago

Did you try this using the latest prerelease? We're focusing on getting that released.

I tested building the north-america routerdb and that worked, now testing the contraction...

Casper-ThinkGeo commented 6 years ago

Hi xivk, Thanks for your help, I'm working on the version v1.4.0-pre67. As the above statements, a "Car fastest" contraction has been added to RouterDb: routerDb.AddContracted(vehicle.Fastest(), true);

This exception has been thrown when the router db is written to file with the "Car fastest" contraction. The thrown point is located on the "Compress" method in "DirectedGraph.cs".

If you have any updates, please tell me know. I'm very appreciate for your help.

Casper-ThinkGeo commented 6 years ago

Hi xivk, I'm testing routing contraction with the latest development branch source code. As above issue, I have tried to build the whole US routing data with all of states .pbf data. It seems the out of range exception was thrown by Router db serialize progress. You can see the exception is thrown when the percent of the progress is 99%, the router db is being written to the disk.