search

itinero / routing

The routing core of itinero.
Apache License 2.0
221 stars 70 forks source link

Exception contracting europe-latest.osm.pbf #253

Open TomQv opened 5 years ago

TomQv commented 5 years ago

I'm compiling europe-latest.osm.pbf to a routerdb. Machine has 144GB memory. Using standard Car.lua and processRestrictions=true.

Without contraction routerDb.LoadOsmData() and routerDb.Serialize() finishes successfully. With contraction the following exception occurs after about 1h running in routerDb.AddContracted(vec.Fastest(), processRestrictions); . Smaller pbf-files are fine. Is Europe too much? Or is it suppose to compile? There is int32 in the exception, should this maybe be changed to int64?

Array dimensions exceeded supported range.
   at System.Collections.Generic.Dictionary`2.Resize(Int32 newSize, Boolean forceNewHashCodes)
   at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
   at Itinero.Algorithms.Contracted.Dual.FastHierarchyBuilder`1.NotifyContracted(UInt32 vertex)
   at Itinero.Algorithms.Contracted.Dual.FastHierarchyBuilder.Contract()
   at Itinero.Algorithms.Contracted.Dual.FastHierarchyBuilder`1.DoRun(CancellationToken cancellationToken)
   at Itinero.Algorithms.AlgorithmBase.Run(CancellationToken cancellationToken)
   at Itinero.RouterDbExtensions.AddContracted(RouterDb db, Profile profile, Boolean forceEdgeBased, CancellationToken cancellationToken)
   at Itinero.RouterDbExtensions.AddContracted(RouterDb db, Profile profile, Boolean forceEdgeBased)
   at QvxItinero.QmrTools.<>c__DisplayClass0_0.<CompileQmr>b__0() in C:\Users\Tom\QuoVadis\Qvx1\QvxItinero\Qmr\CompileQmr.cs:line 77
   at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at QvxItinero.QmrTools.<CompileQmr>d__0.MoveNext() in C:\Users\Tom\QuoVadis\Qvx1\QvxItinero\Qmr\CompileQmr.cs:line 35
TomQv commented 5 years ago

Same error with asia-latest.osm, which is 7.2GB. africa-latest.osm.pbf (3GB), germany-latest.osm.pbf (3.1GB) and south-america-latest.osm.pbf (1.5GB) was ok.

Is there a limit in size?

xivk commented 5 years ago

A standard .NET dictionary/array is limited in size. Replacing this by HugeDictionary could fix the issue:

https://github.com/itinero/routing/blob/develop/src/Itinero/Algorithms/Contracted/Dual/FastHierarchyBuilder.cs#L19 https://github.com/itinero/routing/blob/develop/src/Itinero/Algorithms/Contracted/Dual/FastHierarchyBuilder.cs#L20

https://github.com/itinero/routing/blob/develop/src/Itinero/Algorithms/Collections/HugeDictionary.cs

Will try to do this asap. It's strange however that I never ran into this issue myself.

TomQv commented 5 years ago

Did you contract europe already successfully before? I recently upgraded to .net 4.7.2, maybe there is a problem?

xivk commented 5 years ago

FYI, I'm testing with europe currently. The fix done does seem to solve the issue you were having but I ran into another issue:

image

Something is going wrong during compression of the graph. I hope it's not too big and overflows uint range.

TomQv commented 5 years ago

same here... I was almost getting excited, because it was running the whole night without error, but just now it also ended with this exception..

TomQv commented 5 years ago

The latest commit from Mar 28, did it fix this issue?

TomQv commented 5 years ago

yes, europe was now successful :) It took 26h, but cpu-load didn't exceed 8% on a 16-core cpu. Seems, that only one core is working?