1.4 pre 99 : Exception in Multithreading

[manuprendlair]

Hello,

I got System.Exception' dans Itinero.dll ("Cannot read elements with an id outside of the accessor range.") in a Parallel.For

Here my code:

int max = 100;
Parallel.For(0, max, ctr =>
{
if (TestMultiThreading(v_origine, v_dest)) Interlocked.Increment(ref v_numFailed);
});

public bool TestMultiThreading(Point p_Origin, Point p_Destination)
        {
            bool v_Fail = false;
            try
            {
                Route route = new Route();
                Profile v_profile = Vehicle.Car.Shortest();
                Coordinate v_pointA, v_pointB;
                v_pointA = new Coordinate((float)p_Origin.CoordonneeY, (float)p_Origin.CoordonneeX);
                v_pointB = new Coordinate((float)p_Destination.CoordonneeY, (float)p_Destination.CoordonneeX);
                RouterPoint start, end;
                const float DistanceMaxProj = 500.0F;
                Router router = new Router(v_routerDB);
                start = router.Resolve(v_profile, v_pointA, DistanceMaxProj);
                end = router.Resolve(v_profile, v_pointB, DistanceMaxProj);
                route = router.Calculate(v_profile, start, end);
            }
            catch (Exception exc)
            {
                v_Fail = true;
            }

            return v_Fail;
        }

Regards.

Emmanuel

[juliusfriedman]

I also experience this using similar code to above in the latest version and pre-release.

It seems to stem from

"System.ArgumentException: 'Offset and length were out of bounds for the array or count is greater than the number of elements from index to the end of the source collection.'"

Reminiscence.dll!Reminiscence.IO.Streams.CappedStream.Read(byte[] buffer, int offset, int count) Unknown

The underlying call in Itinero was

Itinero.dll!Itinero.Attributes.AttributesIndex.InternalTagsEnumerator.Current.get() Line 574 C# From Itinero.dll!Itinero.Profiles.DynamicProfile.FactorAndSpeed(Itinero.Attributes.IAttributeCollection attributes) Line 109 C#

[juliusfriedman]

example repo using Reminiscence 1.4.0-pre002

The exception we cite above is gone in most cases but there appears to be other problems in Itinero otherwise then.... see for example

 ParallelEnumerable.ForAll(Enumerable.Range(0, 100).AsParallel(), x =>
            {
                // create a routerpoint from a location.
                // snaps the given location to the nearest routable edge.
               Begin:
                try
                {
                    var starta = router.Resolve(profileFastest, 40.69939f, -80.27956f);
                    var enda = router.Resolve(profileFastest, 40.79308f, -80.33498f);
                    // calculate a route.
                    var routea = router.TryCalculate(profileFastest, starta, enda);
                    if (routea.IsError)
                    {
                        Console.WriteLine("ERROR");
                        return;
                    }

                    Console.WriteLine(routea.Value.ToJson());
                }
                catch(Exception ex)
                {
                    Console.WriteLine(ex.Message);
                    goto Begin;
                }
            });

75% of the time Always causes an exception: (which comes from Resolve)

"Could not resolve point at [40.69939, -80.27956]. Probably too far from closest road or outside of the loaded network"

Other times I see

Cannot read elements with an id outside of the accessor range.

[juliusfriedman]

I have a stack trace string from the latest build:

" at Reminiscence.Indexes.Index`1.Get(Int64 id) in src\Reminiscence\Indexes\Index.cs:line 207"

while(accessorBytesOffset <= id)
            { // keep looping until the accessor is found where the data is located.
                a++;
                if (a >= _accessors.Count)
                {
                    throw new System.Exception("Cannot read elements with an id outside of the accessor range.");
                }
                accessorBytesLostPrevious = accessorBytesLost;
                accessorBytesLost += _accessorBytesLost[a]; 
                accessorBytesOffset = (_accessorSize * (a + 1)) - accessorBytesLost;
            }

a is equal to 1 at the time of the exception and there are 2 _tags underlying TryGetValue call which is being called with i = 0.

accessorBytesOffset = 0, and id = 3840, _accessorBytesLost = [ 0 ]

Also occurs from Resolve... so it makes me think that when segments are in use that DB is removing them and not allowing Read access or something. Hard to tell as I just started really looking and this and I don't have time until Monday 👍

Anyway, If I do find more time this weekend I will let you know!

Btw,

The call under that was Contains with "translated_profile" as key and "yes" as value;

AND

This does NOT seem to occur if the graph is contracted with, routerDb.AddContracted(vehicle.Profile("shortest"));

[juliusfriedman]

This actually also occurs when multiple contractions are present.

When I have only a single contraction I don't see these errors but they come back when I add more then one.

[juliusfriedman]

The issue seems to stem from ProfileFactorAndSpeedCache.CalulcateFor,

var edgeProfileTags = _db.EdgeProfiles.Get(edgeProfile); returns a collection with a Count = 4 and the following tags:

0, 11, 25, 30, 6, 2816, 1520, 8960

However the enumerator throws the same exception "Cannot read elements with an id outside of the accessor range."

I will let you know what else I find.

[juliusfriedman]

This also occurs from CompleteRouteBuilder but with the same reason as above,

var profile = _routerDb.EdgeProfiles.Get(edge.Data.Profile); is a AttributeCollection with count = 4 but cannot be enumerated

[juliusfriedman]

What's odd is that if I call the _routerDb.EdgeProfiles.Get(edge.Data.Profile) a 2nd time after the exception I get the same keys but I can enumerate them....

[juliusfriedman]

Okay... it seems the lock in CalculateFor is causing the issue... at least in some cases.

Changing CaulcateFor to lock the DB instead of the ProfileFactorAndSpeedCache seems to remove the exception with or without contractions

public void CalculateFor(params Profile[] profiles)
        {
            lock (_db)
            { // don't allow multiple threads to fill this cache at the same time.
                var newEdgeProfileFactors = new Dictionary<string, FactorAndSpeed[]>(_edgeProfileFactors);

                var edgeProfileFactors = new FactorAndSpeed[profiles.Length][];
                for (var p = 0; p < profiles.Length; p++)
                {
                    edgeProfileFactors[p] = new FactorAndSpeed[(int) _db.EdgeProfiles.Count];
                }

                for (uint edgeProfile = 0; edgeProfile < _db.EdgeProfiles.Count; edgeProfile++)
                {
                    var edgeProfileTags = _db.EdgeProfiles.Get(edgeProfile);
                    for (var p = 0; p < profiles.Length; p++)
                    {
                        edgeProfileFactors[p][edgeProfile]
                            = profiles[p].FactorAndSpeed(edgeProfileTags);
                    }
                }

                for (var p = 0; p < profiles.Length; p++)
                {
                    newEdgeProfileFactors[profiles[p].FullName] = edgeProfileFactors[p];
                }

                _edgeProfileFactors = newEdgeProfileFactors;
            }
        }

Still not sure of how or why since there are a lot of moving pieces. But it seems the _stringIndex gets corrupted somehow by multiple readers when there is not a lock on the db....

https://github.com/itinero/routing/pull/302

[rodion-m]

So it's still appearing in 1.6.0-pre029:

System.Exception : Cannot read elements with an id outside of the accessor range.
   at Reminiscence.Indexes.Index`1.Get(Int64 id)
   at Itinero.Attributes.AttributesIndex.InternalAttributeCollection.TryGetValue(String key, String& value)
   at Itinero.Profiles.IAttributeCollectionExtensions.TryParseTranslated(IAttributeCollection attributes, String profileName, FactorAndSpeed& factorAndSpeed)
   at Itinero.Profiles.DynamicProfile.FactorAndSpeed(IAttributeCollection attributes)
   at Itinero.Profiles.ProfileFactorAndSpeedCache.CalculateFor(Profile[] profiles)
   at Itinero.RouterBaseExtensions.GetIsAcceptable(RouterBase router, IProfileInstance[] profiles)
   at Itinero.RouterBaseExtensions.TryResolveConnected(RouterBase router, IProfileInstance profileInstance, Single latitude, Single longitude, Single radiusInMeter, Single maxSearchDistance, Nullable`1 forward, CancellationToken cancellationToken)

Any way to fix it? Or just not to use it in multi-thread? Also does it mean that we should not use Router in multi-thread at all? @xivk @juliusfriedman

[YuriiNskyi]

Hello, bumping this issue, is there any progress on resolving this problem? It's really hard to use Itinero with heavy load, due to exception above.