Issues with thread safety ...

[Tom-Cuthill]

If I try to use the VectorTileParser.Parse method running in multiple threads I consistently get exceptions being thrown from the protobuf-net library. Here is the internal exception:

Timeout while inspecting metadata; this may indicate a deadlock. This can often be avoided by preparing necessary serializers during application initialization, rather than allowing multiple threads to perform the initial metadata inspection; please also see the LockContended event

Apparently reading a class's metadata in the protobuf-net library is not thread safe. See here: http://stackoverflow.com/questions/17096359/is-protobuf-net-thread-safe

The recommended solution is prepare the serializer for the class before the Serializer.Deserializer is called.

I tried following the advice, and put this at the top of the VectorTileParser class:

static VectorTileParser() { Serializer.PrepareSerializer(); }

I can see, while using a debugger, that the static initializer is called first, and then the subsequent calls to the Parse method come in. Unfortunately the same exception still happens.

I've attached some sample code to demo the problem. I am not an expert on threading, but I can not see how the mapbox-vector-tile-cs library can be causing a deadlock. Based on the exception, it appears there is contention on reading the Tile classes metadata. I would have thought that doing the PrepareSerializer() in the static constructor would have prevented this from happening?

Being able to load vector tiles in parallel is important to the app I am building so that there is decent performance on the client.

Please let me know your thoughts,

Cheers, Tom Cuthill

TestCode.txt

[Tom-Cuthill]

Sorry there was a small typo, I did include the class name Tile in the PrepareSerializer call. See below:

static VectorTileParser() { Serializer.PrepareSerializer(); }

[Tom-Cuthill]

Apparently the contents of the angle braces get removed when submitted text in this textbox!!

[pauldendulk]

@Tom-Cuthill For code samples use 3 backticks (top left of keyboard) at start and end. See syntax 'highlighting' over here.

static VectorTileParser()
{
   Serializer.PrepareSerializer();
}

[pauldendulk]

@Tom-Cuthill Maybe a workaround is to do all vector tile parsing on a single background thread. So, start a background thread and do all parsing there. The main app is not blocked that way and protobuf-net is accessed on a single thread.

[Tom-Cuthill]

Yeah, well the tiles are all loading in separate background threads, so collapsing them down to a single thread to do parsing is a bottleneck and far from ideal. The protobuf-net library supports multithreading, it is just its initialization which isn't. I'll look for another solution ...

[bertt]

Hi, I'm trying to reproduce the exceptions using your sample code in a UWP project (see https://github.com/bertt/vectortileparsing/), but I don't see any exception occurring in method parseTile. So how can we reproduce the exception?

[Tom-Cuthill]

I'm using Visual Studio 2015 and the project is a Universal Windows 8.1 app. Unfortunately we have to support this platform for the next year or so.

PS. I tried using the official google's official protobuf C# library, using the protoc utility to generate a C# class to do the protobuf parsing. Since it builds the class from the official mapbox vector tile project's proto file (see here: https://github.com/mapbox/vector-tile-spec/blob/master/2.1/vector_tile.proto), the library doesn't need to parse any metadata and consequently runs faster. I've tested it and it runs fine in a multithreaded environment. NOTE: That if you follow this route, you have to update the syntax of the vector_tile.proto file to version 3 of proto.

[bertt]

Can you send a PR of using the Google protobuf lib?

[bertt]

closing because no activity