Range violation on big pbf file

[denizzzka]

Previously it is worked well.

I'm not sure: this error in my code or in dproto code? dproto normally throws only its own exceptions?

core.exception.RangeError@std/bitmanip.d(2987): Range violation
----------------
??:? _d_arraybounds [0x6910cf]
??:? std.bitmanip.__array [0x69d474]
/usr/include/dmd/phobos/std/bitmanip.d:2987 pure nothrow @nogc @safe ulong std.bitmanip.read!(ulong, 1, ubyte[]).read(ref ubyte[]) [0x654bdf]
../../.dub/packages/dproto-2.1.0/dproto/import/dproto/serialize.d:410 pure nothrow @nogc @safe ulong dproto.serialize.readProto!("fixed64", ubyte[]).readProto(ref ubyte[]) [0x659a74]
../../.dub/packages/dproto-2.1.0/dproto/import/dproto/serialize.d:112 pure @safe void dproto.serialize.defaultDecode!(ubyte[]).defaultDecode(ulong, ref ubyte[]) [0x659a20]
../../.dub/packages/dproto-2.1.0/dproto/import/dproto/attributes.d:95 pure @safe void osmproto.Relation.__mixin10.deserialize!(ubyte[]).deserialize(ref ubyte[]) [0x65f4a4]
../../.dub/packages/dproto-2.1.0/dproto/import/dproto/attributes.d:217 pure @safe void dproto.attributes.putSingleProtoVal!("Relation", osmproto.Relation, ubyte[]).putSingleProtoVal(ref osmproto.Relation, ref ubyte[]) [0x65f2ec]
../../.dub/packages/dproto-2.1.0/dproto/import/dproto/attributes.d:193 pure @safe void dproto.attributes.putProtoVal!(osmproto.Relation[] osmproto.PrimitiveGroup.relations, ubyte[]).putProtoVal(ref ubyte[], ulong) [0x65f26f]
../../.dub/packages/dproto-2.1.0/dproto/import/dproto/attributes.d:90 pure @safe void osmproto.PrimitiveGroup.__mixin7.deserialize!(ubyte[]).deserialize(ref ubyte[]) [0x65ca62]
../../.dub/packages/dproto-2.1.0/dproto/import/dproto/attributes.d:217 pure @safe void dproto.attributes.putSingleProtoVal!("PrimitiveGroup", osmproto.PrimitiveGroup, ubyte[]).putSingleProtoVal(ref osmproto.PrimitiveGroup, ref ubyte[]) [0x65c944]
../../.dub/packages/dproto-2.1.0/dproto/import/dproto/attributes.d:193 pure @safe void dproto.attributes.putProtoVal!(osmproto.PrimitiveGroup[] osmproto.PrimitiveBlock.primitivegroup, ubyte[]).putProtoVal(ref ubyte[], ulong) [0x65c8cc]
../../.dub/packages/dproto-2.1.0/dproto/import/dproto/attributes.d:90 pure @safe void osmproto.PrimitiveBlock.__mixin8.deserialize!(ubyte[]).deserialize(ref ubyte[]) [0x65c40c]
../../.dub/packages/dproto-2.1.0/dproto/import/dproto/attributes.d:54 pure ref @safe osmproto.PrimitiveBlock osmproto.PrimitiveBlock.__mixin8.__ctor!(ubyte[]).__ctor(ref ubyte[]) [0x65c33b]
osm.d:310 map.region.region.Region osm.getRegion(immutable(char)[], bool) [0x65422d]
osm.d:384 map.map.Map osm.getMap(immutable(char)[][], bool) [0x654b29]
main.d:36 _Dmain [0x5fcb58]
??:? _D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ6runAllMFZ9__lambda1MFZv [0x6954f2]
??:? void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).tryExec(scope void delegate()) [0x69543c]
??:? void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).runAll() [0x6954ae]
??:? void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).tryExec(scope void delegate()) [0x69543c]
??:? _d_run_main [0x6953ad]
??:? main [0x5fd92f]
??:? __libc_start_main [0x551c5ef]
Program exited with code 1

[denizzzka]

Reproducing:

Download archive with simple DUB project, .proto files and sample of data here: http://rgho.st/private/8NmGNhtrX/8c7fcda3020bca1e457b2fbc6e7431a3

Archieve contains these files:

PrimitiveBlock.pbf
dproto_issue.d
dub.json
fileformat.proto
osmformat.proto

This is D file from that archive:

void main(){}

import dproto.dproto;
import dproto.imports;

mixin(ParseProtoSchema("<none>", `option dproto_reserved_fmt = "%s_"; ` ~ import("fileformat.proto")).toD());
mixin(ParseProtoSchema("<none>", `option dproto_reserved_fmt = "%s_"; ` ~ import("osmformat.proto")).toD());

unittest
{
    import std.file;

    ubyte[] buff = cast(ubyte[]) read("PrimitiveBlock.pbf");

    auto result = PrimitiveBlock(buff);
}

For compile and execute just run dub --build=unittest

I am sure on 90% what PrimitiveBlock.pbf contains valid protobuf data

[msoucy]

Is there any way you're able to reduce the .pbf file? As is, there's WAY too much data for me to be able to debug.

[denizzzka]

Is there any way you're able to reduce the .pbf file?

This file is already reduced from biggest ~1.5 Mb official OpenStreetMap pbf file. This is maximum what I can do with it because dproto fails on parsing this PrimitiveBlock struct.

[denizzzka]

@msoucy , hi! Sorry, but can I expect that you pay attention to this ticket? Many thanks!

[denizzzka]

I checked this pbf file for errors by protoc tool and it isn't shows any error.

$ protoc --decode=OSMPBF.PrimitiveBlock osmformat.proto < PrimitiveBlock.pbf

[msoucy]

Unfortunately, the real world has kept me very busy lately. I haven't had much time for any side projects.

Since you know that the data is good, maybe you can find a subset of it for a smaller test case?

[denizzzka]

Since you know that the data is good, maybe you can find a subset of it for a smaller test case?

No. It fails just at auto result = PrimitiveBlock(buff); and I can't edit original .pbf file

[denizzzka]

Archive with DUB project, .proto files and data sample reuploaded: https://drive.google.com/open?id=0BxYv_ASJV7uuR1Q5Z1hZcUxPTlU

[denizzzka]

Still reproduceable on 2.1.4 version

$ dub --build=unittest
Performing "unittest" build using /usr/bin/dmd for x86_64.
painlesstraits 0.3.0: target for configuration "unittest" is up to date.
dproto 2.1.4: target for configuration "library" is up to date.
gis-stuff ~master: building configuration "application"...
Linking...
To force a rebuild of up-to-date targets, run again with --force.
Running ./gis-stuff 
core.exception.RangeError@/usr/include/dmd/phobos/std/bitmanip.d(3225): Range violation
----------------
??:? _d_arrayboundsp [0x5cd2bc5a]
??:? pure nothrow @nogc @safe ulong std.bitmanip.read!(ulong, 1, ubyte[]).read(ref ubyte[]) [0x5cd0a99e]
??:? pure nothrow @nogc @safe ulong dproto.serialize.readProto!("fixed64", ubyte[]).readProto(ref ubyte[]) [0x5cd0a96c]
??:? pure @safe void dproto.serialize.defaultDecode!(ubyte[]).defaultDecode(ulong, ref ubyte[]) [0x5cd0a713]
??:? pure @safe void dproto_issue.Relation.__mixin10.deserialize!(ubyte[]).deserialize(ref ubyte[]) [0x5cd0d226]
??:? pure @safe void dproto.attributes.putSingleProtoVal!("Relation", dproto_issue.Relation, ubyte[]).putSingleProtoVal(ref dproto_issue.Relation, ref ubyte[]) [0x5cd0d0b0]
??:? pure @safe void dproto.attributes.putProtoVal!(dproto_issue.PrimitiveGroup.relations, ubyte[]).putProtoVal(ref ubyte[], ulong) [0x5cd0d034]
??:? pure @safe void dproto_issue.PrimitiveGroup.__mixin7.deserialize!(ubyte[]).deserialize(ref ubyte[]) [0x5cd0ad70]
??:? pure @safe void dproto.attributes.putSingleProtoVal!("PrimitiveGroup", dproto_issue.PrimitiveGroup, ubyte[]).putSingleProtoVal(ref dproto_issue.PrimitiveGroup, ref ubyte[]) [0x5cd0ac94]
??:? pure @safe void dproto.attributes.putProtoVal!(dproto_issue.PrimitiveBlock.primitivegroup, ubyte[]).putProtoVal(ref ubyte[], ulong) [0x5cd0ac19]
??:? pure @safe void dproto_issue.PrimitiveBlock.__mixin8.deserialize!(ubyte[]).deserialize(ref ubyte[]) [0x5cd09d96]
??:? pure ref @safe dproto_issue.PrimitiveBlock dproto_issue.PrimitiveBlock.__mixin8.__ctor!(ubyte[]).__ctor(ref ubyte[]) [0x5cd09d0b]
??:? void dproto_issue.__unittest_dproto_issue_9_0() [0x5ccf34f7]
??:? void dproto_issue.__modtest() [0x5cd19084]
??:? int core.runtime.runModuleUnitTests().__foreachbody2(object.ModuleInfo*) [0x5cd40867]
??:? int object.ModuleInfo.opApply(scope int delegate(object.ModuleInfo*)).__lambda2(immutable(object.ModuleInfo*)) [0x5cd2b252]
??:? int rt.minfo.moduleinfos_apply(scope int delegate(immutable(object.ModuleInfo*))).__foreachbody2(ref rt.sections_elf_shared.DSO) [0x5cd318dd]
??:? int rt.sections_elf_shared.DSO.opApply(scope int delegate(ref rt.sections_elf_shared.DSO)) [0x5cd31a70]
??:? int rt.minfo.moduleinfos_apply(scope int delegate(immutable(object.ModuleInfo*))) [0x5cd31869]
??:? int object.ModuleInfo.opApply(scope int delegate(object.ModuleInfo*)) [0x5cd2b229]
??:? runModuleUnitTests [0x5cd4063d]
??:? void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).runAll() [0x5cd2cdb0]
??:? void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).tryExec(scope void delegate()) [0x5cd2cd37]
??:? _d_run_main [0x5cd2cca2]
??:? main [0x5cd191ed]
??:? __libc_start_main [0x9dd4cf29]
1/1 unittests FAILED
Program exited with code 1

[denizzzka]

Plz add bug badge