"invalid wire type" error when decoding GTFS realtime feed

[Mchristos]

I'm fetching feed data from a URL endpoint of the form <baseUrl>/<feedName>.proto, but for some reason I get an "invalid wire type" error when I do GtfsRealtimeBindings.transit_realtime.FeedMessage.decode(body) in Node.js. I'm using the example code from the Node.js README:

var GtfsRealtimeBindings = require('gtfs-realtime-bindings');
var request = require('request');

var requestSettings = {
  method: 'GET',
  url: '<baseUrl>/<feedName>.proto',
  encoding: null,
};
request(requestSettings, function (error, response, body) {
  if (!error && response.statusCode == 200) {
    console.log(body)
    var feed = GtfsRealtimeBindings.transit_realtime.FeedMessage.decode(body);
    feed.entity.forEach(function(entity) {
      if (entity.trip_update) {
        console.log(entity.trip_update);
      }
    });
  }
});

// <Buffer 68 65 61 64 65 72 20 7b 0a 20 20 67 74 66 73 5f 72 65 61 6c 74 69 6d 65 5f 76 65 72 73 69 6f 6e 3a 20 22 31 2e 30 22 0a 20 20 69 6e 63 72 65 6d 65 6e ... 37282 more bytes> object
// C:\Users\Chris Marais\code\js\protobuf\node_modules\gtfs-realtime-bindings\node_modules\protobufjs\src\reader.js:375
            throw Error("invalid wire type " + wireType + " at offset " + this.pos);
            ^

// Error: invalid wire type 7 at offset 12

Is the array buffer encoded incorrectly perhaps? Unfortunately I'm unable to share the data, but I can confirm that it conforms to the GTFS realtime specification

[barbeau]

Is the array buffer encoded incorrectly perhaps? Unfortunately I'm unable to share the data, but I can confirm that it conforms to the GTFS realtime specification

@Mchristos Just to confirm the .proto file is a binary file - can you open the .proto file in a text editor and read the contents in plain text?

Typically files with the extension .proto are the protocol buffer definition files, and aren't the compressed binary representation of the actual data - those typically have a .pb extension.

[Mchristos]

Thanks @barbeau - yes I thought something was a bit odd - yes the files are plain text, which from my understanding is not the usual way of sending protobuf. But, this is the data I'm dealing with. Any suggestions?

[Mchristos]

So, to clarify, I'm looking at plain-text snapshots of the binary protobuf data, but I only have access to the plain text. Any way to recover or parse the data somehow?

[barbeau]

@Mchristos Are you sure that the producer isn't maybe providing the .proto file as an extension to the gtfs-realtime.proto file, and the actual message is in another field?

If the .proto file looks like https://developers.google.com/transit/gtfs-realtime/gtfs-realtime-proto, then it's probably an extension file. In that case, you'd need to generate your own code bindings from that .proto file (see https://github.com/MobilityData/gtfs-realtime-bindings/blob/master/nodejs/UPDATING.md for how we do this for Node.js in this project, and this issue for someone else that just did this). And, you'd need to find another binary message from the producer that contains the actual data you could parse using the bindings you created.

If the .proto file looks more like https://github.com/google/transit/blob/master/gtfs-realtime/spec/en/examples/trip-updates-full.asciipb, then it's basically a JSON-ized plain text version of the protocol buffer data. I know Google's protocol buffer compiler gives you an option to export a message object to JSON, and I think it might have an option to read one in too. Or, if the message is valid JSON, you could use any JSON parser.

[Mchristos]

Yup, its definitely the latter (a JSON-ized version of an actual message). But, it doesn't look exactly like JSON to me (if so it would have an array of entities, rather than repeated entity {} declarations). Might be tricky to parse, but I can try and report back. I'll also try using google's protocol directly like you suggested.

[barbeau]

@Mchristos Yes, please let me know what you find. It's certainly worth reporting this issue to the producer too - maybe they don't realize that they actually aren't sharing a true protocol buffer message.

[saretter]

Has this issue been solved? I am seeing a similar error.

err: { "type": "Error", "message": "invalid wire type 6 at offset 77", "stack": Error: invalid wire type 6 at offset 77 at Reader.skipType (/....../node_modules/protobufjs/src/reader.js:377:19) at Reader.skipType (/....../node_modules/protobufjs/src/reader.js:368:22) at Function.decode (/....../node_modules/gtfs-realtime-bindings/gtfs-realtime.js:150:28)

Using protoc --decode transit_realtime.FeedMessage gtfs-realtime.proto < gtfsr_connect.bin the data can be parsed.

The endpoint providing the data is publicly available http://gtfsr.vbn.de/gtfsr_connect.bin I'd be grateful for any hint!

[toops61]

Exactly the same problem for me : I have an url wich is supposed to be a GTFS RealTime update in proto file type. So I downloaded the gtfs-realtime.proto file to get the types but my first problem is that there are multiple messages and types but almost each time I get the same error :

throw Error("invalid wire type " + wireType + " at offset " + this.pos); ^ [ Error: invalid wire type 4 at offset 1

My code looks like this : (i use protobufjs in node with import protobuf from "protobufjs")

try { const response = await fetch(url); if (!response.ok) { throw new Error(Erreur HTTP : ${response.status}); }

    const buffer = await response.arrayBuffer();
    const arrayBuff = new Uint8Array(buffer);

    protobuf.load("gtfs-realtime.proto", function(err, root) {
        if (err)
            throw err;

         // Obtain a message type
        const ProtoMessage = root.lookupType("transit_realtime.FeedMessage");
        // Decode an Uint8Array (browser) or Buffer (node) to a message
        const message = ProtoMessage.decode(arrayBuff);
        // ... do something with message

        console.log(message);
    });

    return response;
}

Does someone find the solution ?

if I download and open the file I get from URL it looks like this in NotePad++ 450375424626689018:02:0020231125 *118219533168149150 
  "3377704015496221(  I"3377704015495624( 'óÿÿÿÿÿÿÿÿ
y"3377704015495803( (Òÿÿÿÿÿÿÿÿ
…
"3377704015496100( i"3377704015495675( DD"3377704015495383(  F"3377704015495283(  g"3377850044383400(   C"3377850044383436( 

[jameslinjl]

Hey @toops61 ! You shouldn't need to use protobufjs at all if you're using gtfs-realtime-bindings. This package already generates the JS code for you based on the protobuf schema.

Take a look at the example usage here: https://github.com/MobilityData/gtfs-realtime-bindings/blob/master/nodejs/README.md#example-code

[jameslinjl]

@saretter Sorry for the super late response, but I was able to use your provided endpoint with no errors! I based my code off the example provided in the README: https://github.com/MobilityData/gtfs-realtime-bindings/blob/master/nodejs/README.md#example-code

[jameslinjl]

For what it's worth, I don't think there is actually any bug here. From what I can tell, this was simply someone producing bad or confusing data, so the GTFS bindings library was being pointed at something that wasn't actually GTFS protobuf data.

@isabelle-dr When you have a minute, please close this issue so that others don't get confused here!

[toops61]

thanks for your answers. I don't use gtfs-realtime-bindings actually, that's why I was using protobufjs.

[toops61]

and I'm also looking for a way to use all messages types that the gtfs-realtime.proto contains if it's possible (FeedMessage, FeedHeader, FeedEntity, TripUpdate, etc...) ?

[jameslinjl]

@toops61 This issue that you're commenting on is for a Github Issue in the gtfs-realtime-bindings repo, so discussion here should revolve around that specific tool! If you have questions about the GTFS realtime spec / proto more broadly or protobufjs, I would advise going to those repos / projects more directly (here and here).

For what it's worth, gtfs-realtime-bindings may be a good use-case for you, since it hides all the complexity typically involved with getting protocol buffers set up by distributing the generated code directly.

[isabelle-dr]

Hello! Thanks for the ping @jameslinjl. Closing this issue since it seems this is a problem with the input data and not the realtime bindings.