Go <-> NodeJS: Serialization question for non primitive types

[diervo]

Problem description

This is more a question than a problem

We have a GRPC service written in Go, and a client written in Javascript (using Nodejs runtime).

Besides some primitives, we are using some google protobuf structures as follows:

message PredicateMatch {
  message Entry {
      google.protobuf.Timestamp timestamp = 1;
      oneof value {
        float scalar = 2;
        string compound = 3;
      }
  }
 ...

When generating the types and using the proto loader in nodejs to get the responses

export interface _api_PredicateMatch_Entry {
    'timestamp'?: (_google_protobuf_Timestamp | null); // this is an object with { seconds: string, nanos: number } 
    'scalar'?: (number | string);
    'compound'?: (string);
    'value'?: "scalar" | "compound";
}

The challenge is that we need to manually convert the Timestamp to an "almost equivalent" with a function like this:

export function convertProtoTimestampToUnixMs({ seconds, nanos }: { seconds: string; nanos: number }) {
    return (parseInt(seconds) + nanos / 1e9) * 1000;
}

Is there a better way to do this? Is there a way to have some "hooks" or interceptors at the deserialization layer to make this abstraction execute in a better layer?

[murgatroid99]

I don't have a better option for you here. Protobuf.js (which proto-loader uses under the hood) has functionality here to convert "well known" types (google.protobuf.x) to their canonical JSON representation, but the only one actually implemented is google.protobuf.Any. And if the transformation for google.protobuf.Timestamp was implemented, it would use the canonical JSON representation for that type, which is an RFC 3339 timestamp string.

[vsly-ru]

You either use a functions like convertProtoTimestampToUnixMs, or you have to use string/int64 instead of well known/best practice type. In case of int64, it's useful to rename the field to timestampMs or timestampS, depending on the value. Pros and cons of each format:

1. google.protobuf.Timestamp (+) 'well known type', (+) always in UTC, (-) not implemented properly in nodejs, (-) requires helper functions, (-) you need to import it first, (-) slightly slower performance: parseInt and division.
2. string (+) ISO 8601/RFC 3339, (-) should be well documented which exact format is used (spaces, ms, tz), (-) some standard language libs may encode it differently, (-) some clients may encode it with timezone different from UTC, (-) performance: parsing ISO 8601 string into js Date is ~5x slower than parsing number with epoch time in ms: 349 ms vs 72 ms for 1e6 iterations.
3. int64 (+) always in UTC, (+) minimum size (just up to 8 bytes), (+) the fastest deserialization, (-) requires to specify units (s/ms).