TypeScript generation to support node-grpc

[MOZGIII]

I'm trying to use protobuf.js with grpc-node native core, and want to have a proper TypeScript definitions.

I have already mentioned this at #1007, but I'm really struggling with incompatibilities of the generated TypeScript types and the grpc implementation. Decided it's worth a separate issue here.

So, the problem is, TypeScript generated code does not reflect the existence of the streams in the rpc definition.

Example:

syntax = "proto3";

package example;

service StreamExample {
  rpc NoStreams(Request) returns (Response) {}
  rpc StreamingResponse(Request) returns (stream Response) {}
  rpc StreamingRequest(stream Request) returns (Response) {}
  rpc StreamingBoth(stream Request) returns (stream Response) {}
}

message Request {
  string a = 1;
  string b = 2;
}

message Response {
  string c = 1;
  string d = 2;
}

export namespace example {
    class StreamExample extends $protobuf.rpc.Service {
        public noStreams(request: example.IRequest, callback: example.StreamExample.NoStreamsCallback): void;
        public noStreams(request: example.IRequest): Promise<example.Response>;

        public streamingResponse(request: example.IRequest, callback: example.StreamExample.StreamingResponseCallback): void;
        public streamingResponse(request: example.IRequest): Promise<example.Response>;

        public streamingRequest(request: example.IRequest, callback: example.StreamExample.StreamingRequestCallback): void;
        public streamingRequest(request: example.IRequest): Promise<example.Response>;

        public streamingBoth(request: example.IRequest, callback: example.StreamExample.StreamingBothCallback): void;
        public streamingBoth(request: example.IRequest): Promise<example.Response>;
    }
    namespace StreamExample {
        type NoStreamsCallback = (error: (Error|null), response?: example.Response) => void;
        type StreamingResponseCallback = (error: (Error|null), response?: example.Response) => void;
        type StreamingRequestCallback = (error: (Error|null), response?: example.Response) => void;
        type StreamingBothCallback = (error: (Error|null), response?: example.Response) => void;
    }
}

(comments and some code parts are removed for clarity)

As seen from the example, function signatures in TypeScript are the same, despite them being different in the protobuf definition.

I'd like to discuss and, probably, correctly implement the extensive support for protobuf format and compatibility with grpc-node.

This is a starting point. I'll publish a sample repo soon with the examples so that we have something to play with.

[coding2012]

Note that Promises are not really allowed to have multiple values (they officially end after one value). But RXJS Observables seem to be a good fit for multiple values... they definitely are designed for streams in this sense. - Just a note that you might find some promise libraries that do technically work after the first response, but are not really supposed to.

[MOZGIII]

There's been quite a discussion at https://github.com/grpc/grpc-node/issues/528 about this recently. Anyone interested, please check that out.

[MOZGIII]

TL;DR: a dedicated support is needed from protobuf.js, in particular we need better understanding and type definitions for the format that fromObject/toObject operate on:

https://github.com/dcodeIO/protobuf.js/blob/69623a91c1e4a99d5210b5295a9e5b39d9517554/tests/data/rpc.d.ts#L27-L28

The type definitions for those two pretty much allow anything, while it practice there are concrete requirements to the input, and the output has a particular shape.

And, what's interesting, the message interfaces do not fit as valid descriptions for the objects passed to fromObject (with the sample above - fromObject does not take IMyRequest in general case). Not sure about toObject, but they output is most likely is incompatible with IMyRequest too (in the general case). In practice, this incompatibility occurs, for example, when message contains enums.

[trajano]

I was able to successfully create a server in TypeScript with protobufjs along with a few additions. https://stackoverflow.com/a/64778825/242042

The protobufjs generates the request and response callbacks properly the most part. However, I had to create the "service" types as follows

The interface for the service, I think this can be generated by a tool

interface IArtifactUpload {
  signedUrlPutObject: handleUnaryCall<IUploadRequest, ISignedUrlPutObjectResponse>;
}

These are needed to expose service. I am wondering if I should I create a bug asking why service is not part of the GrpcObject type when loadPackageDefinition adds it in?

interface ServerDefinition extends GrpcObject {
  service: any;
}
interface ServerPackage extends GrpcObject {
  [name: string]: ServerDefinition
}
const protoDescriptor = loadPackageDefinition(packageDefinition) as ServerPackage;

I can force a type check when adding the service as follows

server.addService<IArtifactUpload>(protoDescriptor.ArtifactUpload.service, {
  signedUrlPutObject(call, callback) {
    callback(null, SignedUrlPutObjectResponse.create({ reply: "hello " + call.request.message }));
  }
});

[trajano]

I've found a way of doing a slightly cleaner typescript and made some alterations to pretend I am crazy enough to put more than one service definition in a single proto file.

This is the common interfaces, maybe it should be part of GRPC itself. I reduced it to a single class

interface ServerPackage<W> extends GrpcObject {
  [name: string]: {
    service: GrpcObject & ServiceDefinition<W>
  }
}

And here are my application specific code which I think can be generated with a tool.

interface IArtifactUpload {
  signedUrlPutObject: handleUnaryCall<IUploadRequest, ISignedUrlPutObjectResponse>;
}
interface IArtifactDownload {
  foo: handleUnaryCall<IUploadRequest, ISignedUrlPutObjectResponse>;
}
type CustomApi = IArtifactUpload | IArtifactDownload;

Then:

const protoDescriptor = loadPackageDefinition(packageDefinition) as ServerPackage<CustomApi>;

Then the server and implementations:

const server = new Server();
server.addService(protoDescriptor.ArtifactUpload.service as ServiceDefinition<IArtifactUpload>, {
  signedUrlPutObject(call, callback) {
    callback(null, SignedUrlPutObjectResponse.create({ reply: "hello " + call.request.message }));
  }
});
server.addService(protoDescriptor.ArtifactUpload.service as ServiceDefinition<IArtifactDownload>, {
  foo(call, callback) {
    callback(null, SignedUrlPutObjectResponse.create({ reply: "hello " + call.request.message }));
  }

});
server.bind('0.0.0.0:50051', ServerCredentials.createInsecure());
server.start();