Research and present GRPC interface for Pilosa

[jaffee]

@codysoyland commented on Tue May 30 2017

Description

GRPC is an RPC framework built on Protobuf and other technologies. It would be interesting to see how Pilosa would work over GRPC.

Success criteria (What criteria will consider this ticket closeable?)

Client and Server are modified to support GRPC and run queries. Usability and performance would be considered when deciding if we would like to support it long term.

[jaffee]

relates somewhat to #641 and #666 as GRPC has streaming support.

[codysoyland]

I spent some time learning and exploring GRPC as a replacement for certain functions in the handler. The possible benefits of switching to GRPC initially looked very optomistic:

1. Decoupling of handler logic from core business logic.
2. Reduction of error-prone [de-]serialization code and content-type handling.
3. Ability to enable streaming RPC.
4. Ability to use grpc-gateway to automatically create HTTP handlers for each GRPC handler.
5. Client library simplification through code generation from .proto file.
6. Sophisticated connection handling (pooling, timeouts, etc).
7. Automatic generation of Swagger metadata for better documentation and clarity about the handler methods.

While GRPC is awesome, there were some things I learned that made me more weary of using it:

1. grpc-gateway is confusing and not very configurable. The generated code is hard to read and impractical to modify. Backwards compatibility would not be attainable. It also proxies HTTP requests to GRPC locally instead of calling the GRPC methods directly, so performance would be reduced.
2. Not realistic (though somewhat possible) to share GRPC handler with HTTP handler, therefore we'd be adding an additional port to pilosa-server, an unfortunate configuration complexity.
3. Built-in authentication support is elementary and lacks integrations.

I published a branch that implements GetMaxSlices() as a GRPC function. Note that GetMaxSlices() requires an argument "Empty" because for some reason all RPC methods must take at least one argument.

In my opinion, the main benefits of using GRPC can be attained instead through handler improvements, yielding code that is easier to understand and modify, without adding a new configurable port to Pilosa server. Furthermore, GRPC-gateway is an imperfect solution to providing standard JSON/HTTP compatibility, so we'd still need to continue to maintain the existing handler code, and thereforce the net effect on the codebase would be an increase in code and complexity. The streaming support is a compelling reason to use GRPC, but can be replicated fairly easily using standard HTTP.

[mcuadros]

Maybe https://github.com/src-d/proteus could help with some of the concerns described.

I suggest not use grpc-gateway, since give you out of the box the HTTP handler, but is not very clean and not nice for sure.

One option could be, write the function handled and wrap it, with proteus and with a demuxer. having one handler for both endpoints.

I will love have GRPC support at pilosa.

[codysoyland]

That's interesting, @mcuadros. I haven't heard of proteus, but I will definitely take a close look at it. At first glance, it looks a lot more flexible than the official protobuf/grpc toolkit.

Thank you for your suggestion!

[mcuadros]

And maybe this https://github.com/twitchtv/twirp, can also help.

[jaffee]

re-opening so we can explore these suggestions

[travisturner]

blocked by #1093

[ghost]

I agree that grpc-gateway should not be used. Its old and really has been passed by much better approaches.

You should try this one: https://github.com/improbable-eng/grpc-web Its production battle teested.

Has webspckets now too :) https://github.com/improbable-eng/grpc-web/issues/167

And here is a great example of using it: https://github.com/johanbrandhorst/grpcweb-example Web demo: https://grpcweb.jbrandhorst.com/ As you can see its fast and can do basic duplex ( chat demo)

Its also using gopherjs for the frotn end with react.