Reusability - grpc

[joeblew99]

We are considering to use Blast for a Project to replace Elastic Search.

But we need to support many languages and GRPC makes it much easier. Also for the HTTP is gives the devs a swagger REST API.

Blast would be easier to use from different languages and the web if the API used GRPC and GRPC Gateway.

https://github.com/grpc-ecosystem/grpc-gateway

Easy example https://github.com/johanbrandhorst/grpcweb-boilerplate

[mosuka]

Hi @joeblew99 , Sounds good.:) I'll replace to grpc-gateway from the current HTTP server.

[joeblew99]

I had a look at the GRPC code you have now. Wondering if you knew about GRPC Gateway back then or not ?

What do you think of this approach ?

1. Client (some MicroService written in any language).

   • They could use swagger ( from GRPC gateway ) to codegen their own code.
   • Or they can take the GRPC and codegen their own code.
   • Or they can just make raw HTTP calls, which would be pretty aweful because the API and types of Blast are not that simple. Types especially are quite complex and deeply needed.
   • All 3 options above use HTTP. Websockets are not needed.

2. Blast Server

   • Runs GRPC and GRPC Gateway, so that client can use either.

About the CLI. I saw some golang lib that can take a GRPC API and code gen the golang CLI with the viper code inside. Its really code generating a CLI can then has a GRPC Client that calls the GRP Server. DO you think this is a good idea ? I like it because it makes it much easier to refactor code when needed.

Security. If someone wanted to use Blast in a Multi-Tenant setup, i wonder if they can use caddy to restrict access to different parts of the Data Blast holds ? I cant actually think of a way except to run different Instances of blast and just lets Caddy on top and restrict by Domain / IP address.

[mosuka]

@joeblew99 I released v0.8.0, rewritten to grpc-gateway. I'm not familiar with swagger, but it creates swagger.json. However, expected swagger.json could not be generated because Bleve's SearchRequest and SearchResult use Protocol buffers's any.proto. I will continue to investigate it. Currently, the user will need to write a raw HTTP request directly.

Can you tell me a few about the CLI library? I'll try it out.

I agree with your opinion about multi-tenancy. Blast does not plan for multi-tenant support yet.

Thanks,

[l4u]

@mosuka what was the protoc command you used? Probably an extra param for proto_path is needed.

[joeblew99]

hey @mosuka awesome stuff. I will have a look soon.

About the GRPC tools. There is a general list here: https://github.com/grpc-ecosystem/awesome-grpc#tools-cli

May Favs:

CLI for GRPC https://github.com/fullstorydev/grpcurl

• pretty cool

Web GUI for GRPC https://github.com/fullstorydev/grpcui OR https://github.com/gogo/letmegrpc

Proxy https://github.com/improbable-eng/grpc-web

Just ask if this is not enough....

• helps cognitively.

[mosuka]

Hi @l4u @joeblew99 , swagger.json has been generated. https://github.com/mosuka/blast/blob/master/protobuf/index/index.swagger.json

However, the response specification is not what I expected. For example, the response body specification of GetResponse will be any.proto. This is because google.protobuf.Any is used for fields in GetResponse.

I would like to hear your opinion. Which is the preferred response body?

1. Current response body using custom json marshaler:

   {
   "fields": {
   "_type": "enwiki",
   "text_en": "A search engine is an ...",
   "timestamp": "2018-07-04T...",
   "title_en": "Search engine..."
   }
   }

2. the response body of the original grpc-gateway:

{
  "value": {
    "type_url": "map[string]interface {}",
    "value": " ... base64 encoded string ... "
  }
}

I have never used swagger so I don't know which one is better.

Thanks,

[joeblew99]

@mosuka

do you have to use google.protobuf.Any ? What is the reason for it ?

Also i noticed your not usin the "openapiv2" options. See: https://github.com/johanbrandhorst/grpc-gateway-boilerplate/blob/master/proto/example.proto#L13

Here is boilerplate Project BTW. https://github.com/johanbrandhorst/grpc-gateway-boilerplate/blob/master/proto/example.proto

[mosuka]

@joeblew99

The index mapping specified at startup determines the field name and type of the document. Therefore, since it is difficult to handle as Struct, Blast handles the document and as map[string]interface{}.

In Protocol Buffers, it is difficult to express map[string]interface{}, bleve.SearchRequest and bleve.SearchResult, so I use the Any message type.

[joeblew99]

@mosuka

Sorry i got side tracked on something else. It seems you did the commit anyway ?

[mosuka]

@joeblew99 I had implement grpc-gateway in v0.8.0. However swagger is still incomplete.

[gedw99]

I found a much easier solution to the grpc aspects.

Started to use envoy to run the grpc-web layer. It is very recommend.

Gloo from solo makes it easy.

We just run it with standard kubernetes.

See here : https://github.com/getcouragenow/packages/tree/master/maintemplate/server/deployments/helm/templates

Blast and cete are then much simpler.

We plan to stand up with in our cluster. I can send you you the PR when we are done with the line manifests to run it if you want ??

[mosuka]

@gedw99 Sure. That sounds great. PR for the grpc-web layer is welcome. :)

[joeblew99]

Ok will work on it.

It can also be used for that rust version . Can't remember it's name :)

Gloo can be used to run envoy for you. It's a golang wrapper for Envoy proxy and so makes it easy to scale without having to get into the complexity of kubernetes.

You can use it with nomad oncloud or onpremise to scale it out and drive it with consul

On Wed, 8 Apr 2020, 14:33 Minoru Osuka, notifications@github.com wrote:

@gedw99 https://github.com/gedw99 Sure. That sounds great. PR for the grpc-web layer is welcome. :)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mosuka/blast/issues/102#issuecomment-610931711, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC3RU474NEHLF3FFHZZJNYDRLRVIXANCNFSM4IKEDJ7A .