Feature/grpc - Githubissues

streamingfast / substreams-sink-kv

Substreams KV sink

Apache License 2.0

2 stars 5 forks source link

Feature/grpc #1

Closed sduchesneau closed 1 year ago

sduchesneau commented 1 year ago

Connect web integration

https://github.com/streamingfast/substreams/issues/94

substreams.sink.kv.v1.Kv.Get
substreams.sink.kv.v1.Kv.GetByPrefix
substreams.sink.kv.v1.Kv.GetMany
substreams.sink.kv.v1.Kv.Scan

Requirements:

[x] rename protobuf prefix under 'sf.substreams....'
[x] add usage examples with grpcurl
[x] add usage example with simple HTML/JS page
[x] add some doc and README regarding this integration
[x] add some flags for exposing this endpoint as SSL
[x] in the substreams-eth-block-meta (used as an example here), store the full protobuf object in the kv store
[x] in the web example, decode a protobuf object from the kv store

maoueh commented 1 year ago

What is the integrate with dauth goal? So that the endpoint is authenticated?

I think that SSL and dauth should be left out for now. We can showcase an example with Docker compose + nginx for reverse proxy and SSL if we want.

sduchesneau commented 1 year ago

What is the integrate with dauth goal? So that the endpoint is authenticated?

I think that SSL and dauth should be left out for now. We can showcase an example with Docker compose + nginx for reverse proxy and SSL if we want.

Two objectives here:

1) we don't have any ConnectWeb helpers for authentication or healthcheck. Adding connectweb server into our github.com/streamingfast/dgrpc package makes it easy for us to add more of those.

2) I'm certain that we will want to roll out a simple hosted kv store sooner rather than later. Having SSL+dauth integration will come up at that point. It is IMHO simpler to roll this out within dgrpc than to try to implement this with a front-end docker

maoueh commented 1 year ago

I'm certain that we will want to roll out a simple hosted kv store sooner rather than later. Having SSL+dauth integration will come up at that point. It is IMHO simpler to roll this out within dgrpc than to try to implement this with a front-end docker

If we do it ourself, pretty sure it will be through k8s, hence why I think to reduce our maintenance/surface area, there is no point to have the tool support SSL/auth directly. We can definitely ask people to secure it they want they want.

I saying Docker more for example purposes where people could see how to do it. I'm even in favor of not showing anything related to that and simply putting a note in the doc/readme that SSL/auth should be managed by something else.

sduchesneau commented 1 year ago

as discussed offline, the --listen-https flag is required for us to deploy behind google load balancer (for which a self-signed cert is sufficient), but I won't implement any auth at the moment.

sduchesneau commented 1 year ago

This should close https://github.com/streamingfast/substreams/issues/94