search

streamingfast / substreams

Powerful Blockchain streaming data engine, based on StreamingFast Firehose technology.
Apache License 2.0
159 stars 45 forks source link

Support connect protocol natively (without proxy) #191

Closed fubhy closed 10 months ago

fubhy commented 1 year ago

I believe it would make sense for the substreams server to support the Connect protocol natively (without requiring a proxy). I see that there's already https://github.com/streamingfast/dgrpc/tree/develop/server/connect-web ... Let's use that? ;-)

fubhy commented 1 year ago

Note that the Connect protocol technically also suports serving the response in JSON directly. That feature would require the server runtime to have access to the full proto schema. Since we are not sending the full .spkg to the server, the DescriptorSet is not part of the request. Therefore, we do not have the necessary information to support this feature. Not sure what to do about that... Either just not support JSON requests (imho reasonable), or change the .blocks grpc method to receive the full package in the request. Alternatively, we can also add a second service method .blocksFromPackage (yikes), that can support JSON and leave the current .blocks method to only support binary requests.

sduchesneau commented 1 year ago

In the case of JSON, the server will send out json-encoded response with base64-encoded "protobuf.Any" bytes for the client to decode. There is no simple way for the server to do the decoding on-the-fly "per request" (the dynamic proto libs are usually set up to have a "global registry", which would cause many headaches).

Correct me if I'm wrong, but I don't think that decoding the app-specific payload server-side is actually needed for a good dev flow, the JS can do it on the browser.

I'm working on this issue right now, we have to change how the substreams apps (tier1 and tier2) are defined, because right now they are attached to firehose and we don't want to connectweb-ify the full firehose stack at the moment.

morethan5minutes

fubhy commented 1 year ago

In that case it is imho better not to support json wire encoding at all. I solved it in my proxy by adding the full package to the request but that is not desirable here.

matthewdarwin commented 1 year ago

Re:

I'm working on this issue right now, we have to change how the substreams apps (tier1 and tier2) are defined, because

It will be good to have the "tier2" identified as a separate app anyway. Right now looking at dashboard can't tell which "app=firehose" is a tier 1 vs tier 2.

fubhy commented 1 year ago

Hey @sduchesneau are you still on the toilet?

jubeless commented 10 months ago

@sduchesneau completed it!