search

input-output-hk / jormungandr

privacy voting blockchain node
https://input-output-hk.github.io/jormungandr/
Apache License 2.0
364 stars 132 forks source link

Move explorer to it's own binary crate #3227

Open ecioppettini opened 3 years ago

ecioppettini commented 3 years ago

partially replaces #2759 (bundling two things in one issue was a bad idea, existing discussion was on the other topic anyway).

About

Make the explorer a separate service instead of a mode of operation for the node (as it works right now per #694).

The idea is for this service to not be a participant on the p2p layer, but to hook to the node and receive messages in the same way that we do right now internally through the intercom. New blocks and tip changes are enough for the most part. Bootstrapping shouldn't differ much from the way the node does it, I think, except (probably) that we only have one node to pull blocks from.

Requirements

Other considerations

Per design, most of the explorer works independently from the Blockchain (basically the ledger), so migration shouldn't be a problem with just sending blocks (and tips).

There are currently two things that need access to the blockchain (and I didn't add them myself, so I'm not sure what to do with them yet). One of those is the Block::treasury query, which I think is used in tests at least. The other one is in the indexing part of the VoteTally certificate. I think this can be ignored for now, but we'll need to think if it needs something that should be provided through an api (I think it needs the tallies, basically), of if it is something that the explorer can keep up-to-date by itself.

eugene-babichenko commented 3 years ago

We may want to move some things to jormungandr_lib to keep some things consistent (logging setup, some settings, maybe some networking stuff or service stuff), but it's not needed from the beginning.

I am not sure about that part. jormungandr_lib has many clients and not all of them would need logging and networking things. Although we can just make these things feature-gated.

Just to make sure: you intend to use gRPC for explorer-to-node communications, right?

ecioppettini commented 3 years ago

I am not sure about that part. jormungandr_lib has many clients and not all of them would need logging and networking things. Although we can just make these things feature-gated.

Well, maybe you are right. It could be a different crate too. But overall I was speculating, maybe we don't need it. It's just that I was thinking than re-using some bits of the codebase could make things more consistent overall. There is a decent deal of tools and helpers in jormungandr, I think.

Just to make sure: you intend to use gRPC for explorer-to-node communications, right?

Yeah, that's what we talked about with @mzabaluev last time, at least. It's already implemented, and websockets are more fit to more user-level clients (with graphql, probably).

Although, nothing is done yet, so if you have any concerns or other ideas you can bring them up.

mzabaluev commented 3 years ago

I support using gRPC for inter-component subscription channels.

BlockService could provide the necessary functionality for bootstrapping and subscription to new block announcements. The server-side handling of client connections assumes p2p nodes indeed. But as long as the client behaves as an anonymous node, does not subscribe to things it does not need (e.g. gossip), and does not send events up the subscription streams, it should be possible to accommodate non-node clients using this protocol.

mzabaluev commented 3 years ago

One thing that BlockService does not currently provide is tip change notifications. I think this could be added backward compatibly to block announcements on the server side, but the explorer could also apply tip selection logic on its own, and the result should be the same.

Previously, there were requests to accommodate thin clients via grpc-web and bidirectional subscription requests were a sticking point as they are not supported in grpc-web. Maybe its time to get the node a non-p2p service endpoint to address the needs of such clients.

eugene-babichenko commented 3 years ago

But as long as the client behaves as an anonymous node, does not subscribe to things it does not need (e.g. gossip), and does not send events up the subscription streams, it should be possible to accommodate non-node clients using this protocol.

This is calling for introducing some sort of filtering. Or at least we need to explicitly state in the documentation that we allow anonymous connections to BlockService, and thus the endpoint may be needed to be protected by a firewall if a node operator doesn't like that.