search

kadena-io / chainweb-stream

Streaming server for chainweb events
5 stars 1 forks source link

Request for Comments: General #2

Open Takadenoshi opened 1 year ago

Takadenoshi commented 1 year ago

We are prototyping a way of providing near-real-time updates for chain events to alleviate some of the inconvenience of polling chainweb-data and/or chainweb-node repeatedly.

Opening this issue as a place for potential users of this service to offer their thoughts and feedback.

Note that this is still in early development; we thought to reach out early in order to ensure that we will deliver something that fits your use cases as much as possible.

daplcor commented 1 year ago

I love the simplicity of the sse server and client. Given my specific requirements, I need help understanding why the data node is required. If it's for orphans or listening to sqlite writes to index(data node) and we look at things in near real-time, how does this differ from custom solutions that index without a data node? I'm interested in testing this out, but I have had a few issues with data nodes and opted to go with Mongodb and socketio to achieve a similar setup. In reality, I've handed my own data feeds out to ~5 projects and would love to figure out a way to offer a robust solution via subscription so projects can tap into stable feeds that I offer. I'd love to help and offer feedback, just need to understand if there's any way to achieve a similar result without relying on data nodes.

trendzetter commented 1 year ago

I commented about this before but I can't find my earlier comment, I think it was in a call instead of on github. I think the need for both chainweb and chainweb-data is a problem that will limit the audience for this software. Requiring both of the software entails a lot of redundancy, huge resources and maintenance. It should be possible to detect orphans without chainweb-data. but you can only know about an orphan when the next block is mined. So you would have to keep a little local db and check which of the blocks is referenced as a parent in the next block when there are more than one blocks for a block-height.

Takadenoshi commented 1 year ago

I opened an issue to hear specifically about this here. The cw-data dependency came about because of the historical context requirement we set for ourselves, but if it is not useful, we can look into rewriting chainweb-stream to use chainweb-node as the only data source.

It doesn't have to do with orphan detection, this happens through chainweb-node.