search

waku-org / pm

Project management, admin, misc
3 stars 1 forks source link

[Epic: js-waku] Reliability Protocol for Resource-Restricted Clients #143

Open chair28980 opened 6 months ago

chair28980 commented 6 months ago

Our user story:

  1. As a developer, I want a simple API to register hooks to process messages on one or several content topics: https://github.com/waku-org/js-waku/pull/2092
  2. As a developer, I do not care about shards or pubsub topics, I am only using autosharding
  3. As developer, I want fair reliability when receiving messages (e..g redundant filter nodes + filter ping + store sync)
  4. As a developer, I do not want the same incoming message to be processed several times (e.g. filter redundancy should be abstracted)
  5. As a developer, I want fair reliability when sending messages.

And our implementation notes:

Store

Combination

Liveness of a node:

Testing

Filter:

LightPush:

First revision ## Tasks (obsolete) - [ ] write interop tests to cover `non-relay client` user stories; - [x] protocol reliability + exposing additional configs to enable / disable behavior + unit tests: - [x] Filter/LightPush uses more than one peer: https://github.com/waku-org/js-waku/issues/1463 - [x] orient protocols to be more RFC oriented + unit tests: https://github.com/waku-org/js-waku/issues/1886 - [x] Filter: https://github.com/waku-org/js-waku/pull/1964 - [x] LightPush: https://github.com/waku-org/js-waku/pull/1887 - [x] Store: https://github.com/waku-org/js-waku/pull/1913 We can de-scope some of these streams in the interest of time.
Second revision #### Filter: - [ ] ensure number of peers to subscribe to is configurable, documented and respected by implementation; - [ ] Filter subscription recovery after offline state https://github.com/waku-org/js-waku/issues/2024 #### LightPush: - [ ] ensure number of peers to use is - [x] configurable - [ ] documented #### Store - [ ] implement `store-v3`: https://github.com/waku-org/js-waku/pull/2036 - [ ] peer renewal for store peers; #### Liveness of a node: - [x] https://github.com/waku-org/js-waku/issues/2070 - [ ] #### Testing - [ ] unit testing; - [ ] https://github.com/waku-org/js-waku/issues/2007
chair28980 commented 6 months ago

@waku-org/js-waku-developers please proceed with creating the tasks required in the js-waku repository, please tag them with the E:js-waku Improve Reliability label, and include them in Task list above in the description.

fryorcraken commented 4 months ago

@waku-org/js-waku-developers please note that we now expect an RFC out of this work. Also, need to understand what is done in status-go and align. @chaitanyaprem would be a good person to discuss with as he has some familiarity with usage of req-res protocols in status-go

weboko commented 4 months ago

in status-go

I was planning to schedule knowledge sharing after Status offsite is done

we now expect an RFC out of this work

after knowledge sharing and revisiting completed work we will discuss RFC internally

fryorcraken commented 4 months ago

is there an understanding of what of the change listed above would give you most ROI?

danisharora099 commented 3 months ago

is there an understanding of what of the change listed above would give you most ROI?

A lot of high-impact items have been merged, or are in progress. Other than that, imo:

weboko commented 2 months ago

Linking some relevant status-go issues that makes sense to take a look at in the scope of js-waku:

danisharora099 commented 2 months ago

https://github.com/status-im/status-go/issues/5344

Based on https://github.com/status-im/status-go/pull/5350 and https://github.com/status-im/status-go/issues/5344#issuecomment-2196347722. it seems that status-go did not filter peers based on shards when discovered using Peer Exchange, thus attempting to open subscriptions with peers with unsupported pubsub topics.

https://github.com/waku-org/go-waku/issues/1124

Investigating this: https://github.com/waku-org/go-waku/issues/1124#issuecomment-2196357378

https://github.com/waku-org/go-waku/pull/1128

Very interesting. Though we haven't run into a problem like this before (perhaps because of the nature of js-waku node being short lived, thus peer updating its addresses was never observed) -- but this seems to be a very important check to introduce. Opened an issue here: https://github.com/waku-org/js-waku/issues/2051

chaitanyaprem commented 2 months ago

This is the master issue i am using to track all fixes/changes wrt lightclient reliability being done in go-waku/status-go. https://github.com/waku-org/pm/issues/219 @danisharora099 , @weboko you can refer to this to see if something that can be applied to js-waku and also can be added to the rfc.

weboko commented 2 months ago

Another point to consider from @jm-clius comment here - https://github.com/waku-org/specs/pull/18#pullrequestreview-2144776176:

Let's discuss at js-waku PM meeting, @danisharora099

danisharora099 commented 1 month ago

Strictly based on the newly merged Reliability RFC, derived features and their current state in the js-waku codebase are:

Node Health Management

weboko commented 1 month ago

Nice, good job at summarizing! Let's discuss it at our PM meeting tomorrow as I believe some of the features we might want to make optional such as Store / Filter message verification and retry.

Also to note: we are waiting for update to Light Push to implement some of the strategies in the RFC such as renewing peers only in some cases and doing retries in others.

Upd#1: additional task https://github.com/waku-org/js-waku/issues/2074 Upd#2: one more https://github.com/waku-org/js-waku/issues/2076