Closed jm-clius closed 2 weeks ago
Thanks for the issue!
Regarding ethdenver requirements: Which also applies to other workshops and demos. I would say we need the following:
--dns-discovery-url
.A quick solution for ethdenver would be to use wakuv2.test
and subscribe them to a different gossipsub topic. With this, even if they end up connected to status fleet, we won't have all traffic present in the main topic, which fixes 4). Note though that since peer discovery nor peer management knows of topics, we can end up connected to eg 20 peers, but none of them support the topic that we are interested in, and afaik we won't know it.
duplicate messages received by Filter clients: slow Status nodes could experience such a large latency before relaying messages that they fall outside of the gossipsub retransmission window of neighbouring peers, which results in the same message being relayed and published to filter clients more than once.
Would we want to use https://github.com/waku-org/pm/issues/9 to avoid the dupe messages in filter? Should I just create an issue for the dupe message and we can track there? We are seeing this problem in js-waku examples.
I agree for a separate pubsub topic for development purposes.
In js-waku, we have a defaultBootstrap
option to enable developers to connect to the Waku network.
This is especially useful in the context of hackathon, workshop, PoC, so that developer can get going and have a reliable experience out-of-the-box.
We currently connect to the wakuv2.prod fleet when such option is enabled. Similarly, if no pubsub topic is specified, js-waku uses the default pubsub topic.
Which such options, developer could use the bootstrap option whether they were doing a hackathon, PoC or promoting software to prod.
Now, by introducing a different pubsub topic for development, it creates some friction: when they want to promote their software to prod, they'll have to specify the "prod" pubsub topic. I think that's fine. Thoughts @hackyguru ?
A quick solution for ethdenver would be to use wakuv2.test and subscribe them to a different gossipsub topic.
Related to my comment above, js-waku default bootstrap option connects to the wakuv2,prod fleet by default. If only the wakuv2.test fleet subscrives to the dev pubsub topic, then we'll have to change default values in js-waku.
For stability reason and to ensure that developer's first experience with Waku is best, I would suggest that we subscrive the wakuv2.prod fleet to the dev topic too.
Set of at least 6 nodes supporting all protocols, which is the minimum amount we need for gossipsub.
Recommended is 6
, lower bound is 4
: https://rfc.vac.dev/spec/29/
These fleet nodes should ideally be stable so that we can run demos using them
That's why I think it should be wakuv2.prod fleet, not test.
That's why I think it should be wakuv2.prod fleet, not test.
Sure, that's fine. And way better indeed, since wakuv2.prod does not restart on every merge to master. Only thing left for ethdenver would be to have a separate topic to avoid having lots of traffic (since that fleet is connected to status.prod right now). Understand the default gossipsub topic is hardcoded, we can discuss this in our brainstorming session.
I agree for a separate pubsub topic for development purposes.
I brought this up a few weeks ago with @jm-clius but never really gave a detailed explanation of my idea. I think in the medium term (since it shouldn't be that difficult to implement), we should have separate and segregated environments (not just pubsub topics).
A "production" (or mainnet) and a "staging" (testnet) one. Similar to what blockchains have (mainnet vs testnet). Over the longer term when we add (des)incentivization these networks will perhaps relate to the existing Ethereum mainnet and (Goerli most likely), but it's too early for this, leaving that aside by now. So I think we need:
Reasons why I think we need separate networks (and not just a dev-topic)
shard-test-0
prefix, but things starts going crazy here.A non-exhaustive list of how to achieve this:
network-id
to the message.network-id
to each client implementation (go, waku, js)!=network-id
the message is not relayed nor stored.network-id
key to the ENR. If I discover a peer that !=
my network-id I skip connecting/storing it.network-id
, 0x01
mainnet? 0x02
testnet?What do you think? Apologies if this derails from the main topic, can move it to another thread. "Waku fleets: separate networks production/development".
Some responses :)
A quick solution for ethdenver would be to use wakuv2.test and subscribe them to a different gossipsub topic
While I agree with doing this on wakuv2.prod
, I see no reason not subscribe wakuv2.test
to that topic as well. The demo can point people to wakuv2.prod
.
Note though that since peer discovery nor peer management knows of topics, we can end up connected to eg 20 peers, but none of them support the topic that we are interested in, and afaik we won't know it.
Indeed! Gossipsub should take care of building the mesh as long as the total number of peers is still relatively low and we can discover and maintain connectivity to "all peers", but this will become an issue in future for multiple networks/shards.
A "production" (or mainnet) and a "staging" (testnet) one. Similar to what blockchains have (mainnet vs testnet).
I roughly agree with your reasoning here for separate networks in the medium term. What I think would be a sufficient definition for what you term a separate network is one which is:
I think the static sharding strategy would provide us with this functionality? We could e.g. have a static shard formally allocated for a mainnet and one or more shards for testnet(s)? This would imply that network-id
as you describe above would be the rshard
ID. The plan is already to have this be a dimension in discovery to skip connecting/storing it. Furthermore this allows gossipsub rules to determine routing and mesh creation.
Linking relevant issue: https://github.com/vacp2p/research/issues/174
While I agree with doing this on wakuv2.prod, I see no reason not subscribe wakuv2.test to that topic as well. The demo can point people to wakuv2.prod.
Yes agree. Disregard my comment :)
I think the static sharding strategy would provide us with this functionality? We could e.g. have a static shard formally allocated for a mainnet and one or more shards for testnet(s)? This would imply that network-id as you describe above would be the rshard ID. The plan is already to have this be a dimension in discovery to skip connecting/storing it. Furthermore this allows gossipsub rules to determine routing and mesh creation.
I'm not sure it would be a good idea to have "test shards", since imho mainnet/testnet should be a mirror/copy of each other to ensure that what works in one, works in the other one with just a minor tweak (change in network-id
). For example, what will happen when we have automatic sharding? How will the "automatic shards" work in testnet if the testing environment is already a shard? Will we have 65535 "production shards" and then other 65535 shards in the same network but with different name? Seems odd to me, but I'm heavily biased by blockchains :P
Some related thoughts here
Short term solution to fix this problem with all gathered feedback. wdyt?
wakuv2.prod
and said new topic to anyone prototyping with waku.Would this actions points comply with the requirements of this issue?
Currently blocked by: https://github.com/waku-org/nwaku/issues/1545
Please don't use spaces to separate values in CLI flags.
I am not sure this can be closed just yet. I think we need some follow-up actions:
Should we make js-waku examples use the dev pubsub topic?
@fryorcraken Good points. Can we make a combo box in js-waku
to pick between default/dev pubsub topic to publish/subscribe? This would be useful for demos since dev topic has almost not traffic. Can't image 100 people subscribing in a venue to the default topic. Initial plan was to use dev topic for eth denver cc @danisharora099
Should js-waku, nwaku, go-waku use dev pubsub topic when no topic is specified?
Since the spec only talks about the default pubsub topic, I would perhaps leave it as it is. Meaning when no topic is specific just subscribe to the default one.
@fryorcraken Good points. Can we make a combo box in js-waku to pick between default/dev pubsub topic to publish/subscribe? This would be useful for demos since dev topic has almost not traffic. Can't image 100 people subscribing in a venue to the default topic. Initial plan was to use dev topic for eth denver cc @danisharora099
Combo box is not really the way forward because that's not something a developer would want in their app, it just creates more confusion.
Since the spec only talks about the default pubsub topic
Good point. I think this warranties a RFC update to mention that the dev topic is recommended for development purposes. Once RFC is updated, then I think it would be a good point in time to updates js-waku examples to dev topic. Possibly update chat2 too? js-waku dev can then take over to decide how we want to update the API for that.
Re-opening as I believe we have pending items,
From Discord:
Ok we said that we should make the dev topic by default on dappnode for nwaku.
Not much point doing that if nwaku on dappnode are the only nodes running on dev topic (with the fleets).
If we decide to use a dev network I think we need to do it properly across our ecosystem.
Trying a more constructive/direct approach. Here is a proposed action plan:
The initial purpose of this change was simply to have another pubsub topic available to developers (specifically aimed at DappNode and the ETHDenver demos). While I agree that we can document its use better, I think fully switching our strategy from default -> dev becomes less important once static sharding for communities etc will be in place, which should be soon-ish. Happy to follow a more comprehensive approach if that's what we agree on.
Right.
The issue is that the usage of the dev pubsub topic has already leaked beyond EthDenver as it is made as the default pubsub topic in nwaku DappNode package.
Something I don't fully agree with but was push to avoid issue with users staking Eth.
Hence we have already started switching strategy from default to dev. Any thoughts?
@fryorcraken closing this issue as not relevant given current shards approach.
Problem
Dogfooding of Status Communities has recently started. Since Status currently uses the default Waku pubsub topic:
The Waku fleets (
wakuv2.prod
andwakuv2.test
) are currently only subscribed to this same default pubsub topic. Everyone using the default pubsub topic plus connecting to one of the Waku fleets will therefore be sharing (1) traffic and (2) infrastructure with the Status community dogfooding.This is potentially an issue for: (1) other operators using the default network and who do not want to deal with Status community artefacts on the same network (2) Status community dogfooding being affected by Waku testers/operators injecting traffic and testing on the same default network (3) testers wanting to tryout Waku who do not have a "clean" network to bootstrap to (e.g. during workshop planned for ETHDenver)
Issues reported
Some issues reported that could be caused by Status Community traffic on the default network:
Longer term solution
Vac Secure Messaging is currently working with Status app to:
Interim suggestion (named shard)
Subscribe Waku fleets to one more pubsub topic for developers/operators/testers to use and remain unaffected on gossipsub level by Status traffic and vice versa. This pubsub topic could for example be
/waku/2/dev-waku/proto
. Fleet nodes would remain subscribed to both this dev topic and the default pubsub topic. However, Waku nodes could choose to subscribe to/waku/2/dev-waku/proto
only and be separated from those (including Status Communities) on the default network.What this does not solve:
Separating out the discv5 layer and differentiating networks/pubsub topics on discovery level. This would require the static
rshard
dimension which is being developed for the longer term use case. Nodes subscribed to the dev network will still discover and attempt connection to nodes in the default network - however, they will only form a mesh around other peers in the dev network.Why not simply have a "separate fleet"?
Separate fleet does not translate to separate network. Since we don't have a discv5 mechanism yet to differentiate networks/shards on the discovery level, fleets will likely still discover and connect to each other until this is in place.
cc @alrevuelta for ETHDenver requirements