Nomos requirements

[fryorcraken]

High level requirements for the Nomos Project:

1. Validators are anonymous within a larger set: of the nodes participating in a larger network, you don't know which ones are part of the Nomos network.
2. You can't analyze network patterns to extract some information about the Nomos network
3. Spam protection: limit the rate of communication for DoS protection.
4. We want to make sure that the leader is not known by the entire network and/or cannot be DoS'd

[alvatar]

Note that we might want to create overlay networks that do not have these requirements so they can have more performant networking. In other words, these different privacy measures ideally are optional and modular.

[madxor]

Just a couple of comments/thoughts from my end.

Can we state what is provided at which abstraction layer? In other words, are we considering that all of these requirements are (or going to be) fulfilled by Waku (network layer)?

Re 1: Is it correct to read it as "Nodes are anonymous, where anonymity is defined as unable to distinguish the function of the node in the network", if so then then this should also include the "network patterns indistinguishability" by definition. Just arguing that cannot achieve 1 without 2.

Re 3: Spam protection might be tricky if we want to achieve high responsiveness of the network. This requires alignment with needs of the consensus algorithm (or at least our expectations for it). Also if we achieve node anonymization then DoSing the network would be seen as a malicious from the perspective of the network but could not be use as a selective tool for turning off a particular node (leader).

Re 4: I think that this is the crux of all of it. We want to have a leader protection mechanism. Which logically also extends to all nodes... And we it can be achieved whether through anonymization or DoS mitigation or both.

To summarize, I see that we have two features: anonymization (levels of it?) and DoS protection (tunable?) provided by the network layer. Which to Alvatar comment, can be turned on or off in order to fulfill the modularity needs.

[alvatar]

The main problem is that we don't know how Carnot, Waku and Carnot+Waku are going to perform, so we are bit blind still. It's too many variables and too early to throw numbers. But what I can say now is:

• I'm assuming the Base Layer will have high latency, it's the cost of a large validator set using Carnot and more so with privacy. Low latency could only be achieved in execution zones. See the Architecture Whitepaper for details.
• Anonymity is not something that can be turned on/off. If you anonymize the base layer but then deanonymize the EZs then you deanonmize the base layer since participants in the EZ also participate in the BL.

Finally, while leader protection can be potentially be solved with privacy at the P2P layer, it's not the main target. The main target is to make participation in the Nomos network anonymous (which is challenging and works against our wish to optimize the data throughput of the entire network). Both 1 and 2 are about this. Number 3 is lower in the priority list, and it's just about p2p protection beyond dropping connections from spammy peers. Number 4 is lowest in the priority list because we could aim at solving it with SSLE at a higher layer, and my guess is that any p2p mechanism that preserves the knowledge of a P2P node to a handful of other nodes will suffer to even greater latency. If every node knows about every other node, then there is no P2P leader protection beyond spam protection via RNL.

Am I missing something?

[MForensic]

I would like to add that there is a possibility of having a relatively efficient way of spam protection similar to the Ethereum network as mentioned by @ksr on Discord. Since the pattern of messages in consensus protocol is predictable the gossipsub relays can detect anomalies to prevent DoS attacks. But adding anonymity can make this complicated.

[alvatar]

Yes, that's exactly the point. In principle, we might not need extra spam protection.

In other words, (1) and (2) are the most unique benefits Waku could bring IMO.

Now this said, there is a specific use case for Spam protection that I'm considering, and why I add it to the list:

MEV. With a strong focus on privacy in the execution layer, we basically shift MEV from frontrunning and sandwhiching to exclusively latency-based backrunning. This means: latency wars. This is gonna happen as soon as we have some degree of success. So even if transactions cost money (even if they fail), we want to make sure we have an extra way to limit them, or make them more expensive when they congest the network. At the very least we will tackle this with the fee system, but we need to make sure there is no mechanism by which MEV searchers can post many transactions and freely/cheaply see them fail. Attempts should cost, or at least past certain use. The hard part is that if we make a scalable system with cheap fees, this works against spam protection, and vice versa, so perhaps there is a way to improve on this by using a p2p layer spam protection mechanism (or that is my intuition at least).

[madxor]

I'm wondering how much control would one have over latency. We are randomly selecting a leader on top of randomly created overlay on top of an anonymizing P2P network. In such setting the only strategy would be to flood the network with large number of requests, from various entry points, hoping that one of them will win. Which means that if the network is larger then the price of executing such attack will also rise. At the same time I don't think that we can effectively limit this behavior for an entity who would spam the network from many wallets and many endpoints - that would be indistinguishable from normal traffic.

Just speculating here, on one specific case.

[alvatar]

That's true, but things get interesting when there is a large staker, and any centralizing force (usually an innovative idea, protocol, or use case/behavior). A typical one would be a liquid derivative protocol that is not decentralized. If that protocol decides to make deals directly with MEV searchers, that's how you have that sort of situation. It's pretty much like HFT in TardFi. So this is mainly a concern if there is an identifiable and centralizing force that can be targeted somehow, via meatspace deals. That's the case in Cosmos, due to their small validator sets. In our case it would be harder, but we should never underestimate the appearance of some centralizing force that would then open the door to the latency wars.

The other type of MEV that would still be present is PGA backruns. Since we still have no clue of our tokenomics, it's early to discuss, and probably there won't be anything to be done at the p2p layer. PGA pollutes the blockchain and raises fees for everyone, so we'll have to think about it.

[alvatar]

This said, I think chances are low that we can use Waku in a larger network, since we need to have high base layer throughput. Belonging to a larger network means that we would be relaying the traffic of other applications, and I certainly don't think we can afford that with our current node requirements. Maybe @fryorcraken has something to say about this?

[alvatar]

@fryorcraken could it be made so that nodes decide whether they want to belong to a larger anonymity network set (thus increasing their bandwidth requirements). I think with this we could have the best of both worlds:

• Not increase the minimum bandwidth for a validator
• If the user has higher bandwidth and chooses to, can have higher validator anonymity.

I'm thinking that a simplified version of this is to have two networks: an anonymous one and a clear one. Nodes that belong to the anonymous one also belong to the clear one, so we ensure relaying between the two.

[fryorcraken]

Some high level comments:

Anonymity

It looks like sender anonymity is very important to Nomos. Both to protect the leader but to make participation in the Nomos network anonymous. Receiver anonymity being needed for participation in Nomos network too.

In both cases, it's a question of providing k-anonymity for a given actor.

For leader protection, I assume that k-anonymity for the leader among Nomos nodes is what we want.

For general participation, then I assume that we want k-anonymity for participant among a wider general purpose network such as Waku (or a mixnet such as Tor/Nym/Hopr).

I understand @kaiserd has identified Tor Push [1] being more promising than Dandelion++ [2] to deliver sender anonymity.

Note that the idea is to use Tor only to push messages to the network to compensate for the fact that gossipsub, and hence Waku Relay, does not provide sender anonymity. However, it does provide receiver anonymity [3]. Note this article also contains an attack-based threat analysis worth reading.

Do note that k-anonymity is available within a pubsub topic used for routing, which is different (on purpose) from the content topic used for content addressing.

Sharding

In terms of latency, we could define several pubsub topics for the Nomos application to turn dials on the anonymity trilemma (more specifically the latency vs anonymity dial). This comes with some caveats of course as as soon as we use a specifically designed pubsub topic then we loose receiver anonymity.

However, I wonder if as part of our auto-sharding strategy, we could have different kind of max participant requirements for different shards. E.g. some shard could be capped to 10k users while other shard would be capped to 1k users (users may not end up being the metric being capped). Meaning that different level of anonymity and latency can be provided by different shards. This in response to https://github.com/waku-org/pm/issues/22#issuecomment-1541544074. @alrevuelta @jm-clius WDYT?

One note to help with the mental mode: one node can participate in several shard. A shard is just a gossipsub network, defined by a pubsub topic. With auto-sharding, we are looking at deriving pubsub topics from content topics while preserving k-anonymity.

Finally, I believe we are also considering how dynamic could auto-sharding be to react to network load. @alrevuelta @jm-clius to confirm I am up to date on this idea.

Hopefully this helps answer https://github.com/waku-org/pm/issues/22#issuecomment-1544249890

User Choose Pubsub Topic

Regarding https://github.com/waku-org/pm/issues/22#issuecomment-1548367550

I'm thinking that a simplified version of this is to have two networks: an anonymous one and a clear one. Nodes that belong to the anonymous one also belong to the clear one, so we ensure relaying between the two.

This is an interesting idea which is new to me.

As discussed with sharding, the idea of auto-sharding + spam protection would be to provide some guarantee on the max bandwidth that a node is expected to use.

If a node wants to limit the bandwith they use then we usually recommend using light client protocols: light push [5] + filter [6].

This comes with some anonymity drawback (if used without a mixnet) as the remote node would be able to:

1. know that a given IP is originator of a message. Linking a specific content topic with said message. Depending on your usage of content topics, it could enable identifying an outgoing transaction or a leader.
2. know what content topic a given IP are interested in, thus identifying a Nomos participant, and again, depending on your usage of content topic, identifying a leader.

(1) can be resolved by using Tor push as previously mentioned Regarding (2), I think an analysis of dedicated pubsub topic vs filter would be needed as in both cases, you would loose k-anonymity re participation in Nomos network. Cc @alvatar @jm-clius

Spam Protection

The idea is still in its infancy but we are looking into defining a modular/plug-in API, to define spam protection. RLN would be plugable to nwaku in one (or several) plugins. This would enable applications (such as Nomos) to define specific spam protection logic. Which could make sense as mentioned previously. We are still to define how this would work, especially when an application needs k-anonymity. Defining a Waku Network MVP [4] is the first step towards this goal.

References

1. 47/WAKU2-TOR-PUSH
2. 44/WAKU2-DANDELION
3. Waku Privacy and Anonymity Analysis Part I: Definitions and Waku Relay
4. https://github.com/waku-org/research/issues/1
5. 19/WAKU2-LIGHTPUSH
6. 12/WAKU2-FILTER

[alvatar]

Thanks @fryorcraken , much appreciated.

Just to recall important design requirements that we have that are on the critical path of Nomos:

• Throughput. Nomos has a Base Layer that focuses on data availability, and it's global (similar to Ethereum, Celestia). This means that the total throughput of the network is given by the constrain of the minimum network rate that we impose to each validator, and how we build the data sharding (which data is where so it can be retrieved). We could use the same network constraints as Ethereum (500kb/s per node) as a starting point. This is why I say that we could have two networks, so that the validators that can provide 500k/s and don't need anonymity, can still participate, but those that are ok with higher network usage can obtain k-anonymity.

• Latency. Latency is definitely an issue at play here. It will affect the overall network latency, which is already quite high (need proper measurements still though). I believe this should be ok for the Base Layer, but then higher layers would definitely need to reduce this latency. The problem is that the nodes that uncover their identity in the execution zones (as they need low latency), will be exposed as participating in the network (as they also participate in the Base Layer).

• Sharding? For consensus, we need enough flexibility as to be able to communicate all nodes with each other (ie no network partitioning). Maybe this is obvious already, but just wanted to make sure. I'm not familiar with how sharding in Waku works, as it sounds like network partitioning.

If Spam protection is in its infancy, it's something that we can for now assume as unavailable, so we can move ahead with that assumption for our design and implementation.

[jm-clius]

A couple of comments which I hope would help clarify some points:

Meaning that different level of anonymity and latency can be provided by different shards. I believe we are also considering how dynamic could auto-sharding be to react to network load

Autosharding is not fleshed out as an idea yet, but the idea is that applications would not have to consider sharding in their designs at all and simply focus on content topic design. There is no current plans to provide any form of service differentiation on the different shards, but simply to use it as an auto load-balancing method based on bandwidth/traffic rates per content topic. For now I'd try to keep the latency and anonymity requirement in balance for each shard in the designs.

This also means that

we need enough flexibility as to be able to communicate all nodes with each other

should be a given: nodes will be able to send and receive the content that they want without considering which network "partitions" are involved.

[alvatar]

Awesome, that clarifies that point. Thanks @jm-clius

BTW wanted to mention that the idea of a node participating in both the anonymous and clear networks as a way to give optionality is not that simple: participating in the clear network makes participating in the anonymous one useless. Some nodes might be ok with this, to provide this service to anonymous validators as bridge, but is less than ideal. Perhaps we can think harder how to solve this.

But in any case these are just ideas to bring up the topic that ideally the validators can choose if they are anonymous or not, so we don't have to reduce the total network throughput, nor increase the minimum requirements. Or perhaps the sharding actually solves this while allowing all nodes to be anonymous without opting-out and without increasing the bandwidth requirements for home stakers.

[alrevuelta]

My 5 cents. Without having much knowledge about nomos/carnot, wondering if it would make sense to use waku:

• spam protection: Protecting against spam in blockchains is way easier than in waku, since in the former, the protocol has very clear rules on which messages to expect and when. For example, Ethereum has very clear rules on whats "spam" an whats not, which makes spam protection built in the protocol. If I were a blockchain, I wouldnt delegate spam protection to another project, since imho its very coupled with the blockchain rules. Actually, waku spam protection mechanism might end up piggybacking on ethereum. So nomos->waku->ethereum. I guess nomos aims to be a Layer 1? So guess it should be sovereign and not relay in another blockchain, which would be more a like a L2?
• gossip sub: If you were to use just gossipsub, would you be lacking any feature? At the end waku is essentially gossipsub with some stuff on top, but before thinking about using waku, I would asses if gossipsub is just enough.
• store protocol: Waku comes with a store protocol that stores messages for nodes that are offline. But with no consensus nor guarantees. Dont think you need it, since I assume the nodes will be storing the state themselves, and serving it to new nodes in the network. I wouldn't delegate this storage to a 3rd party, as imho it should be part of the core protocol.
• integration: Main waku implementations are in nim and go, but can see that nomos use rust? While there are multiple ways of integrating it, if I were to design a blockchain I would prefer native integrations, same language if possible.

Waku is designed to be generic, but I'm not sure it fits projects like nomos or even codex like some people wonder. imho waku is generic for app level integrations, but not for "core" protocols, blockchains, etc.

cc @alvatar

[kaiserd]

@alvatar @fryorcraken @jm-clius

Some points I'd like to add: (I did not read the Nomos Whitepaper yet, but I will soon, and come back here. These are just some thoughts I had while going though the comments on this issue.)

Anonymity

1. Validators are anonymous within a larger set: of the nodes participating in a larger network, you don't know which ones are part of the Nomos network.
2. You can't analyze network patterns to extract some information about the Nomos network

These are very difficult to align with other latency and bandwidth requirements. (more see section "QoS for Nomos")

4. We want to make sure that the leader is not known by the entire network and/or cannot be DoS'd

Tor push, which @fryorcraken mentioned, is a first relatively simple step in this direction. Here is my ethresear.ch post describing this idea. Vac deep research will soon continue (once new hire is onboarded) research in this area. Next steps here are:

• further evaluating Tor push
• continue Tor push Push PoC (towards Ethereum for now)
• combining Tor push with RLN to account for the spam protection problem introduced with anonymity All these steps entail latency and bandwidth overhead analyses. Future work will be in the direction of custom anonymity network solutions, as using Tor comes with its own set of drawbacks.

while leader protection can be potentially be solved with privacy at the P2P layer, it's not the main target.

Number 4 is lowest in the priority list because we could aim at solving it with SSLE at a higher layer, and my guess is that any p2p mechanism that preserves the knowledge of a P2P node to a handful of other nodes will suffer to even greater latency. If every node knows about every other node, then there is no P2P leader protection beyond spam protection via RNL.

The best approach seems offering as much anonymity protection on the P2P layer as can be done at a reasonable latency and bandwidth cost. (I know, this is vague still, we have to do further analysis to identify sweet spots.)

ideally the validators can choose if they are anonymous or not

The Tor push solution would allow that. But, would it not be a problem if for some validators the next-block validator can be DoSed? Maybe lead nodes would have something like Tor push as a requirement. (I need to read the paper give more meaningful input here.)

QoS for Nomos

Belonging to a larger network means that we would be relaying the traffic of other applications, and I certainly don't think we can afford that with our current node requirements

This basically renders 1) infeasible. It looks like the Nomos has very specific and hard requirements on latency and bandwidth consumption for one of its layers (which is expected). I still have to look into the network layering idea in Nomos, however, afaict Waku might not be the right (high requirements) P2P layer for Nomos. Gossipsub (with custom adaptions) might be the better solution. Mechanisms like Tor push are generic enough to plug into both gossipsub directly as well as Waku.

This would be in line with the Ethereum trinity: Ethereum, Swarm, Whisper -> Nomos, Codex, Waku. The messaging component (Waku) does not have to be the P2P layer the Consesus protocol lives on.

For other Nomos layers, Waku could be used. Waku can help with requirements 1) and 2) for these layers.

Sharding

This also means that

we need enough flexibility as to be able to communicate all nodes with each other should be a given: nodes will be able to send and receive the content that they want without considering which network "partitions" are involved.

While this is very convenient, and provides good k-anonymity properties, it would still increase the bandwidth requirements, and would not fulfill this:

Belonging to a larger network means that we would be relaying the traffic of other applications, and I certainly don't think we can afford that with our current node requirements"

But in any case these are just ideas to bring up the topic that ideally the validators can choose if they are anonymous or not, so we don't have to reduce the total network throughput, nor increase the minimum requirements. Or perhaps the sharding actually solves this while allowing all nodes to be anonymous without opting-out and without increasing the bandwidth requirements for home stakers.

Automatic sharding would offer k-anonymity; but it would increase the bandwidth requirements of home stakers as they relay traffic from other (non-Nomos) apps, too.

[kaiserd]

I also agree with all of @alrevuelta 's points :). They were on my list, too. (The post appeared while I was typing mine :sweat_smile: )

[alvatar]

I agree 100% @alrevuelta Definitely Nomos cannot rely on Ethereum. I was aware of this, just wanted to leave it for the future and perhaps solve it circularly with Nomos itself. In any case I agree completely that spam protection is not that useful for Nomos, because we can just rely in principles in the reputation mechanism of libp2p and on the fees for actual transactions.

• The store protocol is not very useful for the validators in principle. If validators are not online then they fail to attest blocks. It could potentially be useful for catching up when temporary offline, but in this case probably other protocols are going to be more adequate (such as just downloading from known peers the missing blocks)

• Indeed, we use Rust in our current node, and ideally the p2p library is in Rust. Being such a foundational piece, having it going through bindings is less than ideal.

[alvatar]

Thanks @kaiserd, I also agree with your points.

In my opinion, to summarize this long discussion (for which I'm greatly thankful, this conversation is really fruitful), I see the greatest potential of Waku for Nomos in the following:

K-anonymity for validators (with automatic sharding). Ideally made optional as described above (is it possible at all, or must be an all-or-nothing?). We could then measure the actual impact in the performance of the protocol and the bandwidth of the validators. It's hard to know without actual measurements, so anything that helps in this direction could let us anticipate (such as: will there be a spectrum of possibilities, or, you have already knowledge of the actual multiplier in bandwidth usage and latency that Waku introduces)

If Waku can do what is described above, it's useful for Nomos. Otherwise it would be an extra layer + bindings that we would not be really taking advantage of.

[fryorcraken]

Some points:

Regarding network partitioning. A node can be in several partitions. My understanding from Bogota re block chunk propagation:

• leader sends block chunk to committee members
• leader sends block chunk to leaders of child committees For this specific process, then you don't need everyone to communicate with everyone for every message. Partitioning might assist with bandwidth usage and latency.

Regarding using of Waku vs libp2p. Indeed, as summarized it depends whether for anonymity purpose, Nomos nodes must be part of a larger network.

However, even if the decision is to go for a separate libp2p-gossipsub network, it may be worth considering if there are any other Waku protocol that might be useful (such as sharding, or light client protocols).

Actually, waku spam protection mechanism might end up piggybacking on ethereum.

If you mean that some spam protection may used deployed EVM contracts, then I would expect to deploy similar contracts on several chains to increase potential user and app usage, including Nomos chain. As @alvatar mentioned, this ccould be resolved circularly by using contracts deployed on Nomos.

[MForensic]

Current Carnot design utilizes simple block propagation without network coding. We rely on the underlying P2P network for block dissemination across the entire network. This approach leverages existing networking infrastructure for block propagation. While network coding techniques offer benefits, our focus is on simplicity and utilizing the capabilities of the P2P network. Future enhancements may explore network coding for further optimization.

[fryorcraken]

We rely on the underlying P2P network for block dissemination across the entire network

Does entire network also include resource-restricted clients? Is the intent to enable mostly offline clients to participate via classic JSON RPC infra or would they somehow be part of the p2p network? If so, would they receive and process succinct data?

[MForensic]

What I meant from my previous comment was that we assume that there is an underlying P2P network that will take care of block propagation. We haven't yet discussed the detailed specification for such a network. Though currently, we are discussing the privacy and anonymity requirements for such a network.

[fryorcraken]

K-anonymity for validators (with automatic sharding). Ideally made optional as described above (is it possible at all, or must be an all-or-nothing?). We could then measure the actual impact in the performance of the protocol and the bandwidth of the validators. It's hard to know without actual measurements, so anything that helps in this direction could let us anticipate (such as: will there be a spectrum of possibilities, or, you have already knowledge of the actual multiplier in bandwidth usage and latency that Waku introduces)

@alvatar, we aim to launch the Waku public network MVP at the end of the year. It will include a rate limiting protocol so that users can have predictable and reasonable bandwidth usage.

We have laid out the characteristics of this gen 0 network here: https://rfc.vac.dev/spec/64/ This is the first versions and we are open to change some of the parameters.

I would be keen to have your thoughts on the matter to understand if this would fit Nomos' usage and expectation in terms of bandwidth usage.