Closed cortze closed 3 months ago
Just a note:
go-libp2p-pubsub will drop IHAVEs for messages not in a topic it is a part of: https://github.com/libp2p/go-libp2p-pubsub/blob/master/gossipsub.go#L654.
This means unless the go-libp2p node is subscribed to the empty topic, it will not handle an IHAVE message with no topic set.
@MarcoPolo is right, it doesn't seem to have any direct impact on the remote peer's side.
However, this would also mean that teku
in this case, wouldn't be helping their mesh peers on the Gossiping feature of GossipSub, as these empty topic message_ids would be dropped
Hm, that looks weird actually...
In jvm libp2p we are just ignoring IHAVE
topic field. That means we never set it for outbound messages and just ignoring them on inbound messages.
(however we should not send IHAVEs to those peers who are not subscribed to a corresponding topic. )
From my memories in the Ethereum network only one implementation was setting up the topic
field for IHAVE
s. Others were ignoring it as well.
Just to double check I've added the following test and it passes for me:
@Test
fun `check that resulting IHAVE has no topics`() {
val partsQueue = TestGossipQueue(gossipParamsWithLimits)
partsQueue.addIHave(WBytes(ByteArray(10)))
val res = partsQueue.takeMerged().first()
assertThat(res.control.ihaveList.get(0).hasTopicID()).isFalse()
val serialized = res.toByteArray()
val desrializedRpc = Rpc.RPC.parseFrom(serialized)
assertThat(desrializedRpc.control.ihaveList.get(0).hasTopicID()).isFalse()
}
I just thought that it could be the case that java proto library treats empty/null field differently @cortze by any chance do you have a message dump? I would check that option
In jvm libp2p we are just ignoring IHAVE topic field. That means we never set it for outbound messages and just ignoring them on inbound messages.
Does that mean that all seen message_ids are shared on a "single array"?
(however we should not send IHAVEs to those peers who are not subscribed to a corresponding topic. )
It doesn't seem to be the case; all the message_ids shared by the jvm-libp2p
peers belong to topics we were subscribed to.
However, this seems to be an implementation mismatch, as at least these do share message_ids by topics:
I just thought that it could be the case that java proto library treats empty/null field differently
It totally could be. Is there any particular reason not to differentiate all the message_ids shared on IHAVEs by topic?
Our only concern here would be that because the topic is empty, as no one is subscribed to an empty topic, those messages are getting directly dropped - expected behaviour, at least at the go implementation.
This makes peers using the JVM-libp2p
host not contribute to the Gossip part of GossipSub.
This makes peers using the JVM-libp2p host not contribute to the Gossip part of GossipSub.
Well, this issue is probably worth to be fixed then.
Just curious how implementations use the inbound IHAVE topic field? Aside of filtering unknown topics (which basically shouldn't be received at all).
Well, this issue is probably worth to be fixed then.
I would say so
Just curious how implementations use the inbound IHAVE topic field? Aside of filtering unknown topics (which basically shouldn't be received at all).
It doesn't seem to be used for anything else that filtering on:
This is how is it handled at the go implementation:
...
iwant := make(map[string]struct{})
for _, ihave := range ctl.GetIhave() {
topic := ihave.GetTopicID()
_, ok := gs.mesh[topic]
if !ok {
continue
}
if !gs.p.peerFilter(p, topic) {
continue
}
for _, mid := range ihave.GetMessageIDs() {
if gs.p.seenMessage(mid) {
continue
}
iwant[mid] = struct{}{}
}
}
...
A second thought on this would be to open the question of whether the peerscore
of a peer should be decreased if it sends IHAVE messages on topics we are not subscribed to. But this would be a separate topic to discuss at the GossipSub spec level.
Just curious how implementations use the inbound IHAVE topic field?
@Nashatyrev can you elaborate on what you mean? The IHAVE primitive is there in order to have a fall back option of hearing about a message that you haven't received through the mesh. It's probabilistic (i.e., you won't necessarily receive IHAVEs for all the messages that have been propagated through the mesh) and messages advertised through IHAVEs should have a topic, otherwise nodes will end up receiving significantly more traffic than interested in (increasing bandwidth and CPU consumption and as a result being an amplification attack vector).
@yiannisbot as per gossipsub spec a peer should not issue IHAVE
s to a peer for topics this peer is not subscribed. Thus if you see a messageId which is not in your seen cache then it should mean you are missing a message from a topic you are subscribed to.
Just curious how implementations use the inbound IHAVE topic field?
For reference, 3 other implemenations besides go-libp2p filters on the IHAVE topic field:
rust-libp2p's implementation:
https://github.com/libp2p/rust-libp2p/blob/master/protocols/gossipsub/src/behaviour.rs#L1210-L1219
nim-libp2p's implementation:
We have a PR which will fix that incompatibility https://github.com/libp2p/jvm-libp2p/pull/365 . I am just wondering about one thing. The protobuf spec for the IHAVE message specifies the topicID as optional
. Is that intentional? Essentially as of now, seems like all client implementation filter out IHAVEs which don't have this optional field set. We implement this a bit differently in the PR and allow IHAVEs without a set topic in order to be compatible with other Teku nodes who wouldn't use the new version of jvm-libp2p
which we will release.
if (msg.hasTopicID() && !mesh.containsKey(msg.topicID)) {
return
}
Summary
After streaming libp2p traces from our (ProbeLab team) custom libp2p-trace generator tool
Hermes
, we’ve spotted that there are some peers in the Ethereum network sharingIHAVE
messages, includingmsg_ids
, on empty topics.After inspecting this behaviour closely and double-checking that this isn’t a problem or a bug on our end, we’ve been able to verify that:
message_ids
on thoseIHAVE
messages with empty topics do belong to other complete Ethereum topicsAfter checking any existing correlation between the remote peerIDs sending those messages and any particular client or libp2p implementation, we could count a total of 55 unique peers that sent those empty-topic messages over 3.5 hours, where 49/55 of those peers were successfully identified as
teku/teku
nodes, and the 6 missing peers were unable to be identified.There hasn’t been any further investigation on which could be the root problem that generates those buggy
IHAVE
control messages, but it seems that this could be related to an implementation bug, where the topics aren’t mapped correctly tomsg_ids
or aren’t correctly added to the control RPC.Let us know if there is anything else we can help you with :)
Expected behavior
Incoming control GossipSub messages should:
msg_ids
from other topics on the wrong topicsActual behavior
These are trace examples read by the
Hermes
tool:Relevant log output
No response
Possible Solution
No response
Version
The error seems to be coming from the Ethereum’s java client
Teku
which seems to be using the lastest versionV1.1.0-RELEASE
version:Would you like to work on fixing this bug ?
Maybe
CC: @Nashatyrev @ajsutton