[BUG] Unconfirmed transactions not being gossiped to persistent peers

[theblockbois]

Describe the bug This is a repost of an issue I opened on Tendermint

I have 5 full Terra nodes distributed around the globe. Each node was initialized with the other 4 nodes specified as persistent peers, and I can verify that the peers are all connected to each other via the net_info RPC query.

On each node, I am running the same code where I am monitoring the local mempool via the unconfirmed_txs RPC query to find a transaction from a specific account. However, I notice that I can detect the target transaction on only 1 of the peers - the other peers never detect it in their local mempools, even after the transaction has been mined.

Given that the nodes are all connected to each other as persistent peers, I expected that if one node receives a transaction in its mempool, it would then broadcast the transaction to the connected persistent peers via TCP gossip, and thus all of the other nodes would eventually detect the transaction in their mempools. This doesn't seem to happen however - for some reason, only 1 particular node is able to detect the transaction I'm looking for, and it isn't being gossiped to the others.

Does anyone know why this might be and how I might fix it?

Context & versions Terra core version: v0.5.17

[hanjukim]

Look at this commit https://github.com/terra-money/tendermint/commit/b81ef6d62c9b0d618c073e76c186cc8314061056 I am not sure this is causing the problem but we patched our server to remove pagination restriction

[theblockbois]

@hanjukim Thanks for sharing that commit, I hadn't noticed that hack to the API before. I've been using Tendermint's page_limit parameter in my request payload. However when I omit all request parameters or set Terra's limit parameter to a really high number, the transaction still doesn't show up. I presume this means it's a networking/gossip issue then?

[hanjukim]

Tendermint doesn't accept more than 100 limit for unconfirmed txs. That's why we patched it

[theblockbois]

@hanjukim Understood, that's a really good patch to make. I still can't detect the transaction even when I set the limit to a very high number, however.

[hanjukim]

@hanjukim Understood, that's a really good patch to make. I still can't detect the transaction even when I set the limit to a very high number, however.

hey what I said was that limit does not matter because tendermint ignores it

[theblockbois]

Ok, so then I should be getting a full snapshot of the mempool state anytime I query it, but the transaction I'm looking for doesn't consistently show up in all the nodes. Is this a network issue then?

[hanjukim]

No, I think you should move this issue to tendermint (is it ignite now?) and I've never seen same issue. Did you try our patch?

[yun-yeo]

or maybe gas prices issue. Did you set gas prices in your server?

[yun-yeo]

Closing inactive issues, please reopen the issue if you still need help