Corrupted validators and untraceable tx ordering causes PoSA abuse & network spam

[lennybakkalian]

I will try to explain my view on the current PoSA at BSC.

It's best if I start with an example:

In all blocks validated by the "CREAM Validator", an address performs transactions for ~1 GWEI to get gas tokens. https://bscscan.com/address/0x111111102ba661adbc11daf8a88d6d54ad3e9ac2 (this is only one of many wallets that performs such transactions in all blocks validated by CREAM)

Then all gas tokens are sent to this wallet and sold there. Here you can trace back how all addresses minted Gas Tokens ONLY in CREAM Validated blocks.

For those who do not yet understand the principle of Gas Tokens: Per Gas Token an empty contract is created and by "Self Destructs" Gas is returned again. This means: CREAM (or their friends) creates cheap contracts and sells Gas Tokens so that they return gas at the normal GasPrice.

Why are validators not punished for such behavior?

Transaction ordering

In addition, MEV opportunities should be shared fairly for all participants in the blockchain. Currently transactions are sorted by Nonce->GasPrice->ArrivalTime. However, validators can sort the transactions with the same GasPrice as they like, because they can change the ArrivalTime themselves. A suggestion from a previous ticket was to sort the transactions by Nonce->GasPrice->Hash, which I think is fairer because the sorting of the transactions is traceable. Then corrupt validators could only slow down the transactions of the "competition" and include them in the next block. I can't think of any way to fix that.

Also, the current method leads to network spam, as transactions with the same nonce/gasPrice but different hash are sent to many nodes to get a desired tx slot. The way tx broadcasting is set up, only a part of the full transactions are sent to other nodes, which leads to this kind of spam.

The blockchain should be decentralized, freely accessible and traceable. Why hasn't this problem, which I see as a major one, been solved earlier?

[lennybakkalian]

Validators should only be paid for validating transactions, but currently some validators offer services that destroy the security & decentralization of the blockchain which have been discussed in other issue tickets (1,2). Apparently no one cares that validators have such a big impact on BSC.

[ktrout1111]

Excellent work. BSC indeed appears to be a rotten cesspool of corruption. I've been pounding the table about the current MEV situation (one that generates huge amounts of spam in the form of excess network traffic, fake nodes and massive tx bloat) for a while now, @'ing the devs, only to get not a single reply from any of them. Why is the whale botnet that is currently sucking up 95%+ of the MEV being allowed to continue ruining the network for their own financial gain?

Not holding my breath for an answer...

[lennybakkalian]

I would be willing to refactor the transaction sorting by GasPrice->Hash instead of GasPrice->ArrivalTime so that a fair distribution of MEV opportunities is possible and tx spam is prevented. However, I don't know if in the end my pullrequest will be merged and all validators will accept this change.

Searching the validator rules I didn't find any information if they are forced to use the latest official Geth version. With the current implementation validators can sort transactions as they like without anyone being able to prove to them that this order has been manipulated or not.

[ktrout1111]

Sadly this has been a long standing issue and the devs could not be more indifferent about it. I'd be amazed if they merged your pull request.

The spam under the current system is especially bad since each of the hundreds of txs being spammed per block are each being spammed by hundreds of different bots, each with different tx hash but same nonce, so really there are as many as tens of thousands of full txs flooding the entire network every block, all for the profit of a single individual or syndicate.

Changing the tx sorting away from arrival time will immediately stop the need for these hundreds of fake spammer nodes and massively ease the network bandwidth demands currently being experienced by full nodes.

Currently, the devs seem more interested in putting large amounts of work into reengineering geth to accommodate and benefit this single profiteer, rather than just disposing of them by changing a few lines of code back to what they used to be before the 1.10 fork.

Searching the validator rules I didn't find any information if they are forced to use the latest official Geth version.

From what I've seen, they generally seem to update to the latest version reasonably promptly.

[lennybakkalian]

Before 1.10 fork, transactions with the same gas price were randomly sorted which also led to tx spam. I can't think of any other way than to use tx hash POW for this kind of tx sorting to make the sorting traceable.

However, corrupt validators could then delay individual txs from the competition that have a better hash for this MEV opportunity and only include them in the next block. I would prefer that instead of the network being spammed with txs.

[ktrout1111]

You and many others have pointed out that validators can manually order txs in any way they see fit to their advantage, which is certainly plausible. However, this would not explain the need of the current MEV monopolist to spam hundreds of txs per block. If they were colluding with the validators (you'd need a backroom deal with 21 separate parties), or just were the validators in disguise, there would be no need for spam - a single tx would do the trick if you had god mode powers to reorder txs within a block.

In fact, this is something I did witness for several months after the 1.10 fork. A single contract was successfully winning 50%+ of the arbs with a single tx. After several months of domination, this success rate dropped and dropped until they were no longer even breaking even, and they cut their losses and shut down. The competition now is so intense that you have to run hundreds of VPS just to have a hope of making a profit.

Before 1.10 fork, transactions with the same gas price were randomly sorted which also led to tx spam.

This is true, it did, it bloated blocks with failed arb txs. This is why the Eth devs changed the sortation to arrival time, the end result of which (on BSC at least), is that not only do we have tx spam of unprecedented levels, but also the network itself is bloated with fake nodes. So the system we have now is the worst of both worlds.

I agree that sorting by tx hash would be the best approach. It would dramatically reduce spam since each bot owner knows not to send another tx until the current best hash has been undercut. It solves the problem of block bloat, and massively eases the burden on the full nodes. It's win/win for everyone except Mr. 0xaf25d6494a341675186f975c4cbc081fdac05dbc.

[lennybakkalian]

I agree. I'll try to discuss with the maintainers via the Discord server if they would agree to such a change. Hopefully Ethereum POS will think about this, so that validators can not exploit their position.

Would be a win for all but the corrupt validators ;)

[psdlt]

When nodereal introduced their DirectRoute, that ruined my previously profitable trader bot, because now malicious actors have a way of rug pulling without going through public mempool (thus no way for my bot to see rug pull tx and try to frontrun it).

so really there are as many as tens of thousands of full txs flooding the entire network every block

I think you're exaggerating. I'm logging how many txs reach my nodes between blocks, and numbers are fairly stable 200-300 txs. There are occasions when I see several thousands, especially during "panic events" when everyone wants to buy/sell something and there are lots of arb opportunities, but those are rare and usually only last for several blocks.

In general that's how decentralised network works - there is no "police", everyone is free to choose whether they're running a vanilla software or if they modify it for their own needs (i.e. different tx sorting, blocking, delaying, whatever).

I think tx hash idea is good for overall network health. MEV searchooors will go from having many fake nodes to having few large nodes that will be constantly calculating a "better hash". But how do you stop CREAM from making exceptions for themselves or their friends, or how do you stop nodereal from offering DirectRoute? And IMO it's very unlikely that Binance will assume this "police" role and will start actively monitoring network and punishing validators for "not following sorting rules".

Just my $0.02.

[ktrout1111]

I think you're exaggerating.

To clarify, these arent "txs" in the sense that you're talking about, i.e. the final txs that are mined into the block by the validator. What this guy is doing is sending a different version of each of those couple hundred txs from each of a few hundred bots. So in terms of network bandwidth, these two values are multiplied. This is what I witnessed a few months ago when I was looking in real time at the actual contents of the messages I was getting from his nodes. Presumably this has something to do with ensuring the best arrival time by calculating the arb from many different fake nodes and sending txs from the same accounts with the same nonce from each. Whichever of these, per nonce, arrive first at the validator wins and the rest are stripped out as duplicates by any nodes they encounter.

As far as network bandwidth is concerned, it could well be tens of thousands of tx messages, although obviously any given node will only forward one of them per nonce.

[lennybakkalian]

I think that 200-300/s transactions arriving at your end is because the Full Nodes do not forward duplicate transactions (same Nonce & GasPrice) because they are invalid. But in the whole network there are suddenly thousands of transactions with the same nonce/gasPrice where only the first arrived tx is taken by the validator.

In general that's how decentralised network works - there is no "police", everyone is free to choose whether they're running a vanilla software or if they modify it for their own needs (i.e. different tx sorting, blocking, delaying, whatever).

That the blockchain is uncontrollable and there is no police is correct. Nevertheless, it is centralized at 21 points (the validators), because it is not otherwise possible with PoSA. I have also rewritten Geth to my benefit, but there must still be clear rules for the validators, because they "control" the network.

[psdlt]

@ktrout1111 I'm sorry, I probably wasn't clear in my original message. I'm logging how many different txs reaches my own mempool between blocks, so my numbers include replacement txs. I don't really care about finalized txs.

@lennybakkalian

I think that 200-300/s transactions arriving at your end is because the Full Nodes do not forward duplicate transactions

that might be true, but I have multiple nodes in multiple locations and numbers are consistent. But I haven't really dug deep into this and you might be right.

it is centralized at 21 points

yeah, but... anyone can become a validator if they pull together enough of BNB to become one. Community previously pushed nodereal into changing how their DirectRoute works, but I have little hope that same would work with 21 validators. Fingers crossed though!

[ktrout1111]

I think that 200-300/s transactions arriving at your end is because the Full Nodes do not forward duplicate transactions (same Nonce & GasPrice) because they are invalid. But in the whole network there are suddenly thousands of transactions with the same nonce/gasPrice where only the first arrived tx is taken by the validator.

This is what I meant. It's extreme spam on the P2P network but has no extra spam effect on the actual blocks than simply sending a single tx per account per nonce.

[lennybakkalian]

Community previously pushed nodereal into changing how their DirectRoute works, but I have little hope that same would work with 21 validators.

DirectRoute is a big problem. If everyone used that, then BSC would be completely centralized.

To the example I wrote at the very beginning: Because millions of gas tokens are minted all the time in all CREAM validated blocks, millions of contracts are also created, which in turn generates a lot of garbage. When the gas tokens are then released, the block rewards become smaller at the expense of the other validators. So validators can piss on each other's legs. These 21 validators should not be unregulated and punished for manipulations. BSC is not decentralized!

[m-e-r-k-l-e-root]

It would be interesting if txs w/ same gas price were sorted by hash as apposed to by arrival time. This would offer on-chain proof that txs (w/ same gas price) are being sorted in an entirely transparent and/or fair manner, as of right now, there is no way to know if validators are sorting txs fairly.

[48ClubIan]

Is there a possibility that CREAM validator is just so stupid that it sets gas floor as low as 1 gwei ? Some other smart guy just exploit this config.

I tend not to explain phenomenon with evil.

[lennybakkalian]

@48ClubSirIan No matter what the reason is: CREAM should be punished for something like this. Such things destroy the blockchain as I explained in this comment.

Also, I don't think the validator changed any configs out of "stupidity". With so much money, they will think a thousand times which config is edited.

[48ClubIan]

@48ClubSirIan No matter what the reason is: CREAM should be punished for something like this. Such things destroy the blockchain as I explained in this comment.

I agree with this part that CREAM is harming other validators' profit, and really want it to stop doing so.

However I am not sure it should be punished if it's only setting an extremely low gasfloor without selling gas token to specified people. It's easy to test whether an ordinary address can also mint gas tokens, right? To be clear I didn't do the test yet.

It's not smart for validators to decide to accept a lower gas price, but not a crime either.

[lennybakkalian]

If the validator has set the GasPrice to 1 GWEI, the "smart" attacker should have a direct connection to the validator node, since full nodes do not forward the txs if the gas price is too low, am I right?

[48ClubIan]

If the validator has set the GasPrice to 1 GWEI, the "smart" attacker should have a direct connection to the validator node, since full nodes do not forward the txs if the gas price is too low, am I right?

• Not exactly. If you run your own node and bypass the local gas price check, it can be broadcast to peers and eventually reach validators.

[lennybakkalian]

Yes you can send the transaction out. But the other full nodes drop transactions with low gas price.

[48ClubIan]

Yes you can send the transaction out. But the other full nodes drop transactions with low gas price.

Well, the fact is not all peers drop it...

[lennybakkalian]

Do we really want to discuss how many full node operators set the gas price at 1 GWEI? Because I can assure you that nobody does that, because it brings them NOTHING.

[48ClubIan]

Do we really want to discuss how many full node operators set the gas price at 1 GWEI? Because I can assure you that nobody does that, because it brings them NOTHING.

I am not discussing how many, but whether it is possible.

If so, we can't simply say CREAM is selling GAS token, which is a serious accusation, solid proof is needed.

BTW once confirmed, me too vote for a punishment.

[lennybakkalian]

Of course you can't confirm it. you will never be able to confirm who is doing what transactions on the blockchain. But you know that for sure :)

The only thing I know is:

• (most) full nodes do not forward transactions below 5 GWEI.
• A scam and CREAM validated blocks are related.

That's exactly how I wrote it in the issue ticket. I have written out my thought processes completely and how I personally come to the suspicion that CREAM is exploiting their position.

And even if CREAM didn't know about it, they should be excluded as validators, because then they seem to have no idea what they are doing.

I'm sorry if CREAM really didn't know about it and just incompetently changes settings. But the way the Binance Smart Chain is currently structured, I don't trust any validator, because nothing is traceable and validators can do what they want. How is it even possible to store transactions with 1GWEI? Such a thing should be recognized by other validators and then be punished.

[48ClubIan]

And even if CREAM didn't know about it, they should be excluded as validators, because then they seem to have no idea what they are doing.

I may have another question: Can a validator legitly set gas floor to 1 gwei with fully aware of it? Why not? 'cause I would like to defend this right.

5 gwei comes from a factly price alliance (default config), I didn't see why a validator must do so.

It's reasonable to accept a lower price in order to raise short-term income. Although not a good idea from bigger view.

The whole GAS token thing isn't good at least on BSC but it doesn't deprive validators' right of editing its own config file, I believe.

Maybe we'd better raise a PR to remove the gas returning mechanism.

[lennybakkalian]

5 gwei comes from a factly price alliance (default config), I didn't see why a validator must do so.

...but it doesn't deprive validators' right of editing its own config file, I believe.

Okey. Assuming that every validator would adjust the price, it would end up being a "battle" between the validators to see who has the best price. This would lead to thousands of txs being spammed on the network to end up only in the blocks with the cheapest transaction costs.

The task of a validator should not be to change prices or to be god! I don't think you fully understand what a blockchain is all about.

But to give you a little help. It should be: decentralized and not controllable by any party. Not even by the validators, who, as the name suggests, should only validate blocks.

Also, I find it a bit rude that someone who works for a validator (from your profile) writes such a comment. This shows that validators have completely missed their target.

[berktaylan]

Why dev team doesnt respond to this topic ?

[TxByTime]

lennybakkalian

(most) full nodes do not forward transactions below 5 GWEI.

This is just wrong. If you use the standard config.toml you will find GasPrice = 1000000000 / 1gwei. This means every fullnode running the stanard config will forward them. If you check your txpool you find find many tx bellow 5gwei. If you don't want to forward them just change the parameter.

Why are validators not punished for such behavior?

Validators sell blockspace. It's their decision to set the minimum price for inclusion. There is a rpc method to change this price: https://geth.ethereum.org/docs/rpc/ns-miner#miner_setgasprice. 5gwei seems to be a constant the validators agreed on. If they don't like someone not obeying it, they should take action against it. But I can not find any offical rule stating that they can not make the gasPrice whatever they like. We should be glad that they keep it constat at 5gwei. Just look at other chains and how their gasPrice can explode during high congestion. If the Validators were that greedy as presented here and the other discussions they could just raise it to make more money. There is nothing preventing that. If you don't like that you can submit a goverance proposal to introduce slashing for validators not obeying this 5gwei constant.

ktrout1111

BSC indeed appears to be a rotten cesspool of corruption.

I don't see how this is corruption. During this 1gwei phase they accepted any tx priced above 1gwei not just the chi mints. I'm sure if you tried to submit a 2gwei tx it would also have been mined. They will get propageted like I mentioned above. Here is some proof: https://bscscan.com/tx/0x73f946935b700844d5098a2655f7ef758b1ec1e3f38c3317820232b73b5528ac https://bscscan.com/tx/0x99c23870fc54850e45142f7876d01d514251c6e4b7b6e3c70a46efe42cc4a52f https://bscscan.com/tx/0x5102206b55296617b111a60167d01018d04b758c02848b07835999e022d01b1f You seriously think the validators and this random guys swapping some tokens are colluding together or that they are paying the validator? Or is it more likley they just saw the opportunity and acted? My opinion: the chi minter is most likley a mev bot watching the gasPrice for opportunities - I'm pretty sure it's one of the bots that are not runnung "hundrets" of VPS.

My last point is that chi minting at 1gwei is helping the chain. Some protocols like 1inch are relying on it to make transactions for the user cheaper. Since the end user will do that at 5gwei it needs to be minted bellow that. Someone has to do that but no one will do so at a loss. So dropping the floor to 1gwei is necessary for this token to be usefull. I'm pretty confident anyone could mint them at this gasPrice if they see that the validator is mining transaction at this level.

Disclosure: I don't run any chi minting bots. There is no conflict of interest for me here.

[lennybakkalian]

My last point is that chi minting at 1gwei is helping the chain. Some protocols like 1inch are relying on it to make transactions for the user cheaper. Since the end user will do that at 5gwei it needs to be minted bellow that.

the sense that SELFDESTRUCT gas returned was not made with the intention that you buy cheap gas and use it when the gas price is high.

Gas refunds for SSTORE and SELFDESTRUCT were originally introduced to motivate application developers to write applications that practice “good state hygiene”, clearing storage slots and contracts that are no longer needed.

Validators sell blockspace. It's their decision to set the minimum price for inclusion.

so if validators decide for themselves how much gas they validate blocks for, i no longer understand the point of a blockchain. as i said before: then a validator could set the gas price lower to mine more txs which in turn would create more profit for that validator. wouldn't this create the incentive for validators to want to undercut each other to be able to include the most txs in the block?

[TxByTime]

My last point is that chi minting at 1gwei is helping the chain. Some protocols like 1inch are relying on it to make transactions for the user cheaper. Since the end user will do that at 5gwei it needs to be minted bellow that.

the sense that SELFDESTRUCT gas returned was not made with the intention that you buy cheap gas and use it when the gas price is high.

Gas refunds for SSTORE and SELFDESTRUCT were originally introduced to motivate application developers to write applications that practice “good state hygiene”, clearing storage slots and contracts that are no longer needed.

Validators sell blockspace. It's their decision to set the minimum price for inclusion.

so if validators decide for themselves how much gas they validate blocks for, i no longer understand the point of a blockchain. as i said before: then a validator could set the gas price lower to mine more txs which in turn would create more profit for that validator. wouldn't this create the incentive for validators to want to undercut each other to be able to include the most txs in the block?

The gastoken was created for this reason. Mint when the gas is cheap and burn it when its expensive to save tx costs. It's their usecase. Why should anyone mint it if they would be losing money?

The miners on eth could do the same thing before EIP-1559. Now the basefee is calculated by some formula. But merging this will lead to gasPrice explosions like we see them on other chains and would not really improve something for the users.

What is your proposal to change this? Hardcode it to 5gwei, change config.toml and remove the rpc method?

[lennybakkalian]

The gastoken was created for this reason. Mint when the gas is cheap and burn it when its expensive to save tx costs. It's their usecase.

yes that's why there are gas tokens, i know that.. but this feature (behind gas tokens) was NOT meant for buying gas cheap and selling it expensive. but gas tokens exploited this loophole. There is a reason why gas is getting more expensive and I don't think it's right to circumvent that with gas tokens. Especially if these tokens are minted only at a single validator for almost free.

[TxByTime]

The gastoken was created for this reason. Mint when the gas is cheap and burn it when its expensive to save tx costs. It's their usecase.

yes that's why there are gas tokens, i know that.. but this feature (behind gas tokens) was NOT meant for buying gas cheap and selling it expensive. but gas tokens exploited this loophole. There is a reason why gas is getting more expensive and I don't think it's right to circumvent that with gas tokens. Especially if these tokens are minted only at a single validator for almost free.

It's not a single validator if you look at past blocks. Others have done it before: Legend: https://bscscan.com/tx/0xed092c2f7d7efb9eaefd4c6ca793dd5ea6f0c553f5e3bcc1f02d080347b9c686 Legend II: https://bscscan.com/tx/0x1fb99c6929f35a1fb54a1ebf82c31bac104f97cd48d1b97a2d3781233f480b98 0x4bae4a1a12f26e26cac5ce7fba5dce201d24c6ef: https://bscscan.com/tx/0x7b811ef5e4ce66d23770d6a4aeaa49c9a850d351e59d568952aadf7b4bede23f

There might be more, but these were the ones I could find with a quick search.

[unclezoro]

Our team talked with CREAM Validator as soon as we saw this issue, it turns out to be a false configuration on their nodes. We request them to reply to this issue after correcting that, so I make a voice in their place since they do not show up yet.

Currently transactions are sorted by Nonce->GasPrice->ArrivalTime. However, validators can sort the transactions with the same GasPrice as they like, because they can change the ArrivalTime themselves. A suggestion from a previous ticket was to sort the transactions by Nonce->GasPrice->Hash, which I think is fairer because the sorting of the transactions is traceable.

The Hash can be brute-forced as long as you get enough hashrate, so if the transactions are sorted by Nonce->GasPrice->Hash, it will cause hashrate competition, the result is that the big organization can always do both frontrun and backrun, it is easy to perform sandwich attack.

Before geth 1.10 , transactions with the same gas price were randomly sorted will also lead to tx spam since the arb robot will do every effort if there is an opportunity. Sorting by ArrivalTime will cause spam on the P2P network, but has no extra spam effect on the blocking network.

[lennybakkalian]

thanks for the clarification. however, the PR was created primarily because of other points. i answered you there.

[ktrout1111]

Sorting by ArrivalTime will cause spam on the P2P network, but has no extra spam effect on the blocking network.

Except this isn't true. The spam is as bad now, or worse, than it was when the sorting was random. At least when the sorting was random there wasn't this extra problem with fake nodes.

https://bscscan.com/txs?a=0x911c6f41f62ae50277dcd2254753bc832aab904b&p=3

See above and the surrounding pages. Dozens of spam txs per block, and that is just one of 4 contracts owned by the same entity. So the spam from a single entity is 4x worse than you can see in the link above.

[Lucienest]

It's very unlikely that the BSC team will take any concrete actions like the previous thread.

[IvanVakhmyanin]

Just made a research on the affiliation between flash arbitrageurs and validators. Scanned all the history of BSC to identify all arbitrage transactions, contract and contract owners. Then I've measured the correlation between profitable transactions and block validators. I could not find any strong evidence of affiliation with validators - at least for the top-earning validators. Their profits seem to be fairly distributed (except for some natural skewness because of better peering with some of the validators).

Of course, that doesn't rule out the possibility of affiliation completely. You can explore the data yourself in our public dashboard: https://datamint.ai/research/#arbitragemonitor

More about the methodology of the research: https://medium.com/datamint-on-chain-data-analytics/can-you-earn-a-lambo-by-being-a-flash-arbitrageur-on-bnb-smart-chain-b119701329fe?source=friends_link&sk=12f2b772d39cf52315c35f514cf5d36f

[bnb-tw]

closing due to inactivity.