Closed phahulin closed 6 years ago
5chdn
commented
7 years ago To accept transactions with more than 4.7 million gas, just run your sealing nodes with --gas-floor-target 6666666 (or any value you like).
To remove transactions from the pool, either restart your nodes with the --no-persistant-txqueue flag or replace them with transactions of same nonce but higher gas price (any simple transaction should do).
phahulin
commented
7 years ago But isn't this a potential DDoS attack vector? A malicious user can join the network and flood it with as many out-of-gas transactions as possible.
5chdn
commented
7 years ago A malicious user would require access to your PoA configuration (chain spec) and some decent amount of ETH to be able to DoS your network.
Reopening for a 2nd opinion.
phahulin
commented
7 years ago Well, our configuration is open as I've provided a link to spec.json in the first post.
Please note, that this user even need not to be a validator, he just needs to connect and run truffle migrate
tomusdrw
commented
7 years ago Transactions significantly above current block gas limit are not accepted by other peers at all. If the transaction is a little bit above gas limit (up to 0.5%) it will be accepted, but evicted from the queue (after some time) when it's not mined. Also it will get a very low priority, so it's easily pushed out of the queue by other transactions in the network. Note that because the gas limit doesn't allow that we don't event attempt to run the transaction, so the only issues is that it can generate some network overhead when transactions are propagated.
keorn
commented
7 years ago Third party accounts will also have not access to the tokens on your PoA network so will be unable to submit a transaction in the first place. If you want to restrict connectivity since tokens are widely distributed, then use reserved peers.
You can only get your own account stuck, does not affect other accounts at all (try it on the mainnet).
phahulin
commented
7 years ago @tomusdrw what is the eviction period equal to?
tomusdrw
commented
7 years ago 
rstormsf
commented
7 years ago @tomusdrw @keorn @5chdn We have over 200k blocks already passed and none of the transactions were evicted. http://testnet.oracles.org:4000/tx/pending
According to @tomusdrw comment for transaction_queue it should have already been rejected, why do we still see them?
Am I understanding it correctly that all nodes are keeping the transaction queue somewhere in memory?
Netstat: http://testnet.oracles.org:3000/
Here is when it gets interesting: When I send a tx from the same account that generated txs after some time, it goes thru and removes all pending tx that were previously generated by the same account. So now whoever generated pending tx, needs to create a new transaction in order for other pending txs to get kicked out.
To reproduce:
8f3d8067ac097bd394eaa929787640002f0e6d802dfd452429394dd97987e66c (there are 64 OEther there)
Public RPC node (for metamask) http://40.117.197.50:8545tomusdrw
commented
7 years ago So most likely the node that initiated the transactions has all of them and keeps propagating them to the network. One of those transactions is being rejected by the mining/authority node - and rest of them goes to future. Sending another transaction renders different behaviour depending on the node, because:
nonce that is used is the one after all of those transactions (new transactions is stuck)nonce is the one that fills the gap.It should be enough to stop the node that contains the account that initiated the transactions, wait until they are removed from the queue of the other node and restart the first node with --no-persistent-txqueue to clear the transactions that got stuck.
So to be clear: the transactions are evicted from the queue, but if there is at least one node that keeps propagating them they will be imported again.
That said, I'm considering removal of accepting "slightly above gas limit" transactions at all.
rstormsf
commented
7 years ago @tomusdrw Nope, I killed the node long time ago and all pending txs were still there. We waited ~ 200k blocks - (http://testnet.oracles.org:4000/tx/pending)
So how do I figure out which node keep propagating txs?
How do we clear pending tx now?
If I just run the node right now:
-l engine=trace,tx=trace,txqueue=trace,own_tx=trace,miner=trace
here is what I see:
2017-08-25 13:12:00 Verifier #5 DEBUG miner Skipping adding transaction to block because of invalid nonce: b16b41b748dfe6c11527477884c247201afaa3d0847695a7557c8ed52bef59f6 (expected: 1, got: 2)
2017-08-25 13:12:00 Verifier #5 TRACE miner Adding tx 2a25cce7afd74eaed0145e2487cfbe72f26f17087b98ce028eb31eed786031d7 took Duration { secs: 0, nanos: 1342 }
2017-08-25 13:12:00 Verifier #5 TRACE miner Adding tx c5c14a0c7ee1b49906323c5c09ce1b44855b373b9b3c41bdb2611a1bf840483b took Duration { secs: 0, nanos: 1181 }
.........
2017-08-25 13:12:00 Verifier #5 TRACE miner Pushed 0/6 transactions
2017-08-25 13:12:00 Verifier #5 TRACE miner update_sealing: engine is not keen to seal internally right now
2017-08-25 13:12:00 Verifier #5 INFO import Imported #254288 11ad…12b0 (0 txs, 0.00 Mgas, 0.82 ms, 0.57 KiB)
2017-08-25 13:12:05 IO Worker #0 TRACE miner update_sealing
2017-08-25 13:12:05 IO Worker #0 TRACE miner requires_reseal: sealing enabled
2017-08-25 13:12:05 IO Worker #0 TRACE miner requires_reseal: should_disable_sealing=false; best_block=254288, last_request=0
2017-08-25 13:12:05 IO Worker #0 TRACE miner update_sealing: preparing a block
2017-08-25 13:12:05 IO Worker #0 TRACE miner prepare_block: No existing work - making new block
2017-08-25 13:12:05 IO Worker #0 TRACE miner Adding tx 2bea9e0999d807af27976dd65c3a49e1e7e1d009dd0ee331911e7a50a660617a took Duration { secs: 0, nanos: 93097 }
2017-08-25 13:12:05 IO Worker #0 DEBUG miner Skipping adding transaction to block because of gas limit: 2bea9e0999d807af27976dd65c3a49e1e7e1d009dd0ee331911e7a50a660617a (limit: 4700000, used: 0, gas: 4712388)Is there any idea on why truffle migrate generates pending tx.
tomusdrw
commented
7 years ago Ok, sorry I got confused. I thought that you meant that future queue is filled up with some transactions.
If the transactions are in the pending queue they will be re-propagated by all those nodes, we anticipate a block gas limit increase, but if it does not happen it won't ever be included.
To remove the transaction you need to replace the one that has this super high gas limit with something that can actually be mined (Parity 1.7 allows you to "cancel" transaction that is pending, and does exactly that afair).
rstormsf
commented
7 years ago why this issue is closed?
@tomusdrw yes the future tx is filled up with the same pending transactions and they never get killed.
when you said - you need to replace. There are some txs that were generated by some unknown users. That's why @phahulin mentioned that someone can just come to the network, generate bunch of pending tx that will never be killed to spam the network. What is the solution?
http://testnet.oracles.org:4000/tx/pending - we still have pending txs. How do we remove them?
Please feel free to take this private key:
8f3d8067ac097bd394eaa929787640002f0e6d802dfd452429394dd97987e66c
run parity against oracles network with these configs: https://gist.github.com/rstormsf/d88b37a8a7bdfb43feb54dd6e01a4dcc
try to simply truffle init && truffle migrate
you will see how easy it is to generate pending txs that will never be killed to spam the network.
Please try.
tomusdrw
commented
7 years ago @rstormsf The PR that lowers maximal gas of a single transaction is pending: #6408. The transactions are still valid, they are just awaiting for a miner willing to rise the block gas limit to accept them (alike low gas price transactions).
They occupy the queue, but they are getting a very low priority so they won't block other transactions from being included and they don't really cause any threat to the network. They cannot be killed from the queue, because other nodes are repropagating them (and they are allowed to do so with any valid transactions).
That said I do understand that you want to keep your queue clean though and not accept such transactions at all.
If you want to get rid of them the only way currently is to increase your gas floor target and mine a slightly bigger block to process them.
rstormsf
commented
7 years ago @tomusdrw Thank you for taking your time to understand our problem and submitting a PR. Looks like it's merged into master. We will test the change in our devtestnet and we will let you know.
Here is what I don't understand: How does Kovan network solves this problem? They always have 0 pending txs: https://kovan.etherscan.io/txsPending
If you want to get rid of them the only way currently is to increase your gas floor target and mine a slightly bigger block to process them.
Could you explain more on that? I don't believe that PoA Kovan chain is doing something like that in order to get rid of pending txs. I'd expect them to become invalid and stop propagating.
So, you are suggesting to restart every validator's node with --gas-floor-target XXX, correct?
FYI - I don't fully understand how does this PR solves the problem if we still need to change gas floor target?
https://raw.githubusercontent.com/oraclesorg/oracles-scripts/alphadevtestnet/spec.json - OraclesPoA - here is our spec, maybe we should add some flags to it? In comparison to kovan.spec
Here is the diff between Kovan.spec.json vs OraclesPoA.spec.json:
1.
In authorityRound they have those params which Oracles doesn't. Do you know what it does?
"validateScoreTransition": 1000000,
"validateStepTransition": 15000002.
in params Kovan spec also have and Oracles-PoA does NOT:
"gasLimitBoundDivisor": "0x400",
"registrar": "0xfAb104398BBefbd47752E7702D9fE23047E1Bca3",
"blockReward": "0x4563918244F40000",
"validateReceiptsTransition": 1000000,
"eip155Transition": 1000000I suspect that maybe some of those missing flags can solve the problem.
@tomusdrw what do you think?
I think maybe gasLimitBoundDivisor has to be there ?
rstormsf
commented
7 years ago @tomusdrw if you may. I've seen @phahulin filed new issue #6429
tomusdrw
commented
7 years ago gasLimitBoundDivisor is indeed related. How much block gas limit is allowed to change between blocks (next limit must be within (gasLimit - gasLimit/divisor, gasLimit + gasLimit/divisor)).
AFAIR this value is mandatory, setting it to some large number would cause block gas limit to stay constant.
The fix doesn't accept such transactions at all, so after you upgrade your nodes to a patched Parity versions those transactions will be discarded.
--gas-floor-target suggestion was a workaround to process those transactions (or reject them completely) by driving the block gas limit up or down respectively - not all nodes need to have the same values, some of them might be trying to push the limit up while some keep it low and it will fluctuate.
bjornwgnr
commented
6 years ago @phahulin @rstormsf: Can this issue be closed?
Hello, everyone
We have a PoA network with validators via contract. Nodes are connected and generate blocks - that's all OK. Here is spec.json Here is node.toml from one of the mining nodes:
If miner tries to deploy some other regular contract that exceeds gas limit (here a default truffle's migration contract is being deployed), his transaction will not be rejected (these logs are obtain with
parity -l engine=trace,tx=trace,txqueue=trace,own_tx=trace,miner=trace)but will get stuck in the queue, being skipped each round
Truffle migration just hangs
If you run contract deployment again, a second pending transaction will appear
We have a separate instance of our network (for real people to test), where there are 20+ transactions like this that have been hanging in queue for days.
If you send a valid tx from that address (e.g. transfer eth to someone), it will also remove one of the pending txs from the queue:
I did this test on a mining node, but it can also be reproduced by any node connected to the network. I think this is incorrect and invalid transaction should be removed from the queue.