Closed Stonygan closed 3 years ago
defichain-bot
commented
3 years ago @Stonygan: Thanks for opening an issue, it is currently awaiting triage.
The triage/accepted label can be added by foundation members by writing /triage accepted in a comment.
sandrich
commented
3 years ago Seen same behavior. Also did some checks against the last 3000 blocks of which cake was the miner of 97% of the blocks. Also communicated that on TG where I think due to the high hashes many miners find a block for the same stake modifier but only one gets accepted obviously. And that is usually cake since they own a lot of master nodes and have a very high chance of finding a new block just after the previous one.
tom-defi
commented
3 years ago The german tg group for masternodes is actively discussing this problem, here are some notes:
The main cause is the speedy masternode upgrade. This upgrade changes the hash power for the masternodes. The masternodes can now try between 200-700 hashes within the first second after each new block. Afterwards this is reduced to 1 hash/s. This leads to a block distribution that looks like:
We can see that there are a lot of blocks which appear within the same second (I call these Instant-Blocks) and afterwards there is a longer pause before the next block is found.
=> The chain is no longer moving smoothly and acts more jumpy.
This results in a major problem for all the non-Cake Masternodes. I assume that Cake is running most of their masternodes with multimining or/and within the same network. Therefore these masternodes instantly receive the block updates from other cake nodes while non cake nodes have to wait till the block is distributed over the network. This means that the Cake nodes can already try 200-700 hashes for the next block (and every newly found Instant-Block afterwards), before any of the other nodes have even received the first block. Therefore Cake is able to produce a chain of Instant-Blocks while the other masternodes are still waiting for the first block
=> Cake finds most of the blocks for every chain of Instant-Blocks
The problem is confirmed by the statement from @sandrich Also did some checks against the last 3000 blocks of which cake was the miner of 97% of the blocks.
The problem mentioned by @Stonygan is an aftereffect of the problem described above. The Instant-Blocks are causing a lot of mini-forks and the chains of Instant-Blocks found by Cake are often replacing blocks found by other masternodes.
=> There is a high chance that a found Instant-Block is replaced by a chain of Instant-Blocks from Cake
Stonygan
commented
3 years ago 
Detailed Statistic of 1 hour. Other snapshots are similar. Not all Instant Blocks are CAKE.
Stonygan
commented
3 years ago Seen same behavior. Also did some checks against the last 3000 blocks of which cake was the miner of 97% of the blocks. Also communicated that on TG where I think due to the high hashes many miners find a block for the same stake modifier but only one gets accepted obviously. And that is usually cake since they own a lot of master nodes and have a very high chance of finding a new block just after the previous one.
I think we have to consider here that the normal APP 2.5.0 does not run with 1.7.10 and therefore without "Speedy" unless this is manually adjusted. Therefore, the share of CAKE will currently be so high.
bvbfan
commented
3 years ago ThreadStaker: (6d6f34a11aacbbe85ce6d7fafafebd468fc6b503) minted a block!
6d6f34a11aacbbe85ce6d7fafafebd468fc6b503 is masternode operator name not a block hash, block is accepted unless you see error in the log, that shows it's not.
The chain is no longer moving smoothly and acts more jumpy.
Yes, it alternates ~ normal blocks period with pure chaotic one.
Stonygan
commented
3 years ago
ThreadStaker: (6d6f34a11aacbbe85ce6d7fafafebd468fc6b503) minted a block!6d6f34a11aacbbe85ce6d7fafafebd468fc6b503is masternode operator name not a block hash, block is accepted unless you see error in the log, that shows it's not.
Of course i check the linked Operator/Owner/Masternode-ID before i open this ticket. Minted Block is not in chain. Owner: 8Rd7AyAEw5rkxq4ziRf7oVn7PodojmMkx5 Operator: 8FRUWXGQKV1WWsZnA3e9TuAbQ8jH8UWQHx MN-ID: 6d666bf982fddda2091324b3e75c54adddec5d8b7b67c92722d498168a2abf8b
bvbfan
commented
3 years ago How you check that block is not part of main chain? https://explorer.defichain.io/#/DFI/mainnet/address/8Rd7AyAEw5rkxq4ziRf7oVn7PodojmMkx5
Stonygan
commented
3 years ago Operator was 100% on correct chain. Logfiles: 2021-06-11T13:57:15Z ThreadStaker: (6d6f34a11aacbbe85ce6d7fafafebd468fc6b503) minted a block! 2021-06-12T01:36:50Z ThreadStaker: (6d6f34a11aacbbe85ce6d7fafafebd468fc6b503) minted a block! 2021-06-13T16:04:26Z ThreadStaker: (6d6f34a11aacbbe85ce6d7fafafebd468fc6b503) minted a block!
But only 2 blocks in the main chain.
Stonygan
commented
3 years ago Here is another Example: 2021-06-14T07:45:13Z UpdateTip: new best=a6cba785eaa37a66853944be4215f7e5417cf920c1a68061fff74a46e2efcd8c height=922741 version=0x20000000 log2_work=81.152057 tx=3189662 date='2021-06-14T07:50:07Z' progress= 1.000017 cache=0.7MiB(5414txo) 2021-06-14T07:45:13Z UpdateTip: new best=6fa396b74b96e5d042e9ae322ecc059c52ee8a59eff4537839c08b22c731dead height=922742 version=0x20000000 log2_work=81.152064 tx=3189663 date='2021-06-14T07:44:49Z' progress= 0.999999 cache=0.7MiB(5416txo) 2021-06-14T07:45:13Z ThreadStaker: (eafc724239da11367a3c2ff32f36ac381efab068) minted a block! 2021-06-14T07:45:14Z UpdateTip: new best=a6cba785eaa37a66853944be4215f7e5417cf920c1a68061fff74a46e2efcd8c height=922741 version=0x20000000 log2_work=81.152057 tx=3189662 date='2021-06-14T07:50:07Z' progress= 1.000017 cache=0.7MiB(5417txo) 2021-06-14T07:45:14Z UpdateTip: new best=7e3a07c26d622d35d56d3130f2a6ab25dd5d5a6f0ac5ba5db6e797d966703cc1 height=922742 version=0x20000000 log2_work=81.152064 tx=3189663 date='2021-06-14T07:48:00Z' progress= 1.000010 cache=0.7MiB(5419txo) Not in chain. The Block 20 minutes later is included in the chain.
2021-06-14T08:04:09Z ThreadStaker: (eafc724239da11367a3c2ff32f36ac381efab068) minted a block!
I activate now debug=staking, maybe we get more infos next time.
Bushstar
commented
3 years ago At 07:45:13 you minted a block, however a second later two blocks came in at 07:45:14 which created a longer chain making your block not part of the main chain. The block times are not accurate, the time in them are set by the staker and can vary due to the super staker changes.
Stonygan
commented
3 years ago At 07:45:13 you minted a block, however a second later two blocks came in at 07:45:14 which created a longer chain making your block not part of the main chain. The block times are not accurate, the time in them are set by the staker and can vary due to the super staker changes.
Ok, i understand. That means the staker above minted the block 922742 with hash 6fa396b74b96e5d042e9ae322ecc059c52ee8a59eff4537839c08b22c731dead (nice hashending^^) and "at the same time" 2 new blocks with another hash of Block 922742 are minted and mine not longer valid. Correct? Is there a possibility to log this? Rollback, longer chain detected or something else.
tom-defi
commented
3 years ago @Bushstar The problem is that before the speedy miners this happened really seldomly. At the moment this is happening quite often.
We believe that this is caused by the Cake Nodes, the hashpower during the first second gives them a advantage by mining on the same server/network. Therefore they are able to replace blocks of other nodes with their chains of Instant-Blocks. Does this make sense?
Yes, it alternates ~ normal blocks period with pure chaotic one. - I would say the behavior is quite predictable now. There are always 3-10 Blocks which are found within the same second and afterwards we have to wait a few minutes for the next block.
uzyn
commented
3 years ago The chain is no longer moving smoothly and acts more jumpy.
This is the expected behavior when the nodes are doing super-mining.
@tom-defi has correctly pointed out the reason:
The main cause is the speedy masternode upgrade. This upgrade changes the hash power for the masternodes. The masternodes can now try between 200-700 hashes within the first second after each new block. Afterwards this is reduced to 1 hash/s.
When a block is found, the likelihood of the next block being found is up to 700x of that when no block is found. Inversely, a block is 700x less likely to be found if when no blocks are found over the last 3-5 mins.
Previously, this was what the superminers were exploiting. Now the playing field is leveled.
I assume that Cake is running most of their masternodes with multimining or/and within the same network. Therefore these masternodes instantly receive the block updates from other cake nodes while non cake nodes have to wait till the block is distributed over the network.
Cake does not run any nodes on private networks. All nodes are run publicly over the Internet and across multiple regions and multiple hosting providers to provide redundant P2P coverage globally. The nodes are also spread with more weightage over Europe as most users are from the Europe region. You can connect directly to our nodes. In fact the seed nodes are that.
Due to the reason of covered in 1. New blocks are more likely to be found when new blocks are published, therefore you see a chain. Also done some random sampling, the blocks that are found are not from the same machine nor from the same region. https://github.com/DeFiCh/ain/issues/525#issuecomment-860381813 has also shown that there are non-Cake blocks that are in-between the successive chain of blocks.
While it may seem that Cake has an advantage over the mining, it is not so. Also, Cake nodes have been under-mining for the last 1 week or so, due to the multi-miner bug and community super miner exploit over Cake, therefore accumulate more coinage.
Blockchain jumpiness is not a security issue, but it's not ideal.
One way for us to smoothen out is to reduce the backward and forward time acceptance window on the next upgrade and we might have to do that.
Thanks for the comments and findings.
tom-defi
commented
3 years ago @uzyn Thanks for your detailed comment.
I hope it's clear that I'm just trying to highlight some consequences (not bugs) of the speedy miner upgrade and that I'm not accusing Cake or anyone else to use it to their advantage.
I guess Cake is also using the multiminer feature? Don't you think that this feature will lead to the problem described above and the more masternodes are combined with multimining (or are located within the same datacenter) the bigger the advantage will be?
But you are right, it's too early to see the real consequences of this problem. We will have the analyze the block distribution over the next coming days/weeks before we can fully understand all the effects of the update.
Martin8617
commented
3 years ago Cake does not run any nodes on private networks. All nodes are run publicly over the Internet and across multiple regions and multiple hosting providers to provide redundant P2P coverage globally. The nodes are also spread with more weightage over Europe as most users are from the Europe region. You can connect directly to our nodes. In fact the seed nodes are that.
@uzyn could you please post IP of the named seed nodes. Thx
uzyn
commented
3 years ago seeds.defichain.com. 300 IN A 18.198.89.109
seeds.defichain.com. 300 IN A 35.156.182.15
seeds.defichain.com. 300 IN A 3.65.157.197
seeds.defichain.com. 300 IN A 3.66.108.218
seeds.defichain.com. 300 IN A 3.66.117.205
seeds.defichain.com. 300 IN A 3.67.114.138
seeds.defichain.com. 300 IN A 3.67.231.246
seeds.defichain.com. 300 IN A 18.159.137.143
seeds.defichain.com. 300 IN A 18.185.207.71Cake does not run any nodes on private networks. All nodes are run publicly over the Internet and across multiple regions and multiple hosting providers to provide redundant P2P coverage globally. The nodes are also spread with more weightage over Europe as most users are from the Europe region. You can connect directly to our nodes. In fact the seed nodes are that.
@uzyn could you please post IP of the named seed nodes. Thx
uzyn
commented
3 years ago Closing this now.
What happened:
Sometimes in the Logfile the Operator minted a block, but its not accepted by the network?. No info available why this happened.
What you expected to happen:
Accept the block or log why its not accepted
How to reproduce it (as minimally and precisely as possible):
Minting and check Logfile for message 'minted a block' and compare with blockchain and Extractor of this Block.
What are your environment parameters:
Operator on Debian 10, AIN-Version 1.7.10
Anything else we need to know?:
2021-06-11T13:57:14Z UpdateTip: new best=cc5624608a1cc23792d94afae978d838a99ecd9cc197f1f78334c3ace4a8f29b height=914511 version=0x20000000 log2_work=81.087534 tx=3155045 date='2021-06-11T14:00:56Z' progress= 1.000013 cache=0.5MiB(3531txo) 2021-06-11T13:57:15Z UpdateTip: new best=c9738dabe11367545ed67d56fdb6b00c7c78b72b585c3578bc8ed3e90530b565 height=914512 version=0x20000000 log2_work=81.087539 tx=3155046 date='2021-06-11T14:02:06Z' progress= 1.000017 cache=0.5MiB(3533txo) 2021-06-11T13:57:15Z ThreadStaker: (6d6f34a11aacbbe85ce6d7fafafebd468fc6b503) minted a block! 2021-06-11T13:58:06Z UpdateTip: new best=cc5624608a1cc23792d94afae978d838a99ecd9cc197f1f78334c3ace4a8f29b height=914511 version=0x20000000 log2_work=81.087534 tx=3155045 date='2021-06-11T14:00:56Z' progress= 1.000010 cache=0.5MiB(3534txo) 2021-06-11T13:58:07Z UpdateTip: new best=07922e9c06baabae9be3b137e63c40599d15e02a88eeb01cee220b1a3ebf7dd3 height=914512 version=0x20000000 log2_work=81.087539 tx=3155046 date='2021-06-11T14:00:42Z' progress= 1.000009 cache=0.5MiB(3536txo) 2021-06-11T13:58:07Z UpdateTip: new best=bc8fc413facbaa20c623f187a8bbfa608936ddba26bd24dc2686dfc104a3f192 height=914513 version=0x20000000 log2_work=81.087543 tx=3155052 date='2021-06-11T14:03:01Z' progress= 1.000017 cache=0.5MiB(3542txo)