Can't sync : Block verification failed

[1towia]

Hi, I'm trying to sync but the process stuck at block height 798358/798359. I tried several times and the same problems occur.

There is a lot of errors in the aeond.log : http://pastebin.com/LiBBWPBv

These errors :

2017-Mar-02 16:08:33.786044 [P2P5]Block with id: <72dc340601a863973fc1b818a554b1cbff6fdde317ad675dacfb125d88b1814d>have at least one unknown transaction with id: <2d1f88a0c67ca908695abd69dcd97c3039dee0041ce013df7f946dfe1d6bf73c>
2017-Mar-02 16:08:33.786509 [P2P5][5.9.102.108:11180 OUT]Block verification failed, dropping connection

Then :

2017-Mar-02 16:10:13.096962 [P2P2]----- BLOCK ADDED AS ALTERNATIVE ON HEIGHT 798358
id: <72dc340601a863973fc1b818a554b1cbff6fdde317ad675dacfb125d88b1814d>
PoW:    <f570bebd911eeba5817a1770ad16ab3d30ab8616702813bd60193a7b06000000>
difficulty: 320369628

And :

2017-Mar-02 16:10:18.978448 [P2P2]ERROR /root/aeon/src/cryptonote_core/blockchain_storage.cpp:783 alternative chain have wrong connection to main chain
2017-Mar-02 16:10:18.978885 [P2P2]Block recognized as orphaned and rejected, id = <e542a4be58bd653e989c0b21409681c39db05d45c46807e60cb6924c7f935c06>
2017-Mar-02 16:10:18.979185 [P2P2][178.63.72.22:11180 OUT]Block received at sync phase was marked as orphaned, dropping connection

edit: It seems that the block 798358 have a strange transaction in it. tx hash : 86358f3aecbb5a1e5319bb0b771dd7bc139d043adab6a63d8c1577e74157c760 The global indexes are all 0 :

http://chainradar.com/aeon/transaction/86358f3aecbb5a1e5319bb0b771dd7bc139d043adab6a63d8c1577e74157c760 http://chainradar.com/aeon/block/798358

[iamsmooth]

being investigated thanks

[gosha12]

the same mistake, I cannot synchronize a purse

2017-Mar-03 11:51:12.215414 [P2P6]Block with id: <72dc340601a863973fc1b818a554b1cbff6fdde317ad675dacfb125d88b1814d>have at least one unknown transaction with id: <2d1f88a0c67ca908695abd69dcd97c3039dee0041ce013df7f946dfe1d6bf73c> 2017-Mar-03 11:51:12.231014 [P2P6][5.9.69.142:40838 INC]Block verification failed, dropping connection 2017-Mar-03 11:51:12.231014 [P2P6]Ip 5.9.69.142 blocked.

[Congabonga]

Happening here too. Cannot get past block 798363.

SYNCHRONIZATION started
2017-Mar-03 09:39:54.528452 [P2P7]Block recognized as orphaned and rejected, id = <2632f348d1f895a708b992fe0116b8069285b4ec26fdbc1443987a729dc25443>
2017-Mar-03 09:39:54.528452 [P2P7][98.238.231.31:50364 INC]Block received at sync phase was marked as orphaned, dropping connection
2017-Mar-03 09:39:55.684765 [P2P7]Block recognized as orphaned and rejected, id = <2632f348d1f895a708b992fe0116b8069285b4ec26fdbc1443987a729dc25443>
2017-Mar-03 09:39:55.684765 [P2P7][91.152.122.132:48442 INC]Block received at sync phase was marked as orphaned, dropping connection
2017-Mar-03 09:39:55.919177 [P2P7][98.238.231.31:50394 INC]Sync data returned unknown top block: 798363 -> 798772 [409 blocks (1 days) behind]
SYNCHRONIZATION started
2017-Mar-03 09:39:57.544262 [P2P1][91.152.122.132:48444 INC]Sync data returned unknown top block: 798363 -> 798772 [409 blocks (1 days) behind]
SYNCHRONIZATION started
2017-Mar-03 09:40:00.419392 [P2P7]Block recognized as orphaned and rejected, id = <2632f348d1f895a708b992fe0116b8069285b4ec26fdbc1443987a729dc25443>
2017-Mar-03 09:40:00.435013 [P2P7][91.152.122.132:48444 INC]Block received at sync phase was marked as orphaned, dropping connection
2017-Mar-03 09:40:00.700651 [P2P1]Block recognized as orphaned and rejected, id = <2632f348d1f895a708b992fe0116b8069285b4ec26fdbc1443987a729dc25443>
2017-Mar-03 09:40:00.716275 [P2P1][98.238.231.31:50394 INC]Block received at sync phase was marked as orphaned, dropping connection
2017-Mar-03 09:40:00.716275 [P2P1]Ip 98.238.231.31 blocked.
2017-Mar-03 09:40:01.747608 [P2P7][91.152.122.132:48446 INC]Sync data returned unknown top block: 798363 -> 798772 [409 blocks (1 days) behind]
SYNCHRONIZATION started
2017-Mar-03 09:40:02.435121 [P2P7]Block recognized as orphaned and rejected, id = <2632f348d1f895a708b992fe0116b8069285b4ec26fdbc1443987a729dc25443>
2017-Mar-03 09:40:02.450745 [P2P7][91.152.122.132:48446 INC]Block received at sync phase was marked as orphaned, dropping connection
2017-Mar-03 09:40:03.935222 [P2P7][91.152.122.132:48448 INC]Sync data returned unknown top block: 798363 -> 798772 [409 blocks (1 days) behind]
SYNCHRONIZATION started
2017-Mar-03 09:40:04.638363 [P2P1]Block recognized as orphaned and rejected, id = <2632f348d1f895a708b992fe0116b8069285b4ec26fdbc1443987a729dc25443>
2017-Mar-03 09:40:04.653984 [P2P1][91.152.122.132:48448 INC]Block received at sync phase was marked as orphaned, dropping connection

[iamsmooth]

Here is an update https://bitcointalk.org/index.php?topic=641696.msg18047243#msg18047243

There is a serious (old) bug in the blockchain code. I'm working on finalizing a fix.

[billaue2]

As I saw at http://chainradar.com/aeon/block/798358 , the mentioned block is still valid and my mining pool (http://aeon.sumominer.com) is still running without any problem so I'm not sure what really happened. Also, I don't think it's easy to roll back to a bypass block leaving all miner's transactions invalid though a fix is of course necessary. So can you release a hardfork that marks and force-adds the block 798358 (Monero did the same for another bug, if I remember correctly)?

[Aidess]

My mining pool (http://aeon.rupool.tk) running

[billaue2]

Meanwhile, anyone who has sync problem can download the (nearly) latest blockchain snapshot from here http://aeon.sumominer.com/download/aeon_blockchain.tar.gz2 (Linux only)

[iamsmooth]

So can you release a hardfork that marks and force-adds the block 798358 (Monero did the same for another bug, if I remember correctly)?

Unfortunately no. The nature of the issue is different. In this case the block has been accepted by nodes but the data store is in an invalid state. While your node (as with others) appears to have accepted the block, they didn't do so correctly.

As things stand now, the blocks stored in the node database at height 798358 are invalid because they are missing a transaction. They can't be part of a valid chain and were only accepted due to a bug in the software which has now been corrected causing them to be correctly rejected. The transaction that appears to be present in that block is actually a different one (txid does not match the block header)

Without extensive analysis it is impossible to know what problems this may cause for subsequent blocks. To undertake such analysis would take an indefinite time and if it turned out that rejecting the invalid chain were still necessary we would be left with an even larger amount of blocks to rejected.

[iamsmooth]

This issue has been resolved by release 0.9.8.0. Please update all nodes ASAP.

https://github.com/aeonix/aeon/releases/tag/v0.9.8.0

As mentioned in the update notice linked above, I will accept well-documented claims for reimbursement from the development fund if you have mined blocks on the invalidated chain or otherwise lost coins. There is no other feasible solution.

[billaue2]

Interesting case in term of concesus. We'll need 51% hash power to make this happen, right?

[Aidess]

2017-Mar-04 12:52:35.582504 ERROR /aeon_0.9.8.0/src/cryptonote_core/checkpoints.cpp:71 CHECKPOINT FAILED FOR HEIGHT 798358. EXPECTED HASH: <804c7fe07511d9387e7cda534c9e8b644d406d8d0ff299799a8177850d4e75a0>, FETCHED HASH: <72dc340601a863973fc1b818a554b1cbff6fdde317ad675dacfb125d88b1814d>
2017-Mar-04 12:52:35.582561 ERROR /aeon_0.9.8.0/src/cryptonote_core/blockchain_storage.cpp:99 checkpoint fail, blockchain.bin invalid
2017-Mar-04 12:52:35.582580 ERROR /aeon_0.9.8.0/src/cryptonote_core/cryptonote_core.cpp:128 Failed to initialize blockchain storage
2017-Mar-04 12:52:35.582602 ERROR /aeon_0.9.8.0/src/daemon/daemon.cpp:190 Failed to initialize core

It is evident need to re-sync blockchain

[iamsmooth]

We'll need 51% hash power to make this happen, right?

In a sense yes. The corrected code will not recognize the existing chain^1 so to people running the new code, the miners still on the other chain simply don't exist. They might as well be mining another coin. However, there is nothing to prevent people from continuing to use the old code and they would only switch to the new chain if it has >50% hash power. I have no idea what the eventual outcome of the data store corruption might be. Perhaps it could be okay, perhaps not?

^1 for two reasons, one being the checkpoint that @Aidess just noted (which I mostly added to force a resync) but even if you take out that checkpoint you will see that the corrected validation code still rejects the block.

[billaue2]

The danger of using old codes is that it may introduce more invalid blocks and one day old blockchain would be no longer sync, so I'll go with new codes. Luckily, my pool is nearly half of hash power but I'll be happy if other pools will do the same, especially the Minergate guys, to make sure the network won't be screwed up. Anyone else can confirm?

[Aidess]

I'm updating

[billaue2]

@iamsmooth Another concern is if the exchanges like Bittrex, Changelly etc. are still using old blockchain they may lost money and miners who mine on hard-forked blockchain cannot sell AEON there. Do you have any contact to send notice to them?

[billaue2]

I think that's why we'll need majority to upgrade to reach consensus. I still need more confirmation from other pools, maybe my final decision will come on Monday.

[sammy007]

I still need more confirmation from other pools

What consensus you wanna reach here? You either don't understand wtf you are doing or just trying to troll us here. There is emergency hard fork, you either support it or you are running yet another "classic" and wasting your customers money.

[billaue2]

Trolling? Thanks, you just gave me the idea to run "classic" version while seeking for better solution. Maybe. :$

[billaue2]

Sorry for not in good mood now. After thought, I'll go with new fork only if major exchanges accept it and the compensation for miners is clear. So, I'll need a bit more time for sure.

[sammy007]

Time to decide what? To go with corrupted blockchain or with correct chain? All exchanges aware of problems and suspended withdrawals and deposits long ago. Miners are always at risk like all users of experimental software, i vote against bail-out for miners because both miners and pool ops must be responsible, follow repository notifications, follow development. it's not set and forget business.

[notsuhz]

aeon.kryptopool.com has been updated to 0.9.8.0

[quangvu3]

i'm a miner and although the rollback won't cost me a fortune, i don't like that at all. i hardly remember a case that blockchain has to rollback and it's also different case of ETH vs ETC.

P/s i don't like @sammy007 attitude that pass all responsibility to miners. i think only a few loyal miners make AEON still alive although the proj had been almost abandoned for months and this not a good sign :(

[iamsmooth]

@sammy007 miners did nothing wrong from the time of the original problem until the first announcement of trouble and likely chain corruption several hours later. However, after that I agree it was a responsibility to decide to stay informed and decide what to do

Original announcement that was linked here above:

Unfortunately it looks like there is going to need to be a rollback to block 798358. I do not recommend making any further transactions at this time until after the fix.

Coins that you sent since block 798358 will be returned to you if you rescan your wallet after the fix. If you received coins after block 798358 they will be lost unless you get the sender to resend them. This is unfortunate, but it is the nature of experimental blockchain software. I will consider reimbursement from the development fund if anyone wants to make a well-supported claim.

@billaue2 bitttrex, crytpopia have all been notified. Both suspended wallet transactions when told of the original problem. I'm not sure of contact information for changelly. If someone has contact info please contact them or pass it to me.

@quangvhg I agree with you people who have stuck with this coin when the original developer abandoned it and it was worth nothing to almost nothing for a long time deserve our respect. I understand the frustration of 'wasted' mining but I would ask that everyone pull together and try to restore things to normal asap. Trying to stick with a broken chain makes no sense and would destroy trust when it very easily could break again, and in a worse manner.

We are in the process of rebasing to the Monero code which, while it has its own potential problems (nothing is guaranteed error free) is much more actively developed and tested. Let's pull together, move forward past this frustrating (for everyone) event, and continue building the community and the coin.

[billaue2]

Thank @iamsmooth, I'm on the way to new fork now. I agree that we'll have to move forward and build better community.

My pool is shutdown now, all miners are redirected to aeon.kryptopool.com @suhz

[Aidess]

As a result, about 1,000 coins disappeared from the reserve to the pool

[billaue2]

@Aidess our miners would have about 7k coins missed, and http://aeon.sumomier.com now has a brand new start

[iamsmooth]

Hi, all

Unfortunately I still see a problem with the data store. It looks like there is going to need to be another fix which may or may not invalidate blocks. I don't know yet and I'm on a plane right now so I can't analyze it right away.

Will follow up soon.

[iamsmooth]

preliminary analysis does not show a problem but i will be doing some further tests. for now i believe the new release and data store or are okay. that is not a guarantee, just my best opinion

[iamsmooth]

After some more investigation I don't see any problem. I was misreading the output of one of the daemon commands. Sorry for the false alarm. I'll continue to keep an eye on things but at this moment it looks like everything is working correctly.

[Aidess]

@billaue2 But I also started my pool http://aeon.rupool.tk

[iamsmooth]

Hello all. Looks like the problem may not yet be completely solved. I've added some additional sanity checks to my development version of the code which tripped possibly indicating more closely the source of the issue and I'm working to resolve it. At this point I can only advise to be careful and await further announcements.

EDIT: clarify that the suspicious output may or may not be evidence of a problem (not yet determined)

[Aidess]

2017-Mar-09 15:07:28.643379 [P2P9][138.201.21.7:58425 INC]Sync data returned unknown top block: 800432 -> 798359 [2073 blocks (-5 days) ahead] 
SYNCHRONIZATION started
2017-Mar-09 15:07:29.031796 [P2P1]ERROR /aeon-0.9.8.0/src/cryptonote_protocol/cryptonote_protocol_handler.inl:362 [138.201.21.7:58425 INC]sent wrong NOTIFY_RESPONSE_GET_OBJECTS: block with id=6ac6ec217c69ca0f535ae5acaa2225d1d9a55fd01813cd752d1a399404c32dfc, tx_hashes.size()=2 mismatch with block_complete_entry.m_txs.size()=0, dropping connection

[iamsmooth]

@Aidess the peer is sending invalid data when syncing, which your node is rejecting. That by itself is nor harmful but clutters the log message and I'll be adding that missing ban case in the new update.

[sskyspear]

2017-Dec-03 12:22:53.681802 [P2P3]Block with id: <9135f7c24d67fc7ad618e231ce2d38f59307b50ae2b23e3aa8640f02aebf0f45> can't be accepted for alternative chain, block height: 406 blockchain height: 45198 2017-Dec-03 12:22:53.697428 [P2P3][88.162.4.146:11180 OUT]Block verification failed, dropping connection 2017-Dec-03 12:25:40.579879 [P2P1][sock 788] Some problems at write: ╙фрыхээ√щ їюёЄ яЁшэєфшЄхы№эю ЁрчюЁтры ёє∙хёЄтє■∙хх яюфъы■ўхэшх:10054 2017-Dec-03 12:25:42.395309 [P2P2][74.73.146.248:11180 OUT]Sync data returned unknown top block: 54320 -> 897435 [843115 blocks behind] SYNCHRONIZATION started

[iamsmooth]

@sskyspear This is not a problem. It means the other node has an invalid chain. (I believe someone tried to create a fork or something and mined starting from the genesis block.) Your node is correctly rejecting it, and should still be syncing from other nodes.

[sskyspear]

thx a lot for your reply. sync was very slow (5mbit internet, 4g ram), seems its very dependent on theese 2. Is there a way to move or specify blockchain location on hard drive?

[stoomart]

@sskyspear 4Gb won't due for aeond because it keeps the entire blocks in memory, the slowness is due to your swap space getting crushed. Either upgrade RAM (16Gb worked for me) or sync your wallet with a public daemon.