Open beggsdl opened 5 months ago
CheckForkWarningConditions: Warning: Found invalid chain at least ~6 blocks longer than our best chain. Chain state database corruption likely.
forked testnet chain: 2754930 Testnet chain: 2570156
After 185,000 blocks we need to update the seeder GetRequireHeight to 2,760,000 https://github.com/DigiByte-Core/digibyte-seeder/blob/d75ea2c5796b823098a34d48af605177693d5047/db.h#L21
Thanks for discovering this, and testing like this. Definitely worth checking out. My initial hunch is this might be an issue specific to testnet if all the nodes on our current testnet have not been updated with many of the recent fixes we made. Could be competing forks causing issues here and DB corruption.
CheckForkWarningConditions
is a big clue here. We should not be getting "Found invalid chain"... this is leading to "Chain state database corruption likely." Looks like that's how its crashing
2024-01-29T10:38:29Z CheckForkWarningConditions: Warning: Found invalid chain at least ~6 blocks longer than our best chain. Chain state database corruption likely.
2024-01-29T10:39:06Z GUI: createDIB: CreateDIBSection failed (182x1, format: 6)
CheckForkWarningConditions
is defined here:
https://github.com/DigiByte-Core/digibyte/blob/e8247b0d596a443eb75f7bd6a9760534970eb25d/src/validation.cpp#L1366-L1383
Did you compile the node you are getting these logs from with the debug flag? For more verbose logs?
./configure --enable-debug
Also have you tried something like LLDB (macOS debugger) to debug this? Or GDB on linux
Let me clarify what "large number of transactions" means. I am talking about 3000 or more. On the one node, I think I counted just over 3000 coinbase txs that had gone to that node when it crashed. My other node was probably similar.
@JaredTate As for the debug flag, no. The version that I am running with the maturity fixes was given to me by @Jongjan88. The other nodes are just running RC2.
I do see there are still at least a few testnet nodes running on the forked chain.
To be honest, I don't even know what LLDB and GDB are, so no, I have not tried those.
I imagine in the next 12 hours or so, I should know if this same thing happens in 7.17.3.
Thanks to some help from @Jongjan88, we were able to flood my 7.17.3 core wallet with transactions. It did crash, just like the v8 wallet.
2024-01-30T00:47:30Z GUI: QPixmap::fromWinHICON(), failed to GetIconInfo() (The parameter is incorrect.)
2024-01-30T00:47:30Z GUI: QPixmap::fromWinHICON(), failed to GetIconInfo() (The parameter is incorrect.)
2024-01-30T00:47:30Z GUI: QPixmap::fromWinHICON(), failed to GetIconInfo() (The parameter is incorrect.)
2024-01-30T00:47:30Z GUI: qt_pixmapToWinHBITMAP, failed to create dibsection (The parameter is incorrect.)
2024-01-30T00:47:36Z [default wallet] AddToWallet d0e499372ca353308e9c4cf58f760972f58cb22ec8225f4a7af70d6f4c3dde5e new
2024-01-30T00:47:36Z [default wallet] AddToWallet d0e499372ca353308e9c4cf58f760972f58cb22ec8225f4a7af70d6f4c3dde5e
2024-01-30T00:47:36Z GUI: QPixmap::fromWinHICON(), failed to GetIconInfo() (The parameter is incorrect.)
2024-01-30T00:47:36Z GUI: QPixmap::fromWinHICON(), failed to GetIconInfo() (The parameter is incorrect.)
2024-01-30T00:47:36Z GUI: QPixmap::fromWinHICON(), failed to GetIconInfo() (The parameter is incorrect.)
2024-01-30T00:47:36Z GUI: qt_pixmapToWinHBITMAP, failed to create dibsection (The parameter is incorrect.)
2024-01-30T00:48:01Z block height for reward is 2570592
2024-01-30T00:48:01Z UpdateTip: new best=cfa0831532ddd549b30c3c10ca4851a5313c67138d92e5fff2e3e5e99fc7244a height=2570592 version=0x20000e02 algo=7 (odo) log2_work=50.965048 tx=2866303 date='2024-01-30T00:47:59Z' progress=0.022260 cache=1.8MiB(12765txo)
2024-01-30T00:48:01Z CheckForkWarningConditions: Warning: Found invalid chain at least ~6 blocks longer than our best chain.
Chain state database corruption likely.
2024-01-30T00:48:01Z [default wallet] AddToWallet f3a8ddbcab6c1e1ba4762617e912e0dc41c3d7474ece2474c300626714843d00 update
2024-01-30T00:48:01Z [default wallet] AddToWallet 02628935105b07e701f4f0397aa08c003232c7d37a2cad98ca09ab65fd68d003 update
2024-01-30T00:48:01Z [default wallet] AddToWallet 921578f38dab9b9bd6fa433d38441c940b40a03f65d3f32f39398d9e2f8fa005 update
2024-01-30T00:48:01Z [default wallet] AddToWallet 11b2d6566ed4f8d36a94ad8e03e346f7bf8fa5d857f05c8230cf74ea9e64f708 update
2024-01-30T00:48:01Z [default wallet] AddToWallet 36dfc5fb2eed05c419f1dd6cc4b34d42580814d8b4b91075fffe8427f6d39b0d update
2024-01-30T00:48:01Z [default wallet] AddToWallet b88a464f27865760de3769e6c9b0baef18db7655398b0f10eaf37106ee32450e update
2024-01-30T00:48:01Z [default wallet] AddToWallet 64158ab26875b46b897df14c9bb150aa7787a363ca6e4c77f84c83a4e7f12a0f update
2024-01-30T00:48:01Z [default wallet] AddToWallet 0076bd1b64b6d79e554d3e6f7e4fff0b89e833eaf2ddf7eafd42a4c125107d13 update
2024-01-30T00:48:01Z [default wallet] AddToWallet b098bf976d3b1065aeff20c3d7c14fdd52846deaa9aa37f0ddc527861b28071f update
2024-01-30T00:48:01Z [default wallet] AddToWallet ca6d4145cb638e9d8ae1e1668c99b2d066209f18fd1484b5f39df38e8c3dbb23 update
2024-01-30T00:48:01Z [default wallet] AddToWallet 88f229fb43ed712ea569901976308b140b0409f66cc686fa804338b4c5a81324 update
2024-01-30T00:48:01Z [default wallet] AddToWallet 5a99902f0fee8d1bedb2850522925980bbd33938f407a2be95af7d8b40d49831 update
2024-01-30T00:48:01Z [default wallet] AddToWallet 0af79451bc427412cc967128a5bd8759bd7465c60712e2619ba83dc99e90dc58 update
2024-01-30T00:48:01Z [default wallet] AddToWallet d0e499372ca353308e9c4cf58f760972f58cb22ec8225f4a7af70d6f4c3dde5e update
2024-01-30T00:48:01Z [default wallet] AddToWallet a39449143b068f5cd62e809636952ec88c121cbf0942ab59e94c3ca273fcb869 update
2024-01-30T00:48:01Z [default wallet] AddToWallet a4d938ebfbdc3f20e9a1eece875d1ec270200b3693f1974aef2254b8f4c0aa79 update
2024-01-30T00:48:01Z [default wallet] AddToWallet fb4d52da08c9de5441d12c82de91fbd330bfd2562989bc30fa341c637c248682 update
2024-01-30T00:48:01Z [default wallet] AddToWallet 12730ad87ef437efcaec70ccb2e2267a5f3a1af98e831e6a759f7eaa8e19ed8e update
2024-01-30T00:48:01Z [default wallet] AddToWallet ed1e852622c029a8299775b49534a8b03a33cf08572274988a56db85f1461fa3 update
2024-01-30T00:48:01Z [default wallet] AddToWallet e5cd52c192601daf16b37a456c8deef86f5da55b4f328f92d59412473f74fca5 update
2024-01-30T00:48:01Z [default wallet] AddToWallet 76e158fa942894b8117b347a39f543993c8a2102563aebe8ec22ca9de19282b2 update
2024-01-30T00:48:01Z [default wallet] AddToWallet d1e476a672a4188289b22b922caa8c2c412357b6418fc00ea92f6fe54dcfceb3 update
2024-01-30T00:48:01Z [default wallet] AddToWallet edb94379f66d94bb3e9f9a84fb3ab8b67d625df88a8615082213e5f9b1d4b6bf update
2024-01-30T00:48:01Z [default wallet] AddToWallet 977c19e7d3e3ba9aac453e19650ac2fdeb805666eab916a1c5f3b7dc4f49f4c2 update
2024-01-30T00:48:01Z [default wallet] AddToWallet 3f07a824339bb583f2ea52098be5c3665f7c9888039ef2c81f8e71dc998fbac6 update
2024-01-30T00:48:01Z [default wallet] AddToWallet 2aa2d41329add10c1d7522c73b64643b88a380eecd76f3fd591e64ec03be70c9 update
2024-01-30T00:48:01Z [default wallet] AddToWallet 3ca536880db9d480b5d552591734f1510891e36582d97a25d0d89bf934f9aacc update
2024-01-30T00:48:01Z [default wallet] AddToWallet ae73f7a2640511aced76139c13de4b802a516b674f70f18faddf5794578ef8cf update
2024-01-30T00:48:01Z [default wallet] AddToWallet 5318706b27a1f327c83314c61d124c4d2942e878f12e28418f2fe126323763d5 update
2024-01-30T00:48:01Z [default wallet] AddToWallet 0ee0fe18c61fa832d9e31144e2c5fe7e4687c1eec59c4987d71aa5a22fb9abd9 update
2024-01-30T00:48:01Z [default wallet] AddToWallet 6b560c8d1fec2365b10b50d71efa0de29ea16b73d623ba74fd68993144e0d4ef update
2024-01-30T00:48:01Z [default wallet] AddToWallet 354c7a073d3ee884016b4d060c8b0390eda4491ca3b80d13caa4d07ec71bccf8 update
2024-01-30T00:48:01Z [default wallet] AddToWallet ca99b1d6a1447c6ed284ac873c9819101be927d6c232fe92d47821c23fab4afd update
2024-01-30T00:48:01Z [default wallet] AddToWallet 0fe2f04fbe879d5236c1a67bae06c42e0238416cd3c40e21787d226580d641ff update
2024-01-30T00:48:06Z [default wallet] AddToWallet 7b04bb1b3bb8c855b3c69a22cd9000cc886d55cb33e03af368b31f761d38e879 new
2024-01-30T00:48:06Z [default wallet] AddToWallet 7b04bb1b3bb8c855b3c69a22cd9000cc886d55cb33e03af368b31f761d38e879
2024-01-30T00:48:06Z [default wallet] AddToWallet 23a99caad85563debf23f88fcf7fafa7904744b87038f047d39d17f4bc3bcc00 new
2024-01-30T00:48:06Z GUI: QPixmap::fromWinHICON(), failed to GetIconInfo() (The parameter is incorrect.)
2024-01-30T00:48:06Z [default wallet] AddToWallet 23a99caad85563debf23f88fcf7fafa7904744b87038f047d39d17f4bc3bcc00
2024-01-30T00:48:06Z GUI: QPixmap::fromWinHICON(), failed to GetIconInfo() (The parameter is incorrect.)
2024-01-30T00:48:06Z GUI: QPixmap::fromWinHICON(), failed to GetIconInfo() (The parameter is incorrect.)
2024-01-30T00:48:06Z GUI: qt_pixmapToWinHBITMAP, failed to create dibsection (The parameter is incorrect.)
2024-01-30T00:48:06Z GUI: QPixmap::fromWinHICON(), failed to GetIconInfo() (The parameter is incorrect.)
2024-01-30T00:48:06Z GUI: QPixmap::fromWinHICON(), failed to GetIconInfo() (The parameter is incorrect.)
2024-01-30T00:48:06Z GUI: QPixmap::fromWinHICON(), failed to GetIconInfo() (The parameter is incorrect.)
2024-01-30T00:48:06Z GUI: qt_pixmapToWinHBITMAP, failed to create dibsection (The parameter is incorrect.)
2024-01-30T00:48:08Z block height for reward is 2570593
2024-01-30T00:48:08Z UpdateTip: new best=e8d2784f15081cd4f24fd49d928036f25a724ec32654837652448be6264ef6a3 height=2570593 version=0x20000802 algo=4 (qubit) log2_work=50.96505 tx=2866304 date='2024-01-30T00:48:04Z' progress=0.022260 cache=1.8MiB(12766txo)
2024-01-30T00:48:08Z CheckForkWarningConditions: Warning: Found invalid chain at least ~6 blocks longer than our best chain.
Chain state database corruption likely.
2024-01-30T00:48:08Z [default wallet] AddToWallet cbd13f8116cc8f29728e565963ff9f39609397187a4a0839ec90d297018c086f new
2024-01-30T00:48:08Z GUI: QPixmap::fromWinHICON(), failed to GetIconInfo() (The parameter is incorrect.)
2024-01-30T00:48:08Z GUI: QPixmap::fromWinHICON(), failed to GetIconInfo() (The parameter is incorrect.)
2024-01-30T00:48:08Z GUI: QPixmap::fromWinHICON(), failed to GetIconInfo() (The parameter is incorrect.)
2024-01-30T00:48:08Z GUI: qt_pixmapToWinHBITMAP, failed to create dibsection (The parameter is incorrect.)
2024-01-30T00:48:28Z block height for reward is 2570594
2024-01-30T00:48:28Z UpdateTip: new best=aded5a7a021fae53e8433241fa5ca9460c6dfa5355dd6d51979837020803eeb2 height=2570594 version=0x20000e02 algo=7 (odo) log2_work=50.965051 tx=2866307 date='2024-01-30T00:48:26Z' progress=0.022260 cache=1.8MiB(12769txo)
2024-01-30T00:48:28Z CheckForkWarningConditions: Warning: Found invalid chain at least ~6 blocks longer than our best chain.
Chain state database corruption likely.
2024-01-30T00:48:28Z [default wallet] AddToWallet 23a99caad85563debf23f88fcf7fafa7904744b87038f047d39d17f4bc3bcc00 update
2024-01-30T00:48:28Z [default wallet] AddToWallet 7b04bb1b3bb8c855b3c69a22cd9000cc886d55cb33e03af368b31f761d38e879 update
2024-01-30T00:49:07Z block height for reward is 2570595
2024-01-30T00:49:07Z UpdateTip: new best=fb02a794360455f215bbb59fe5162e3ccfcbad26cc455f0009115b2aaf27b6d6 height=2570595 version=0x20000002 algo=1 (scrypt) log2_work=50.965052 tx=2866308 date='2024-01-30T00:48:30Z' progress=0.022260 cache=1.8MiB(12770txo)
2024-01-30T00:49:07Z CheckForkWarningConditions: Warning: Found invalid chain at least ~6 blocks longer than our best chain.
Chain state database corruption likely.
2024-01-30T00:49:07Z [default wallet] AddToWallet 6ee6b25566da4d9756228e0fd5c40e8d3138b68744526bc16f53393a43a65d68 new
2024-01-30T00:49:07Z GUI: QPixmap::fromWinHICON(), failed to GetIconInfo() (The parameter is incorrect.)
2024-01-30T00:49:07Z GUI: QPixmap::fromWinHICON(), failed to GetIconInfo() (The parameter is incorrect.)
2024-01-30T00:49:07Z GUI: QPixmap::fromWinHICON(), failed to GetIconInfo() (The parameter is incorrect.)
2024-01-30T00:49:07Z GUI: qt_pixmapToWinHBITMAP, failed to create dibsection (The parameter is incorrect.)
2024-01-30T00:49:09Z [default wallet] AddToWallet 804c5cb1b2847df47acae51651d679e1227b963f04993289d4f3408b9f9d047a new
2024-01-30T00:49:09Z [default wallet] AddToWallet 804c5cb1b2847df47acae51651d679e1227b963f04993289d4f3408b9f9d047a
2024-01-30T00:49:09Z GUI: QPixmap::fromWinHICON(), failed to GetIconInfo() (The parameter is incorrect.)
2024-01-30T00:49:09Z GUI: QPixmap::fromWinHICON(), failed to GetIconInfo() (The parameter is incorrect.)
2024-01-30T00:49:09Z GUI: QPixmap::fromWinHICON(), failed to GetIconInfo() (The parameter is incorrect.)
2024-01-30T00:49:09Z GUI: qt_pixmapToWinHBITMAP, failed to create dibsection (The parameter is incorrect.)
2024-01-30T00:49:10Z GUI: qt_pixmapToWinHBITMAP, failed to create dibsection (The parameter is incorrect.)
2024-01-30T00:49:10Z GUI: qt_pixmapToWinHBITMAP, failed to create dibsection (The parameter is incorrect.)
2024-01-30T00:49:10Z GUI: qt_pixmapToWinHBITMAP, failed to create dibsection (The parameter is incorrect.)
2024-01-30T00:49:10Z GUI: qt_pixmapToWinHBITMAP, failed to create dibsection (The parameter is incorrect.)
2024-01-30T00:49:11Z GUI: createDIB: CreateDIBSection failed.
2 transactions per second. Crash after 1869 transactions.
2 transactions per second. Crash after 1869 transactions.
Mining rewards from mining 3 algos were also going to it for approximately 12 hours before we flooded it with more txs.
We are talking about the QT desktop core client crashing here correct? aka the GUI? I have seen the client crash in the past while flooding a desktop GUI client with 1000s of tx's. In both cases, I switched to digibyte-cli on a server via the command line and it handled any use case I threw at it. In those two cases, it was an exchange and a mining pool.
Actually I had the same wallet.dat on a desktop GUI and a server so I could watch it. The desktop crashed but digibyte-cli kept up just fine. Also similar problem when de-dusting 1000s of artifact UTXOs on a mining pool. It memory crashed the desktop GUI but cli was fine.
So my question is how often will a normal end user send 1000s of txs in a few seconds to a desktop GUI? And at what point is it appropriate to use the command line digibyte-cli on a server. Now if this same bug is persistent on a digibyte-cli is definitely something to look much deeper at.
You are correct @JaredTate, it is the GUI. I had the same thought that this might not be a practical real-world scenario.
It’s not just 1000’s in a few seconds though. In most of my cases, it was over 1 to 3 days. But again, maybe not a scenario that people would really run into.
You are correct @JaredTate, it is the GUI. I had the same thought that this might not be a practical real-world scenario.
It’s not just 1000’s in a few seconds though. In most of my cases, it was over 1 to 3 days. But again, maybe not a scenario that people would really run into.
I would be very curious to see if the same problem persists with digibyte-cli on a server. I doubt it will. My hunch is this is a memory leak error in the QT GUI related to some other crashes we have seen in the past. I have spent some time trying to isolate memory leaks in the QT, which we have a few of them for sure, but its not an easy task to isolate. I think we have some pointers in a few places not properly being closed.
This time i tested 4000 transactions. 2 per second. Digibyte QT remained stable. During the transactions I also put the GUI to the test. constantly minimized and open. Switch to Information, Console, Network traffic and Peerlist. trying to crash it! DigiByte QT was already running when i start the test for 6 days and receiving transactions from a miner.
I am mining now for more then a month on Testnet. Sometimes with 5 miners at the same time. And send all the coins to one QT wallet. I never had a crash.
System 1: DigiByte QT Dev branch 2024-01-31 ( Sender ) System 2: DigiByte QT Dev branch with pr160 ( Receiver 16gb ram / Debian 12 )
Small memory leaks could be the cause. But don't forget the rc2 nodes without the coinbase maturity fix.
https://github.com/DigiByte-Core/digibyte/issues/164#issue-2106570920
Now that I look at my NodeB and NodeC again, they are running a version with PR157, so the coin maturity fix was in there. I am going to put the newest version in place and will see if I get the same crash after a day or so.
Our testnet chain is corrupt atm. So this crash will probably not stop till 2024-03-01. I think the best thing we can do with this pr is to leave it open. After 2024-03-01 we can ban/block nodes lower than block 2,754,931 on the testnet chain.
I updated the node versions being used in the issue description. None of my nodes were RC2, they all had at least PR157.
@beggsdl are you using Descriptor Wallet or Legacy Wallet?
https://github.com/DigiByte-Core/digibyte/issues/199#issuecomment-2014853317
@beggsdl are you using Descriptor Wallet or Legacy Wallet?
It would be legacy because I have never checked the Descriptor checkbox when creating a new wallet.
Was this fixed with RC4? Thanks for your feedback
Setup (solo cpu mining against each of these testnet nodes): NodeA:
NodeB:
NodeC:
NodeD:
Solo mining against testnet core NodeA with a wallet (walletA) setup. This core node is configured for mining scrypt. I have 2 other testnet nodes currently configured for mining sha256 and quibit. All coinbase transactions are directed to walletA. This wallet (walletA) has several addresses for it to receive coinbase txs from solo mining scrypt, sha256, and qubit (some from odo and skein also). With walletA, after some period of time (typically about once per day, with coinbase txs from 3 algos going to this wallet), the node will crash. The following messages are visible in debug.log:
Changing the algo mining against NodeA does not impact the result. I still get the same result of NodeA crashing about 1 time per day.
I then changed the sha256 coinbase txs (mining against NodeB) to be directed to walletB on NodeB to see if NodeB would also crash at some point. NodeB did crash. It took a bit longer to do so, maybe because there were coinbase txs from only one algo going to it. The log is basically the same as above, except some different messages at the very end:
My next test was to direct all coinbase txs to walletD on NodeD instead of to a wallet on a node being mined against. NodeD was NOT being mined against. It was only receiving coinbase txs from the three solo miners. I moved the wallet.dat file from the NodeA folder to the NodeD folder, so the wallet receiving all the coinbase txs had the same addresses and all the previous coinbase txs. NodeA was still being mined against, but now did not have a wallet configured. After running like this for about 24 hours, NodeD crashed:
My conclusion is a core wallet that is open/running and receiving lots of transactions will eventually crash. If this core wallet is being mined against, mining will obviously stop when it crashes. The crashing from receiving lots of transactions does not seem to be impacted at all by mining against that core node or not.
The only test I have not yet completed is to direct all coinbase txs to the 7.17.3 node to see if it will also crash. I am currently doing this test and update with the result.