nnmfnwl commented 9 months ago

Describe the bug version 0.22.0 compiled from source is not working. behavior depends by used compiler. Follow the text for details:

To Reproduce compile manually with gcc or clang at 0.22. commit-id 78a88a94d6943a1af9868415217f41f1cf93f9b6 on Latest Debian stable.

Expected behavior

not doing segfault(139) when compiled by clang.
not showing corrupted messages like "Staker worker thread started for \x00\x00��\x07\x00\x00\x00w*q*DF**king" it seems like data been overwritten by another thread worker. Expected is also to staking be working when wallet unlocked but not.

Details

Using latest stable Debian, fully up to date, many other chains working 100%.
test repeated with deleted all source to be sure it is not caused by makerules.
when using export CC=clang export CXX=clang++ pkoin node Segfaults(139) probably right after exiting worker thread loadblk thread exit because dmesg: b-loadblk[2827261]: segfault at 7f2fc3542eff ip 00007f30b2971b30 sp 00007f2fbaffb658 error 4 in libc.so.6[7f30b2845000+155000] likely on CPU 0 (core 0, socket 0)
when using default gcc compiler, the debug log contains probably invalid, maybe by some other thread worker overwritten message "Staker worker thread started for \x00\x00��\x07\x00\x00\x00w*q*DF**king" and also it makes that staking is not able to be turn on.
this bug was probably there before but with gcc been doing overwrites some minor data.

Part of debug log from clang compiled node before segfault:


block tree size = 2711708
Best block: 1d5ea7388fd79f050d8ecdc4a48e8ab955b9f603fbd393cb175d0ba3ae56285f (2550467)
loadblk thread start
Imported mempool transactions from disk: 0 succeeded, 0 failed, 0 expired, 0 already there, 0 waiting for initial broadcast
Bound to 127.0.0.1:37070
Bound to 127.0.0.1:38084
init message: Loading P2P addresses...
loadblk thread exit

echo $? 139

dmesg b-loadblk[2841361]: segfault at 7f4b27543000 ip 00007f4c155718ca sp 00007f4b1dffb658 error 4 in libc.so.6[7f4c15445000+155000] likely on CPU 0 (core 0, socket 0)



Any ides how to workaround this?

andyoknen commented 9 months ago

Hi @nnmfnwl

Could you provide the compilation steps that you are following? I can try to run a docker container to reproduce the problem. There may be problems with dependencies - leveldb or boost or ...

The exact Debian version is also needed.

Also, which database do you use? Snapshot or downloaded from scratch using the instance you are building?

nnmfnwl commented 9 months ago

Hi @andyoknen Thank you for reply.

The exact Debian version is also needed.

It is Debian stable, always fully upgraded, now Release 12 (bookworm) 64-bit, Kernel Linux 6.1.0-13-amd64 x86_64.

Also, which database do you use? Snapshot or downloaded from scratch using the instance you are building?

It was downloaded database from scratch with previous versions and migrated into latest with 0.22.0. compiled by gcc.

Could you provide the compilation steps that you are following? I can try to run a docker container to reproduce the problem. There may be problems with dependencies - leveldb or boost or ...
apt install curl git make cmake clang clang-tools clang-format libclang1 libboost-all-dev wget basez libprotobuf-dev protobuf-compiler libssl-dev openssl gcc g++ python3-pip python3-dateutil cargo pkg-config libseccomp-dev libcap-dev libsecp256k1-dev libsodium-dev libgmp-dev qt5-qmake-bin qt5-qmake qttools5-dev-tools qttools5-dev qtbase5-dev-tools qtbase5-dev libqt5charts5-dev 

coin=pocketcoin mkdir -p ~/Downloads/ccwallets/${coin}/git.src cd ~/Downloads/ccwallets/${coin}/git.src

git clone https://github.com/pocketnetteam/pocketnet.core.git ./ git checkout 0.22.0

with gcc

echo "nothing needed"

or with clang

export CC=clang export CXX=clang++

sh autogen.sh

without boost

make -j4 -C depends bdb

or with boost

make -j4 -C depends bdb boost

cd "depends/built/" && cc_archdir=ls && cd ../../ tar xvzf depends/built/${cc_archdir}/bdb/*.tar.gz -C ./depends/${cc_archdir}/

without boost

echo "nothing needed"

or with boost

tar xvzf depends/built/x86_64-pc-linux-gnu/boost/*.tar.gz -C ./depends/x86_64-pc-linux-gnu/

without boost

./configure LDFLAGS="-Lpwd/depends/${cc_archdir}/lib/" CPPFLAGS="-Ipwd/depends/${cc_archdir}/include/" CXXFLAGS="-O3 -march=native" \ --disable-bench --disable-gui-tests --disable-tests --enable-reduce-exports --without-miniupnpc --without-zmq --with-gui=qt5

or without boost

./configure LDFLAGS="-Lpwd/depends/${cc_archdir}/lib/" CPPFLAGS="-Ipwd/depends/${cc_archdir}/include/" CXXFLAGS="-O3 -march=native" \ --with-boost-libdir=pwd/depends/${cc_archdir}/lib/ \ --disable-bench --disable-gui-tests --disable-tests --enable-reduce-exports --without-miniupnpc --without-zmq --with-gui=qt5

make -j4



Above are my compilation steps from scratch. It is using by pocketnet predefined depends bdb. With by pocketnet predefined boost, the compilation is failing with gcc and also failing with clang. With default debian boost the compilation works well, but with gcc build package there is weird message on runtime(mentioned in above bug report) and staking is not able to be enabled, with clang build package there is segfault on start(mentioned in above bug report).

nnmfnwl commented 9 months ago

Hi @andyoknen I will try one more test with empty blockchain files ~/.pocketcoin/ with clang and gcc, if the problem could be found as corrupted chain files, will see and let you know.

nnmfnwl commented 9 months ago

Nothing changed with gcc/clang and empty ~/.pocketcoin ( I do dexsetup development and building many chains https://github.com/nnmfnwl/dexsetup but i have no clue what to do with this pkoin.

andyoknen commented 9 months ago

Great! Thanks for the information, I'll try to reproduce the problem after the holidays. Happy New Year!

nnmfnwl commented 9 months ago

Thank you, Happy New Year also to you and all bastyon developers.

andyoknen commented 9 months ago

Hi, I just realized that this is a very strange build path. On a clean system, I was unable to build depends..

We use the build via depends, I set it up according to this instruction for each platform: https://github.com/pocketnetteam/pocketnet.core/blob/0.22/depends/README.md

For development, I use cmake or earlier https://github.com/pocketnetteam/pocketnet.core/blob/0.22/doc/build-unix.md

I am confused by this make -j4 -C depends .. and further usage.

When using depends, configuration is performed using the config.site file - I'm not sure if it works the same way if you just specify the lib and include directory during configuration.

https://github.com/pocketnetteam/pocketnet.core/blob/0.22/doc/build-windows.md

nnmfnwl commented 8 months ago

Hi, I just realized that this is a very strange build path. I can see nothing strange on that, it is just low level way to build node with maximum number of development libraries which security is managed directly by Debian developers, they take security seriously, i can not trust all the dependencies(depends) which are provided by every wallet, those can be full of bugs and rarely updated. This process works for me for years and i am working with many nodes.

On a clean system, I was unable to build depends.. Yes, because it needs to install dev dependencies on system first, like below.

apt install curl git make cmake clang clang-tools clang-format libclang1 libboost-all-dev wget basez libprotobuf-dev protobuf-compiler libssl-dev openssl gcc g++ python3-pip python3-dateutil cargo pkg-config libseccomp-dev libcap-dev libsecp256k1-dev libsodium-dev libgmp-dev qt5-qmake-bin qt5-qmake qttools5-dev-tools qttools5-dev qtbase5-dev-tools qtbase5-dev libqt5charts5-dev

Those os depends might be also depends specific only for other wallets like libqtchart for advanced gui wallets. I do also advanced security by sandboxing whole build/run process inside native kernel features with firefail and adding privacy by sandboxing by default with tor.

We use the build via depends... Yes seems like you building all depends and maybe out of date library in depends with existing bug still not rise a segfault, but maybe i am all wrong. Do you do some static analysis or valgrind?

When using depends, configuration is performed using the config.site file - I'm not sure if it works the same way if you just specify the lib and include directory during configuration. By me above mentioned steps works for me for years, but maybe for many years I am wrong.

I am confused by this make -j4 -C depends .. and further usage. Which one step you are confused by?

   # build only deprecated Berkley db from depends(i know chain files would not be universal with new version)
make -j4 -C depends bdb

   # detect architecture
cd "depends/built/" && cc_archdir=`ls` && cd ../../

   # unpack berkley db
tar xvzf depends/built/${cc_archdir}/bdb/*.tar.gz -C ./depends/${cc_archdir}/

   # configure by official help message and docs. include lib and headers for bdb, optimisation only for this machine, disable build bench, disable build tests, reduce exports for less debug, disable upnpc disable zmq, and do qt5 could be set as auto.
./configure LDFLAGS="-L`pwd`/depends/${cc_archdir}/lib/" CPPFLAGS="-I`pwd`/depends/${cc_archdir}/include/" CXXFLAGS="-O3 -march=native" \
--disable-bench --disable-gui-tests --disable-tests --enable-reduce-exports --without-miniupnpc --without-zmq --with-gui=qt5

   # finally built it.
make -j4

So you was not able to reproduce build process nor able to reproduce a bug? Could you try to build again with installed above mentioned build apt install os packages?

Could you provide me your build steps?

nnmfnwl commented 8 months ago

@Decentralizedaz You are right, i removed CXXFLAGS="-O3 -march=native" and it stops crashing, so the compiler optimization tight up the mem and bug revealed. But staking still does not working!

@andyoknen So i try also download and sync and staking with official binary from github here and it is OBVIOUSLY something wrong also with published binary files! Like: Staker worker thread started for \x00�6�\x07\x00\x00\x00��#{�fking and staking still no ability to be activated, wallet is unlocked and i also try staking=1 to add into conf. Please do something with it, this bug is repeating from the past and i am starting to suspect you of doing this on purpose to earn on Staking as unique.

andyoknen commented 8 months ago

OK, I'm glad your build stopped crashing. I have reassembled along your path and so far I am not experiencing problems with the node, it is working. Although I collected using CXXFLAGS="-O3 -march=native". There may be an optimization accident..

About stacking. The strange message is the name of your wallet. I have already encountered poor unicode support in the code - I think this is just the case. https://github.com/pocketnetteam/pocketnet.core/blob/8c25d45bd9bfa6e6ca07114a3e1df0d3740a44d3/src/staker.cpp#L113

Next, could you provide the output of the RPC commands:

pocketcoin-cli getblockchaininfo
pocketcoin-cli getstakinginfo
pocketcoin-cli getstakereport

In getblockchaininfo, the most interesting thing is the coincidence of blocks and headers. And also initialblockdownload=false?

getstakinginfo notifies about the status of the staking process

andyoknen commented 8 months ago

I collected using gcc - maybe clang is crashing because of the name in the wallet?

nnmfnwl commented 8 months ago

@andyoknen

About stacking. The strange message is the name of your wallet. I have already encountered poor unicode support in the code - I think this is just the case. Look, my wallet name is not using any special unicode characters and i do not think that \x00\x00 two literally zeroes could be any unicode char. Wallet name is wallet_pkoin_staking but we can see ....king. The CXXFLAGS="-O3 -march=native" reorganized/optimized/align memory, but without reorg for yoru case something else in memory is rewritten in process, but it is eventually not needed so much later. The bug is about something going out of range or double delete or free or whatever. The case when it is not crashing but malfunctioning is case when rw part of memory is rewritten. The case when it is crashing is the case when something try to rewrite protected part of memory. Have any pkoin developer ever run valgrind on pkoin node? I can say it is not related to QT, because even daemon qithout QT is crashing same way. Valgrind memory checker could show the bug with love.

Look here my wallet is all ok, and i synced chain many times, this time i try 3 times.

./getblockchaininfo.basics 
  "blocks": 2616315,
  "headers": 2616315,
  "bestblockhash": "9c0607cf0afd1f6d0b197e64fa0d77435524cb0f6b903d5df5bec15b9993cdb9",

Yes synced correctly with https://bastyon.com/blockexplorer/

./getstakinginfo 
{
  "enabled": true,
  "staking": false,
  "errors": "",
  "difficulty": 347886.3630944471,
  "search-interval": 0,
  "weight": 145331506010,
  "balance": 145331506010,
  "netstakeweight": 402181813841515,
  "expectedtime": 0
}

Staking is not able to activate and no errors reported...

./getwalletinfo.basic | grep unlocked
  "unlocked_until": 1807306711,

Yes wallet is unlocked

./getstakereport
{
  "2024-02-06T23:00:00Z": "0.00",
.... all zero
"2024-01-08T23:00:00Z": "0.00",
  "Last 24H": "0.00",
  "Last 7 Days": "0.00",
  "Last 30 Days": "0.00",
  "Last 365 Days": "NUMBER",
  "Latest Stake": "2.43",
  "Latest Time": "2023-12...",
  "Stake counted": NUMBER,
  "time took (ms)": 0
}

There is no stake with latest 2 wallet releases.

andyoknen commented 8 months ago

./getwalletinfo.basic | grep unlocked
  "unlocked_until": 1807306711,

Have you encrypted your wallet? and manually unlocked it?

nnmfnwl commented 8 months ago

./getwalletinfo.basic | grep unlocked
  "unlocked_until": 1807306711,
Have you encrypted your wallet? and manually unlocked it?

yes, it was encrypted and unlocked. it is standard process. it was always working. i changed version not a wallet. 1807306711 number means it is unlocked. if it would be 0, it would mean locked.

nnmfnwl commented 8 months ago

Bug disappeared when wallet name was renamed from wallet_pkoin_staking to wallet_p whaaaaat? Now it seems like we know how to reproduce a bug, maybe variable holding copy name of wallet or path is not enough sized?

nnmfnwl commented 8 months ago

Now i will try clang with o3 native to see if would be working or it is something deeper.

andyoknen commented 8 months ago

I renamed the wallet in your long name - I got a broken encoding in the log, and also this condition does not trigger a staking thread.

https://vscode.dev/github/pocketnetteam/pocketnet.core/blob/0.22/src/staker.cpp#L83

nnmfnwl commented 8 months ago

I renamed the wallet in your long name - I got a broken encoding in the log, and also this condition does not trigger a staking thread.

https://vscode.dev/github/pocketnetteam/pocketnet.core/blob/0.22/src/staker.cpp#L83

Thank you, I will try now to build with gcc/clang non/o3native at https://github.com/pocketnetteam/pocketnet.core/commits/fix/stake_worker_wallet_access/ 1530f7e3cfdcde83790284d0cadf1111ad143a63 and will let know you know asa i know the results.

nnmfnwl commented 8 months ago

For now, clang, CXXFLAGS="-O3 -march=native", wallet_pkoin_staking, seems everything works, but maybe we need to wait for first stake to be confirmed to be sure it all works well.

andyoknen commented 8 months ago

Great! I still have some doubts about the encrypted wallet - whether stacking will work normally in this case.

nnmfnwl commented 7 months ago

Great! I still have some doubts about the encrypted wallet - whether stacking will work normally in this case.

What could goes wrong when "Encrypted and password unlocked wallet" is staking? Doesn't it behave same like not encrypted wallet?

andyoknen commented 7 months ago

What could goes wrong when "Encrypted and password unlocked wallet" is staking? Doesn't it behave same like not encrypted wallet?

I haven't had time to clarify the difference in implementation yet. Did you get stake?

nnmfnwl commented 7 months ago

What could goes wrong when "Encrypted and password unlocked wallet" is staking? Doesn't it behave same like not encrypted wallet?

I haven't had time to clarify the difference in implementation yet. Did you get stake?

On pkoin node version 19 I got stake multiple times per day, with same encrypted and unlocked wallet. I still get no stake, still waiting, it is so weird, previously i get stake multiple times per day.

nnmfnwl commented 7 months ago

By the way, wallet is checking new version every minute, it would be great to reconfigure it once per day or hour maybe or have it in conf file 0-disabled X-minutes/seconds whatever.

andyoknen commented 7 months ago

On pkoin node version 19 I got stake multiple times per day, with same encrypted and unlocked wallet. I still get no stake, still waiting, it is so weird, previously i get stake multiple times per day.

getstakereport and getstakinginfo normal?

andyoknen commented 7 months ago

By the way, wallet is checking new version every minute, it would be great to reconfigure it once per day or hour maybe or have it in conf file 0-disabled X-minutes/seconds whatever.

Checking the new version? A new version of the software happens every hour https://github.com/pocketnetteam/pocketnet.core/blob/d9b7ffade67cac59498a9c31502f00e0fcc64b69/src/qt/pocketcoin.cpp#L498-L500

nnmfnwl commented 7 months ago

On pkoin node version 19 I got stake multiple times per day, with same encrypted and unlocked wallet. I still get no stake, still waiting, it is so weird, previously i get stake multiple times per day.

getstakereport and getstakinginfo normal?

Looks ok, correct chain, staking, no errors.

./getstakinginfo

  "enabled": true,
  "staking": true,
  "errors": "",

./getblockchaininfo.basics

  "blocks": 2627760,
  "headers": 2627760,
  "bestblockhash": "72bf295d575eb8a4ffe9961e69013cc8d69025a30451d787a63b840e1663e2b7",

nnmfnwl commented 7 months ago

By the way, wallet is checking new version every minute, it would be great to reconfigure it once per day or hour maybe or have it in conf file 0-disabled X-minutes/seconds whatever.

Checking the new version? A new version of the software happens every hour

https://github.com/pocketnetteam/pocketnet.core/blob/d9b7ffade67cac59498a9c31502f00e0fcc64b69/src/qt/pocketcoin.cpp#L498-L500

I am so sorry, yes, you right, it is every hour. Anyway it looks like a spam every time i open console, disabling case when no new version is found would make console more clean. Showing error case and case when new version is found would looks much better. "No new version found" maybe just for debug purposes.

andyoknen commented 7 months ago

I am so sorry, yes, you right, it is every hour. Anyway it looks like a spam every time i open console, disabling case when no new version is found would make console more clean. Showing error case and case when new version is found would looks much better. "No new version found" maybe just for debug purposes.

I agree, for the purity of the output, it makes sense to leave only errors and the availability of a new version. I also think it makes sense to transfer this functionality to the pocketcoind daemon so that users of the console version or docker are also aware of the release of the new version.

andyoknen commented 7 months ago

Looks ok, correct chain, staking, no errors.

You see messages in the log like: Available coins count X BestHeader: ... ?

nnmfnwl commented 7 months ago

Looks ok, correct chain, staking, no errors.

You see messages in the log like: Available coins count X BestHeader: ... ?

No i don't see. I am using -printtoconsole -nodebuglogfile

Finally i got first stake after days. So different in comparison how it been staking previously. === Staking : new PoS block found hash:

andyoknen commented 7 months ago

To analyze the number of stakes, we can chat in Bastion's personal chat - you can find me by nickname aok

pocketnetteam / pocketnet.core

0.22.0 compiled by clang segmentation fault, by gcc corrupted messages and not staking. #641

with gcc

or with clang

without boost

or with boost

without boost

or with boost

without boost

or without boost