Update bitcoin code

[lostystyg]

Notes

Commits:

• It was decided to use 0.21 version of bitcoin to update from, because it is the most last stable version. There is already 22 version but is is too new. So target commit is af591f2068d0363c92d9756ca39c43db85e5804c
• Also after a bit research it was identified that the base commit of bitcoin from that our project was checkouted is 3832c25f176753b7ddb724c26ee7543fb1e1819e

Workflow:

• Because pocketcoin project was not literally git checkouted and has no git tree roots with bitcoin and there are more than 10000 commits between base and target commits it was decided to port changes file by file.
• The following concept was used to update pocketcoin codebase:
  • All actions below are performed to a single file.
  • Bitcoin codebase at base commit (3832c25f176753b7ddb724c26ee7543fb1e1819e) is SOT.
  • All files should have same preformatting to show only real differences
  • diff between current pocketcoin version and base bitcoin are pocketcoin additions.
  • diff between target bitcoin and base bitcoin are bitcoin updates
  • pocketcoin additions and bitcoin updates are manually merged to bitcoin base
  • bitcoin base becomes result
• This concept allows us to determine real pocketcoin and bitcoin changes from point where pocketcoin was checkouted and make an opinion about how to manually merge them.
• Still there are some changes e.x. moving functional from one file to another, it was needed to make diffs of all such related files and perform a complex merge keeping in mind all these changes.
• File by file porting results in non-buildable intermediate result.

Progress:

• Ported:
  • src/*
  • test/*
  • autotools build system
• Not ported:
  • Other non-code stuff, like contrib/*, docs, share/*, README.md, CI stuff, etc.

Currently there are critical issues (generally could be found by searching "losty-critical" in codebase) that should be resolved before testing stage:

• [x] (Critical) GetTransaction usage in pos.cpp should be changed to use Get method from pocketdb.
• [x] (Critical) Staker::worker - rework logic around removed CReverseScript (see comments in code for details).
• [x] (Critical) Staker::signBlock - rework logic around SigningProvider. Previously it was a CWallet class, now it is not. It seems like FillableSigningProvider should be used but need a research. See at the end of DescriptorScriptPubKeyMan::FillPSBT(..) method
• [x] (Critical) Logging in pocketdb. RPC_POCKETTX_MATURITY should go to TxValidationResult and state.GetRejectCode() should be changed to state.GetResult()
• [x] (Critical) ConsensusFailed() method of ValidationState and its usage should be reworked because there is no longer DoS() method used by it. Also is uses SocialConsensusResult codes that are required for webui, but Invalid() method is no longer accepts pure codes. Probably need to inject SocialConsensusResult codes to TxValidationResult
• [x] (Critical) ArgsManager::AddArg() is accepting now usage flags instead of boolean for debug mode. Need to update this behavior for remained pocketnet specific args (currently still boolean).
• [x] (Critical) CMake for QT. Qt gui was updated so need to update cmake too.
• [x] (Critical) CMake validation. Currently cmake seems to work but need to check nothing was forgotten.

Testing stage issues:

• [x] (Critical) Common logging issues. Logging in a lot of places was reworked. Especially removed DoS method of ValidationState. In some such places (can be searched by "losty-fur") logging messages could be changed. Update! : this is marked as critical because there is a new ban logic based on type of error of ValidationState. During porting most of pocketnet errors where reworked to cause a CONSENSUS error witch will result in banning nodes. All such errors could be found by "losty-fur" and should be inspected and CONSENSUS error should probably be changed to something less critical to protect nodes from unnecessary banning. Update2!: This was validated and fixed.
• [ ] (Critical) Chainparams issues. Deployment parameters for networks should be discovered because logic there was changed a bit. Update!: seems there are no errors for test and main nets but still need to pay additional attention on this
• [ ] (Ctitical) Boost content transaction generation. Boosting content transactions can overflow max raw tx fee and it is legal. However, logic around this in updated bitcoin code is a lot reworked. Need some workaround to allow boosting content without max tx fee restriction.
• [x] (Critical) SignTransaction method changed a lot and requires vector of coins. Validate usage ecpecially in PocketTranscationRpc.cpp
• [x] (Critical) Frontend: websocket and rpc
• [x] (Critical) Staking

Other issues:

• [x] Not shutting down after sigterm signal. The reason is staking and scheduler threads are not going to stop even after shutdown requested.
• [ ] RPC methods description. A lot of pocketcoin rpc methods were missing description. A "losty-rpc" todo was placed in such places
• [ ] pocketcoin-wallet executable removed from autotools because it caused a lot of linking errors due to stake methods require server lib. Figure out how to fix and add this executable to cmake.
• [x] (Critical) tests. Currently lots of new tests fail or even segfault because of bitcoin's far-fetched assumtions of some variables without assertions. E.x. here there is nullptr returned by active[20] that causes a segfault
• [ ] CMake common rpc library. There are POCKETCOIN_COMMON_RPC lib declared in CMake that is used by both server and cli but currently it depends on pocketdb headers that are part of server. It is ok for now because it is resolved internally during build because of public common headers but this should be reworked.
• [ ] CMake libmultiprocess. Autotools are using libmultiprocess but cmake not. Need a research about it.

[tawmaz]

One request I would have is to port the refactored utility functions from bitcoin in src/util (https://github.com/bitcoin/bitcoin/tree/0.21/src/util). These often present conflicts when trying to port over or compare code from Bitcoin and they are typically are OS helper functions which do not need to be modified for Pocketcoin. There has been a lot of refactoring to clean up these functions over what we have now in util.h/util.cpp. Perhaps this part could be split out to it's own task.

[lostystyg]

@tawmaz all source code has been already ported, check #156

[lostystyg]

It seems like node is performing very good on testnet. Staking, new logic of accessing transaction from sqlite, validating blocks and more is working without errors. Next stage is trying to use frontend on updated node.

[lostystyg]

Node is fully usable with frontend on testnet. This had required to fix some rpc methods (typo in name or something like that). I think there will be further issues with rpc but they are all one-second-fixable and they only need to be noticed to fix them. I have started sync in main net with blocksonly=1 and listen=0, will update this issue with results.

[andyoknen]

I can ask our testers to check the functionality of the node from different sides - can you ensure its operation 24/7 or do you need to allocate one of the test servers in the cloud?

[lostystyg]

I can ask our testers to check the functionality of the node from different sides

I think it is not necessary to do this right now because we are targeting 0.21 only after 0.20.19 and there will be a lot other things for testers to check instead of spending time on this "pre-alpha". Also there are still things that i can test myself and i can do this with my personal node without cloud even that i can't run it 24/7. I suggest involve testers afterwards when we will have time for this and i would have already performed basic testing. However, it is good idea and if it won't distract anyone from other useful tasks, i would prefer cloud server.

[andyoknen]

I support it, we will start testing later

[SSLvzaimodaystvie1]

Извините по ходу мне, здесь не комфортно ключевое слово, вчерашней проблемы. Заблудился хочу выполнить требование обновиться, а где это чудесное для "чайников" место, не ужели на газовой плите!? И нам опять к Адольфу на поклон, что бы он о бабушке своей, не так сильно переживал, в среде "чайников" в обновлениях открытых дорог, в светлое будущие приятного и комфортного общения Яндекс с каналом Кью, передразнивает "Кинд дзадзу" с приветствием "КУ". Простите великодушно, за отвлечение ВАС от серьёзной задачи, в процесс разрешения которой я так нелепо попал. Где выход к обновлению, сегодня последний день или меня в очередной раз ввели в блуд!?

[lostystyg]

The update has been finished. See 0.21 branch history for details.