update codebase and get it up to date with Rivine

GlenDC commented 4 years ago

update all vendored dependencies (dep)
get codebase up to date with Rivine's new codebase
move threebot code to extension internal package
remove most internal erc20 logic and use the new rivine erc20 extension
adapt and fix where needed
improve linting, formatting and testing
integrate latest rivinecg faucet (closes #369)
ensure explorer frontend JS code is up to date with latest bug fixes and features (closes #368)

Closes #363, closes #362, closes #354, closes #357, closes #351 (disabled for standard net) and also closes #350 (disabled for standard net).

GlenDC commented 4 years ago

Currently when starting a daemon using this version you'll see the following error:

$ tfchaind -Mtbewcg --network testnet -v           
Loading...
Binding API Address and serving the API...
Loading gateway (0/6)...
Loading consensus set (1/6)...
Closing consensus set...
Context is done, quitting...
context is done, quitting...
Closing gateway...
daemon failed failed to register the threebot extension: tx not writable

GlenDC commented 4 years ago

Currently when starting a daemon using this version you'll see the following error:

$ tfchaind -Mtbewcg --network testnet -v           
Loading...
Binding API Address and serving the API...
Loading gateway (0/6)...
Loading consensus set (1/6)...
Closing consensus set...
Context is done, quitting...
context is done, quitting...
Closing gateway...
daemon failed failed to register the threebot extension: tx not writable

Fixed in Rivine vendored dep code in commit a2ac052c118db11a1128e22a83e02248ae32306f

LeeSmet commented 4 years ago

syncing testnet, synced untill block 94130. then this happened:

goroutine 58 [running]:
runtime/debug.Stack(0x4664d7, 0x0, 0xc00018cd98)
        /usr/lib/go/src/runtime/debug/stack.go:24 +0x9d
runtime/debug.PrintStack()
        /usr/lib/go/src/runtime/debug/stack.go:16 +0x22
github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/build.Critical(0xc00018cec8, 0x1, 0x1)
        /home/lee/go/src/github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/build/critical.go:15 +0xaa
github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/pkg/encoding/siabin.(*Decoder).decode(0xc00040fc40, 0x1095040, 0xc0007e8790, 0x199)
        /home/lee/go/src/github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/pkg/encoding/siabin/marshal.go:273 +0xd1f
github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/pkg/encoding/siabin.(*Decoder).decode(0xc00040fc40, 0xf070a0, 0xc00011ee00, 0x197)
        /home/lee/go/src/github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/pkg/encoding/siabin/marshal.go:334 +0x519
github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/pkg/encoding/siabin.(*Decoder).Decode(0xc00040fc40, 0xedd8c0, 0xc00011ee00, 0x0, 0x0)
        /home/lee/go/src/github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/pkg/encoding/siabin/marshal.go:230 +0x1d8
github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/pkg/encoding/siabin.Unmarshal(0xc000a0d300, 0x1172, 0x1172, 0xedd8c0, 0xc00011ee00, 0x1172, 0x0)
        /home/lee/go/src/github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/pkg/encoding/siabin/marshal.go:357 +0xb8
github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/pkg/encoding/siabin.ReadObject(0x7f10e4189130, 0xc00041a640, 0xedd8c0, 0xc00011ee00, 0x1312d00, 0x0, 0x1c16900)
        /home/lee/go/src/github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/pkg/encoding/siabin/prefix.go:32 +0x9f
github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/consensus.(*ConsensusSet).managedReceiveBlocks(0xc0007e8000, 0x134a540, 0xc00041a640, 0x0, 0x0)
        /home/lee/go/src/github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/consensus/synchronize.go:200 +0x40c
github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/gateway.(*Gateway).managedRPC(0xc00021a700, 0xc00003bf00, 0x13, 0x10ee145, 0xa, 0xc0004664f0, 0x0, 0x0)
        /home/lee/go/src/github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/gateway/rpc.go:66 +0x2bf
github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/gateway.(*Gateway).RPC(0xc00021a700, 0xc00003bf00, 0x13, 0x10ee145, 0xa, 0xc0004664f0, 0x0, 0x0)
        /home/lee/go/src/github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/gateway/rpc.go:76 +0xf2
github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/consensus.(*ConsensusSet).threadedInitialBlockchainDownload.func2(0xc0007e8000, 0xc00018def0, 0xc00018de48, 0xc00018de98, 0xc00018de60, 0xc00018deb8, 0xc00018de50, 0x0, 0x0)
        /home/lee/go/src/github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/consensus/synchronize.go:531 +0x126
github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/consensus.(*ConsensusSet).threadedInitialBlockchainDownload(0xc0007e8000, 0xc0002ce7b8, 0x1)
        /home/lee/go/src/github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/consensus/synchronize.go:561 +0x485
github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/consensus.(*ConsensusSet).Start.func1(0xc0007e8000)
        /home/lee/go/src/github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/consensus/consensusset.go:249 +0x480
created by github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/consensus.(*ConsensusSet).Start
        /home/lee/go/src/github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/consensus/consensusset.go:241 +0x3f
Critical error: could not decode type []types.MinerPayout: Critical error: length 8712303899067005499 exceeds maxLen of 256
Please submit a bug report here: https://github.com/threefoldtech/rivine/issues

GlenDC commented 4 years ago

syncing testnet, synced untill block 94130. then this happened: ...

I did not give the required opt-in options when creating the minting Rivine plugin on the daemon sides (tfchaind and bridged). Miner fees are required and sia encoding needs to be used in tfchain, both are opt-in. Commit fcf09e62e3ef356e7254b3e4f052fb9ebf85df02 passes these options correctly when creating the Minting plugin.

LeeSmet commented 4 years ago

Still same issue after updating however when syncing from scratch now, block 1 fails with following cs logs:

2019/09/26 20:12:28.608106 block_validation.go:75: [SBV] Validating new block for height 1
2019/09/26 20:12:28.615529 diffs.go:188: WARN: block 0624c4830353c83e75683ce47683d0acc11216e528cd46c87350c2516a24d743 cannot be applied: tx 251eeeec00528899ac61d6662b32a2b43502e19a8c64a87ac258ee01381f2112 is invalid: all coin outputs (minus miner fees) (0) are refunded for tx 251eeeec00528899ac61d6662b32a2b43502e19a8c64a87ac258ee01381f2112, this is not allowed for a coin destruction transaction

GlenDC commented 4 years ago

Still same issue after updating however when syncing from scratch now, block 1 fails with following cs logs: ...

Both problems should be resolved by c5fa07c:

Rivine minting extension did not correctly register minting transactions, as it did not pass on the opt-in legacy options to the registered minting txs when defined
CoinDestruction opt-in minting tx was enabled as soon as options was defined, did not check that it wasn't 0

LeeSmet commented 4 years ago

After the latest change, the daemons: now sync again from block 0, and manage to sync block 94130. The bad news is it now stops at block 94136, dumping the following stack trace:

goroutine 68 [running]:                                                                                                                                                                                                                [4/1065]
runtime/debug.Stack(0x4664d7, 0x0, 0xc000937408)                                                                                                                                                                                               
        /usr/lib/go/src/runtime/debug/stack.go:24 +0x9d                                                                                                                                                                                        
runtime/debug.PrintStack()                                                                                                                                                                                                                     
        /usr/lib/go/src/runtime/debug/stack.go:16 +0x22                                                                                                                                                                                        
github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/build.Critical(0xc000937538, 0x1, 0x1)
        /home/lee/go/src/github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/build/critical.go:15 +0xaa
github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/pkg/encoding/siabin.(*Decoder).decode(0xc001034400, 0x1061100, 0xc001024340, 0x199)
        /home/lee/go/src/github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/pkg/encoding/siabin/marshal.go:273 +0xd1f
github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/pkg/encoding/siabin.(*Decoder).Decode(0xc001034400, 0x105c000, 0xc001024340, 0x0, 0x0)
        /home/lee/go/src/github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/pkg/encoding/siabin/marshal.go:230 +0x1d8
github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/pkg/encoding/siabin.Unmarshal(0x7f7384004066, 0x6e, 0x6e, 0x105c000, 0xc001024340, 0xa6, 0x7f738400402e)
        /home/lee/go/src/github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/pkg/encoding/siabin/marshal.go:357 +0xb8
github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/extensions/minting.(*Plugin).getMintConditionFromBucketAt(0xc0001132c0, 0xc000fb95c0, 0x16fb9, 0xe, 0xc000fb95c0, 0x0, 0x0)
        /home/lee/go/src/github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/extensions/minting/minting.go:306 +0xc3
github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/extensions/minting.(*Plugin).getMintConditionFromBucketWithContextInfo(0xc0001132c0, 0xc000fb95c0, 0x1, 0x16fb9, 0xc000fb95c0, 0x0, 0x0, 0x0)
        /home/lee/go/src/github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/extensions/minting/minting.go:329 +0x50
github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/extensions/minting.(*Plugin).validateCoinCreationTx(0xc0001132c0, 0x81, 0x0, 0x0, 0x0, 0xc000515230, 0x1, 0x1, 0x0, 0x0, ...)
        /home/lee/go/src/github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/extensions/minting/minting.go:429 +0x1cb
github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/consensus.(*ConsensusSet).validateTransactionUsingPlugins(0xc00054ed80, 0x81, 0x0, 0x0, 0x0, 0xc000515230, 0x1, 0x1, 0x0, 0x0, ...)
        /home/lee/go/src/github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/consensus/plugin.go:381 +0x26e
github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/consensus.(*ConsensusSet).validTransaction(0xc00054ed80, 0xc000fe7a40, 0x81, 0x0, 0x0, 0x0, 0xc000515230, 0x1, 0x1, 0x0, ...)
        /home/lee/go/src/github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/consensus/validtransaction.go:49 +0x33a
github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/consensus.(*ConsensusSet).generateAndApplyDiff(0xc00054ed80, 0xc000fe7a40, 0xc000fea280, 0x2, 0x2)
        /home/lee/go/src/github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/consensus/diffs.go:182 +0x634
github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/consensus.(*ConsensusSet).applyUntilBlock(0xc00054ed80, 0xc000fe7a40, 0xc000fea280, 0x0, 0x0, 0x0, 0xc0009e17c4, 0xc0009e1948)
        /home/lee/go/src/github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/consensus/fork.go:90 +0x272
github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/consensus.(*ConsensusSet).forkBlockchain(0xc00054ed80, 0xc000fe7a40, 0xc000fea280, 0xc0009e1948, 0x5b76bb24, 0x16fb6, 0x0, 0x0, 0xc0007f8120, 0x2, ...)
        /home/lee/go/src/github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/consensus/fork.go:189 +0xac
github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/consensus.(*ConsensusSet).addBlockToTree.func1(0xc000fe7a40, 0xc000938d01, 0xc000fe7a40)
        /home/lee/go/src/github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/consensus/accept.go:156 +0x323
github.com/threefoldfoundation/tfchain/vendor/github.com/rivine/bbolt.(*DB).Update(0xc00052ac00, 0xc000938e80, 0x0, 0x0)
        /home/lee/go/src/github.com/threefoldfoundation/tfchain/vendor/github.com/rivine/bbolt/db.go:677 +0xa8
github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/consensus.(*ConsensusSet).addBlockToTree(0xc00054ed80, 0x5ad4b6eb9a7ce35a, 0x1a6bb3d008978ecd, 0x1aa5d35fc483181a, 0x40b6a5b5af2f2f64, 0x5b76bb24, 0x16f$
6, 0x0, 0x0, 0xc0007f8120, ...)                            
        /home/lee/go/src/github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/consensus/accept.go:137 +0xdc
github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/consensus.(*ConsensusSet).managedAcceptBlock(0xc00054ed80, 0x5ad4b6eb9a7ce35a, 0x1a6bb3d008978ecd, 0x1aa5d35fc483181a, 0x40b6a5b5af2f2f64, 0x5b76bb24, 0$
16fb6, 0x0, 0x0, 0xc0007f8120, ...)                        
        /home/lee/go/src/github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/consensus/accept.go:293 +0x124
github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/consensus.(*ConsensusSet).managedReceiveBlocks(0xc00054ed80, 0x134a540, 0xc00086cde0, 0x0, 0x0)
        /home/lee/go/src/github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/consensus/synchronize.go:212 +0x556
github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/gateway.(*Gateway).managedRPC(0xc000532e00, 0xc00003bac0, 0x13, 0x10ee145, 0xa, 0xc00079c730, 0x0, 0x0)
        /home/lee/go/src/github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/gateway/rpc.go:66 +0x2bf
github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/gateway.(*Gateway).RPC(0xc000532e00, 0xc00003bac0, 0x13, 0x10ee145, 0xa, 0xc00079c730, 0x0, 0x0)
        /home/lee/go/src/github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/gateway/rpc.go:76 +0xf2
github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/consensus.(*ConsensusSet).threadedInitialBlockchainDownload.func2(0xc00054ed80, 0xc000939ef0, 0xc000939e48, 0xc000939e98, 0xc000939e60, 0xc000939eb8, 0x$
000939e50, 0x0, 0x0)                                       
        /home/lee/go/src/github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/consensus/synchronize.go:531 +0x126
github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/consensus.(*ConsensusSet).threadedInitialBlockchainDownload(0xc00054ed80, 0xc00056c7b8, 0x1)
        /home/lee/go/src/github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/consensus/synchronize.go:561 +0x485
github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/consensus.(*ConsensusSet).Start.func1(0xc00054ed80)
        /home/lee/go/src/github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/consensus/consensusset.go:249 +0x480
created by github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/consensus.(*ConsensusSet).Start
        /home/lee/go/src/github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/modules/consensus/consensusset.go:241 +0x3f
Critical error: could not decode type []uint8: Critical error: unexpected EOF

GlenDC commented 4 years ago

Your error indicates that your consensus plugin DB is corrupt. This is expected though as you suffered from that precious bug. As a result your DB will be encoded using rivbin encoding, while really it now correctly expects siabin encoding for the minting consensus db data. Wiping your DB will resolve it and correctly encode it next time.

GlenDC commented 4 years ago

Ok good news is that it seems to sync fine now on standard and testnet networks. @LeeSmet told me he's syncing up several daemons over night, so that will be a first good test and hopefully continue to give us good news.

LeeSmet commented 4 years ago

Coming back this morning, found both testnet daemons stopped after syncing up until block 246897. Following is the last log from the cs log:

2019/09/26 23:43:27.565754 block_validation.go:75: [SBV] Validating new block for height 246898
2019/09/26 23:43:27.568103 diffs.go:188: WARN: block ee04e3d3903f4a21c149bd54c04cc70a7892ead481e0c26ec9672bf572507fc8 cannot be applied: tx 991b8f861ea5c0e4f2c3cfc1fea0707a998256ef8c5502bb610a18cf0b1cf5b8 is invalid: bot cannot be updated: areBotNamesAvailable: bot name is already registered

Std net daemon synced until block 277851. Following is the last log from the cs log:

2019/09/27 00:01:52.336387 block_validation.go:75: [SBV] Validating new block for height 277852
2019/09/27 00:01:52.337789 diffs.go:188: WARN: block 01c243ea7d996f898d2e01d4f1417fe2751c340be33f3a7112d720e7ac633c58 cannot be applied: tx 4fb1c2ffc57abb7e6b29f778b5deb8b59418005d6ee526c4d15b394e3c66b1fb is invalid: tx 4fb1c2ffc57abb7e6b29f778b5deb8b59418005d6ee526c4d15b394e3c66b1fb does not contain any miner fees while at least one was expected

No output on the stdout or stderr of the daemon

GlenDC commented 4 years ago

All problems should be resolved, comments and bugs both included. Work added as commit 31e4c20ea3aa3cb774798bac6dac7736c5e7432f. @LeeSmet can you please continue your daemons on testnet with a latest built?

GlenDC commented 4 years ago

Closes #357.

GlenDC commented 4 years ago

This PR seems bug free and is feature complete. Let's do some end-to-end tests on Monday @LeeSmet and @DylanVerstraete. If all goes well then, we should be able to merge and deploy on testnet that day or the day after.

LeeSmet commented 4 years ago

We only tested syncing from scratch, I'd like to test upgrading existing nodes as well, that is upgrading from 1.1.1 production nodes and 1.1.2-rc1 (:thinking:) testnet nodes.

LeeSmet commented 4 years ago

Got this after letting a daemon with explorer run for a couple of days:

Finished loading in 0.036480003 seconds
2019/09/28 16:09:02 http: superfluous response.WriteHeader call from github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/pkg/api.WriteJSON (http.go:121)
2019/09/29 08:36:48 http: superfluous response.WriteHeader call from github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/pkg/api.WriteJSON (http.go:121)
2019/09/29 08:40:51 http: superfluous response.WriteHeader call from github.com/threefoldfoundation/tfchain/vendor/github.com/threefoldtech/rivine/pkg/api.WriteJSON (http.go:121)

GlenDC commented 4 years ago

Got this after letting a daemon with explorer run for a couple of days: ...

Hard to debug without more info (e.g. a stacktrace)

DylanVerstraete commented 4 years ago

Coin and blockstake transactions work.

LeeSmet commented 4 years ago

Got this after letting a daemon with explorer run for a couple of days: ...

Hard to debug without more info (e.g. a stacktrace)

Sadly that is the only output I have

GlenDC commented 4 years ago

I've confirmed that threebot transactions also seem to behave fine on the devnet running this code.

@LeeSmet and @DylanVerstraete please take over to get this PR merged ASAP in master, so we can start preparing a v1.2 release. Prior to the release @LeeSmet can start redeploying some existing testnet nodes as a test.

LeeSmet commented 4 years ago

I've set up a rinkeby node, and enabled light servers to connect to it. However it seems the bridge always crashes due to a panic in the geth code it uses, so until said bug is resolved, erc20 functionality can not be verified

DylanVerstraete commented 4 years ago

Tested following cases with cli

Transactions (coins/blockstakes)
Transaction without minerfees
Transaction with invalid minerfees and coinoutputs
Minting transaction
Minter definition transaction

GlenDC commented 4 years ago

Codebase in this PR is also up to date with latest latest Rivine.

Please merge and use in the testnet field ASAP @LeeSmet. Unless you can't sign off for some reason. And if so, place another review with the comments.

threefoldfoundation / tfchain

update codebase and get it up to date with Rivine #367