karmacoma
commented
8 years ago @karek314: I will give my explanation on why this fork occurs.
This type of fork is caused by a mismatch between the expected public key, and the received block's generator public key, when processing a block.
The following gives an example of the error message which will be recorded when a node enters this type of fork:
Note: In 0.2.1, we improved the logging in this case to give more detailed information, and within this explanation I have added comments to the code.
Error message
{
"level":"error",
"message":"Expected generator: 7ac9d4b708fb19eaa200eb883be56601ddceed96290a3a033114750b7fda9d0b Received generator: fa7bfd3a2dc0ca55b700247aae4694709d6cdfa34c6bfb0237e032d7aae404f0",
"timestamp":"2016-05-07 03:40:44"
}Fork log
{
"level":"info",
"message": "Fork",
"timestamp":"2016-05-07 03:40:44",
"data": {
"delegate":"fa7bfd3a2dc0ca55b700247aae4694709d6cdfa34c6bfb0237e032d7aae404f0",
"block":{
"id":"13609465205005996164",
"timestamp":34054830,
"height":61118,
"previousBlock":
"1280968842307106697"
},
"cause":3
}In this example the node has received the following public key: fa7bfd3a2dc0ca55b700247aae4694709d6cdfa34c6bfb0237e032d7aae404f0
The expected public key is determined, like so:
1. Lisk gets a list of all 101 delegates sorted by vote
private.getKeysSortByVote = function (cb) {
modules.accounts.getAccounts({
isDelegate: 1,
sort: { "vote": -1, "publicKey": 1 },
limit: slots.delegates
}, ["publicKey"], function (err, rows) {
if (err) {
cb(err)
}
cb(null, rows.map(function (el) {
return el.publicKey
}))
});
}2. Lisk determines an order of delegates based on an initial seed
- The initial seed is a sha256 hash of the current round, derived from block height.
- Delegates are indexed according to currentSeed[x] % 101, with x being an individual byte of said sha256 hash.
- For every 4th byte of the current seed, a new hash is created based on the previous one.
Delegates.prototype.generateDelegateList = function (height, cb) {
private.getKeysSortByVote(function (err, truncDelegateList) {
if (err) {
return cb(err);
}
var seedSource = modules.round.calc(height).toString(); // Establish current round as the seed source
var currentSeed = crypto.createHash('sha256').update(seedSource, 'utf8').digest(); // Create first sha256 hash seed from current round
// Iterate of 101 delegates list sorted by vote
for (var i = 0, delCount = truncDelegateList.length; i < delCount; i++) {
// Iterate over each byte in current hash (up to 4th byte)
for (var x = 0; x < 4 && i < delCount; i++, x++) {
var newIndex = currentSeed[x] % delCount; // Determine new index
var b = truncDelegateList[newIndex]; // Get current delegate at new index
// Swap delegate stack positions
truncDelegateList[newIndex] = truncDelegateList[i];
truncDelegateList[i] = b;
}
// Generate a new hash seed based on current seed
currentSeed = crypto.createHash('sha256').update(currentSeed).digest();
}
cb(null, truncDelegateList);
});
}3. Lisk determines a slot number according to the block timestamp, and selects the expected public key from the previously generated delegates list
Delegates.prototype.validateBlockSlot = function (block, cb) {
// Generate delegate list based on current block height (round)
self.generateDelegateList(block.height, function (err, activeDelegates) {
if (err) {
return cb(err);
}
// Get slot number according to given block timestamp
var currentSlot = slots.getSlotNumber(block.timestamp); // block.timestamp / 10 block time = 3405483
var delegate_id = activeDelegates[currentSlot % slots.delegates]; // delegate_id = 3405483 % 101 delegates = 66
// Expected: 7ac9d4b708fb19eaa200eb883be56601ddceed96290a3a033114750b7fda9d0b (66)
// Received: fa7bfd3a2dc0ca55b700247aae4694709d6cdfa34c6bfb0237e032d7aae404f0
// Compare public keys
if (delegate_id && block.generatorPublicKey == delegate_id) {
return cb(); // Keys match
} else {
// Keys do not match
library.logger.error("Expected generator: " + delegate_id + " Received generator: " + block.generatorPublicKey);
return cb("Failed to verify slot: " + currentSlot);
}
});
}If the received and expected public keys do not match. The node enters a fork, named cause 3.
Possible Causes
- The block signer's system clock was adrift, producing an invalid block timestamp, and therefore a different delegate slot number is used to that which is later determined when validating the delegate slot.
- The initial delegates list used to validate a delegate slot is sorted by vote. If the votes are recorded differently on an individual node (possibly as a result of missed blocks), this will affect the order of the delegate list used to select delegates according to the given slot number.
I hope this is a full enough explanation on why this fork is occurring. I am currently working on a solution, but posting this explanation to open the discussion and improve understanding.
fix
2mdoty
Isabello
mrv777
jfeldis
ByronAP
qtipwax
ghost
lichuan
It has been fixed with bash lisk.sh rebuild but lisk couldn't automatically fix db. I guess issue happened during connectivity problems, i have reported similar issue with previous database engine.