daemon blocks do not form a chain

[cipherzzz]

Hello, I am deserializing the Decred transactions, but keep getting a loop of

WARNING:BlockProcessor:daemon blocks do not form a chain; resetting the prefetcher
INFO:Prefetcher:catching up to daemon height 247,926 (247,927 blocks behind)
WARNING:BlockProcessor:ignoring 500 blocks starting height 500, expected 0
WARNING:BlockProcessor:ignoring 500 blocks starting height 1,000, expected 0
WARNING:BlockProcessor:ignoring 500 blocks starting height 1,500, expected 0
WARNING:BlockProcessor:ignoring 500 blocks starting height 2,000, expected 0
WARNING:BlockProcessor:ignoring 500 blocks starting height 2,500, expected 0
INFO:Prefetcher:verified genesis block with hash 4261602a9d07d80ad47621a64ba6a07754902e496777edc4ff581946bd7bc29c
WARNING:BlockProcessor:daemon blocks do not form a chain; resetting the prefetcher```

I am not sure what is causing the above error. Here is my code - any help would be appreciated:

coins.py

class Decred(Coin):
    NAME = "Decred"
    SHORTNAME = "DCR"
    NET = "mainnet"
    BASIC_HEADER_SIZE = 180
    DESERIALIZER = lib_tx.DeserializerDecred
    TX_COUNT = 217380620
    TX_COUNT_HEIGHT = 464000
    TX_PER_BLOCK = 1800
    RPC_PORT = 9109
    XPUB_VERBYTES = bytes('dpub', 'utf-8')
    XPRV_VERBYTES = bytes('dprv', 'utf-8')
    P2PKH_VERBYTE = bytes('Ds', 'utf-8')
    P2SH_VERBYTES = bytes('Dc', 'utf-8')
    WIF_BYTE = bytes('Pm', 'utf-8')

    @classmethod
    def block_header(cls, block, height):
        '''Return the block header bytes'''
        deserializer = cls.DESERIALIZER(block)
        return deserializer.read_header(height, cls.BASIC_HEADER_SIZE)

class DecredTestnet(Decred):
    SHORTNAME = "XDCR"
    NET = "testnet"
    TX_COUNT = 3061451
    TX_COUNT_HEIGHT = 245079
    TX_PER_BLOCK = 1800  
    RPC_PORT = 19119
    GENESIS_HASH = ('9ff151e3a9f6aac1879f09e3715ec5f2f7a2559840471f8e4a0896f9de33217e')
    #No Peers yet - haven't set anything up in prod yet

tx.py

class TxDecred(namedtuple("Tx", "version inputs outputs "
                          "lock_time expiry witnesses")):
    '''Class representing a Decred transaction.'''    

    @cachedproperty
    def is_coinbase(self):
        return self.inputs[0].is_coinbase

    def __str__(self):
        return ("Tx(version={}, inputs={}, outputs={}, lock_time={}, expiry={}, witnesses={})"
                .format(self.version, self.inputs, self.outputs, self.lock_time, self.expiry, self.witnesses))    

class TxDecredInput(namedtuple("TxDecredInput", "prev_hash prev_idx tree sequence")):
    '''Class representing a decred transaction input.'''

    ZERO = bytes(32)
    MINUS_1 = 4294967295

    @cachedproperty
    def is_coinbase(self):
        return (self.prev_hash == TxInput.ZERO and
                self.prev_idx == TxInput.MINUS_1)

    def __str__(self):
        prev_hash = hash_to_str(self.prev_hash)
        return ("Input(prev_hash={}, prev_idx={}, tree={}, sequence={})"
                .format(prev_hash, self.prev_idx, self.tree, self.sequence))        

class TxDecredOutput(namedtuple("TxDecredOutput", "value version script")):

    def __str__(self):
        script = self.script.hex()
        return ("Output(value={}, version={}, script={})"
                .format(self.value, self.version, self.script))

class TxDecredWitness(namedtuple("TxDecredWitness", "value blockHeight blockIndex script")):
    def __str__(self):
        script = self.script.hex()
        return ("Witness(value={}, blockHeight={}, blockIndex={}, script={})"
                .format(self.value, self.blockHeight, self.blockIndex, script)) 

class DeserializerDecred(Deserializer):

    def read_tx(self):
        version = self._read_le_uint16()
        txType = self._read_le_uint16()
        inputs = ''
        outputs = ''
        lock_time = ''
        expiry = ''
        witnesses = '' 

        if txType == 0 or txType == 1:
            inputs = self._read_inputs()  
            outputs = self._read_outputs()
            lock_time = self._read_le_uint32()
            expiry = self._read_le_uint32()

        if txType != 1:    
            witnesses = self._read_witnesses(txType)

        return TxDecred(
            version,
            inputs,
            outputs,
            lock_time,
            expiry,
            witnesses
        )    

    def _read_witnesses(self, txType):
        read_witness = self._read_witness
        return [read_witness(txType) for i in range(self._read_varint())]

    def _read_witness(self, txType):

        value = ''
        blockHeight = ''
        blockIndex = ''
        script = ''

        if txType == 0 or txType == 2: 
            value = self._read_le_int64()
            blockHeight = self._read_le_uint32()
            blockIndex = self._read_le_uint32()
            script = self._read_varbytes()
        if txType == 3:
            script = self._read_varbytes()
        if txType == 4:
            value = self._read_le_int64()
            script = self._read_varbytes()

        return TxDecredWitness(
            value,
            blockHeight,
            blockIndex,
            script
        )

    def _read_inputs(self):
        read_input = self._read_input
        return [read_input() for i in range(self._read_varint())]

    def _read_input(self):
        return TxDecredInput(
            self._read_nbytes(32),  #prev_hash
            self._read_le_uint32(), #prev_idx
            self._read_byte(),       #tree
            self._read_le_uint32() #sequence
        )

    def _read_outputs(self):
        read_output = self._read_output
        return [read_output() for i in range(self._read_varint())]

    def _read_output(self):
        return TxDecredOutput(
            self._read_le_int64(),                   #value
            self._read_le_uint16(),                  #version
            self._read_varbytes(),  #script
        )

    def read_header(self, height, static_header_size):
        '''Return the block header bytes'''
        return self._read_nbytes(static_header_size) 

Also, here is my ENV startup

macuser@macusersmacbook:~/Projects/Decred/electrumx (master *)$  python3 ./electrumx_server.py
INFO:root:ElectrumX server starting
WARNING:Env:lowered maximum sessions from 1,000 to 0 because your open file limit is 256
INFO:Controller:event loop policy: None
INFO:Daemon:daemon #1 at 127.0.0.1:19119/ (current)
INFO:BlockProcessor:switching current directory to /Users/macuser/.electrumx/
INFO:BlockProcessor:using leveldb for DB backend
INFO:BlockProcessor:created new database
INFO:BlockProcessor:creating metadata directory
INFO:BlockProcessor:software version: ElectrumX 1.2.1
INFO:BlockProcessor:DB version: 6
INFO:BlockProcessor:coin: Decred
INFO:BlockProcessor:network: testnet
INFO:BlockProcessor:height: -1
INFO:BlockProcessor:tip: 0000000000000000000000000000000000000000000000000000000000000000
INFO:BlockProcessor:tx count: 0
INFO:BlockProcessor:flush count: 0
INFO:BlockProcessor:sync time so far: 00s
INFO:BlockProcessor:reorg limit is 200 blocks
INFO:BlockProcessor:flushing DB cache at 1,200 MB
INFO:Controller:RPC server listening on localhost:8000
INFO:Prefetcher:catching up to daemon height 247,976 (247,977 blocks behind)
INFO:Prefetcher:verified genesis block with hash 4261602a9d07d80ad47621a64ba6a07754902e496777edc4ff581946bd7bc29c
WARNING:BlockProcessor:daemon blocks do not form a chain; resetting the prefetcher
INFO:Prefetcher:catching up to daemon height 247,976 (247,977 blocks behind)
WARNING:BlockProcessor:ignoring 500 blocks starting height 10, expected 0
WARNING:BlockProcessor:ignoring 500 blocks starting height 510, expected 0
WARNING:BlockProcessor:ignoring 500 blocks starting height 1,010, expected 0
WARNING:BlockProcessor:ignoring 500 blocks starting height 1,510, expected 0
WARNING:BlockProcessor:ignoring 500 blocks starting height 2,010, expected 0
INFO:Prefetcher:verified genesis block with hash 4261602a9d07d80ad47621a64ba6a07754902e496777edc4ff581946bd7bc29c

[erasmospunk]

The headers are not hashed with blake256.

Take a look at my branch. I managed to make it work but with an ugly hack "Allow advancing errors" where it circumvents the transaction invalidation feature of the Decred blockchain.

Also I haven't had the time to clean up and release the pow_hash lib, but any python blake256 lib will do, just run the tests to be sure that it works.

[kyuupichan]

Closing as @erasmospunk explains. Have you thought about how blocks form a chain? It has nothing to do with transactions.

[cipherzzz]

@kyuupichan - Thank you for your feedback and a great product. I understand the chaining issue - we have to use a different algorithm for decred. This is the first python project I've worked on and I am new to blockchain tech.

@erasmospunk - I guess we are both working on the same thing... I looked at your deserialization logic and there is an issue that is not apparent from the decred documentation. The transactions are deserialized based on a 'type' that is derived from the 'version' bytes. There are 5 'types' (0-5) and for the types that contain witness data, there is a witness object for each input. Take a look at this repo and excuse my python - this is my first python project. https://github.com/cipherzzz/decred_tx_py_parser