Getting Error : HeaderAncestryMismatch while connecting to Polkadex Testnet

[zktony]

Sep 25 15:37:35 Ubuntu-2004-focal-64-minimal start-tee.sh[1840152]: Remaining blocks to fetch until last synced header: 2000
Sep 25 15:37:35 Ubuntu-2004-focal-64-minimal start-tee.sh[1840152]: Remaining blocks to fetch until last synced header: 1000
Sep 25 15:37:35 Ubuntu-2004-focal-64-minimal start-tee.sh[1840152]: [2021-09-25T13:37:35Z ERROR substratee_worker_enclave] Block verification failed. Error : HeaderAncestryMismatch
Sep 25 15:37:35 Ubuntu-2004-focal-64-minimal start-tee.sh[1840152]: [2021-09-25T13:37:35Z ERROR substratee_worker] SGX_ERROR_UNEXPECTED
Sep 25 15:37:37 Ubuntu-2004-focal-64-minimal start-tee.sh[1840152]: Chain Relay is synced until block: 0
Sep 25 15:37:37 Ubuntu-2004-focal-64-minimal start-tee.sh[1840152]: Last finalized block number: 2586
Sep 25 15:37:37 Ubuntu-2004-focal-64-minimal start-tee.sh[1840152]: Remaining blocks to fetch until last synced header: 2000
Sep 25 15:37:37 Ubuntu-2004-focal-64-minimal start-tee.sh[1840152]: Remaining blocks to fetch until last synced header: 1000
Sep 25 15:37:37 Ubuntu-2004-focal-64-minimal start-tee.sh[1840152]: [2021-09-25T13:37:37Z ERROR substratee_worker_enclave] Block verification failed. Error : HeaderAncestryMismatch
Sep 25 15:37:37 Ubuntu-2004-focal-64-minimal start-tee.sh[1840152]: [2021-09-25T13:37:37Z ERROR substratee_worker] SGX_ERROR_UNEXPECTED

[zktony]

Update: When running same binary locally in dev mode then its working

[haerdib]

The only time this error is thrown is when the following is the case:

https://github.com/Polkadex-Substrate/polkadexTEE-worker/blob/15ad2bdac0049a6fb2f3d73e478836674bcb237f/enclave/chain_relay/src/lib.rs#L184-L186

Meaning, that the currently imported block is not the next inline of the block chain. So somewhere, there was a block loss. And this is not acceptable - a block being lost means that there might be extrinsic missed. So in this case, we should rerequest blocks from the node.

[haerdib]

But seeing this error happened many times, that seems strange - a Block lost all the time? Seems unlikely. Could you give me the following information: 1) the bin folder of the worker 2) on which block number is the error thrown?

[haerdib]

I've tried to reproduce the problem locally, but I was not able to. I tried on a node having above to 4'000 block and didn't get a HeaderAncestryMismatch:

 ./substratee-worker -P 2094 -p 9994 -F 8001/api/v2/ws -f 127.0.0.1 run
*** Starting substraTEE-worker
Advertising worker api at ws://127.0.0.1:2094
Encointer Worker v0.8.0
*** Starting enclave in development mode
MRENCLAVE=A5y3vA2e5bme77cxohoDo5QzZwqCnY8eRLL5EaX39KrD
MU-RA server listening on ws://127.0.0.1:3443
OpenFinex Client listening on ws://127.0.0.1:8001:/api/v2/ws
rpc worker server listening on ws://127.0.0.1:2094
Successfully initialised top pool
[2021-10-05T11:48:15Z ERROR substratee_worker_enclave::openfinex::openfinex_client] Could not handshake with openfinex server: Unknown error: 111 (os error: 111)
[+] Httpclient successfully created within enclave
[2021-10-05T11:48:15Z ERROR substratee_worker_enclave::openfinex::openfinex_client] Could not read from TCP stream: EOF
TLS read error: -1
Connection closed
[+] Tcp Client closed
    [Enclave] Create attestation report successful
[>] Register the enclave (send the extrinsic)
[<] Extrinsic got finalized. Hash: Some(0x98049fb583865ac3d1cb954d7f1759a55dd840a0e258645ae786660eeaa51996)

Chain Relay is synced until block: 0
Last finalized block number: 4457

Remaining blocks to fetch until last synced header: 4000
Remaining blocks to fetch until last synced header: 3000
Remaining blocks to fetch until last synced header: 2000
Remaining blocks to fetch until last synced header: 1000
Enclave wants to send 1002 extrinsics
Synced 1001 blocks out of 4458 finalized blocks
[2021-10-05T11:48:48Z ERROR chain_relay] Block 1536 contained invalid justification: InvalidFinalityProof(BadJustification("invalid signature for precommit in grandpa justification"))
Enclave wants to send 1002 extrinsics
Synced 2001 blocks out of 4458 finalized blocks
[2021-10-05T11:49:03Z ERROR chain_relay] Block 2048 contained invalid justification: InvalidFinalityProof(BadJustification("invalid signature for precommit in grandpa justification"))
[2021-10-05T11:49:05Z ERROR chain_relay] Block 2129 contained invalid justification: InvalidFinalityProof(BadJustification("invalid signature for precommit in grandpa justification"))
[2021-10-05T11:49:13Z ERROR chain_relay] Block 2560 contained invalid justification: InvalidFinalityProof(BadJustification("invalid signature for precommit in grandpa justification"))
Enclave wants to send 1002 extrinsics
Synced 3001 blocks out of 4458 finalized blocks
[2021-10-05T11:49:28Z ERROR chain_relay] Block 3072 contained invalid justification: InvalidFinalityProof(BadJustification("invalid signature for precommit in grandpa justification"))
[2021-10-05T11:49:31Z ERROR chain_relay] Block 3199 contained invalid justification: InvalidFinalityProof(BadJustification("invalid signature for precommit in grandpa justification"))
[2021-10-05T11:49:45Z ERROR chain_relay] Block 3584 contained invalid justification: InvalidFinalityProof(BadJustification("invalid signature for precommit in grandpa justification"))
Enclave wants to send 1002 extrinsics
Synced 4001 blocks out of 4458 finalized blocks
[2021-10-05T11:50:12Z ERROR chain_relay] Block 4096 contained invalid justification: InvalidFinalityProof(BadJustification("invalid signature for precommit in grandpa justification"))
[2021-10-05T11:50:40Z ERROR chain_relay] Block 4260 contained invalid justification: InvalidFinalityProof(BadJustification("invalid signature for precommit in grandpa justification"))
Enclave wants to send 459 extrinsics
Synced 4458 blocks out of 4458 finalized blocks
*** [+] Finished syncing chain relay

*** Subscribing to events
Successfully started snapshot loop
[+] Subscribed to events. waiting...
Chain Relay is synced until block: 4457
Last finalized block number: 4501

Some invalid finality proofs, but we're ignoring that for now.. Not sure how I can help here.