search

Concordium / concordium-misc-tools

A collection of small tools with a well-defined purpose
Apache License 2.0
4 stars 6 forks source link

kpi-tracker seems to try to insert a duplicate block #137

Open abizjak opened 5 months ago

abizjak commented 5 months ago 
[2024-02-13T03:13:58Z INFO  concordium_kpi_tracker] Processed blocks 8e9f885d953f27a3d9d1c2c8e37b9cf75db9b04fda5f75a256764628c21e015c at heights 12846889 with 2 transactions in 4ms
[2024-02-13T03:14:01Z INFO  concordium_kpi_tracker] Processed blocks fdb768f058b7d829ee78bf1a8cb90117e36677f47bdb85d91980abab50014fdd at heights 12846890 with 2 transactions in 3ms
[2024-02-13T03:14:02Z INFO  concordium_kpi_tracker] Processed blocks 03c60aa5c5de8f505e557fbf49c59c2843e5324759b7c7796bd6bb94ab4fd5c1 at heights 12846891 with 5 transactions in 5ms
[2024-02-13T03:14:04Z INFO  concordium_kpi_tracker] Processed blocks 9980e2c857bbd4f6f8841c0f9f5e1d73f8ce9a0fcd239564675cc631b909ae47 at heights 12846892 with 5 transactions in 4ms
[2024-02-13T03:14:06Z INFO  concordium_kpi_tracker] Processed blocks d871af341616a2c8c8d813e73b30e3fcc691908f56e9637d746a19b456c03d9d at heights 12846893 with 3 transactions in 4ms
[2024-02-13T03:14:08Z INFO  concordium_kpi_tracker] Processed blocks c8c197f26aede00133530a6c5ff723eb409e6abe880e7d2eba5dd2d8fab8b252 at heights 12846894 with 4 transactions in 4ms
[2024-02-13T03:14:10Z INFO  concordium_kpi_tracker] Processed blocks 8fa7731e0ffb0b716fa143a0fee1172797e0f1ced7c0d27bede5d9061272462e at heights 12846895 with 3 transactions in 4ms
[2024-02-13T03:14:12Z INFO  concordium_kpi_tracker] Processed blocks b3e6ea7377fc95fe47c9d94cfc1b4d5d7035b06d41682c8408814b03978af2e2 at heights 12846896 with 4 transactions in 4ms
[2024-02-13T03:14:14Z INFO  concordium_kpi_tracker] Processed blocks f9df791f2f464c926f6a7b79b78182dd2fa36f5ff2ea7e5b02683f39bcad297e at heights 12846897 with 6 transactions in 5ms
[2024-02-13T03:14:16Z INFO  concordium_kpi_tracker] Processed blocks b9e48c166cb03122e2b645f113dab626a6bbe769685ee7038141dd9b2df43c0c at heights 12846898 with 4 transactions in 4ms
[2024-02-13T03:14:18Z INFO  concordium_kpi_tracker] Processed blocks 79af1c4a76a7c13927517d5cec6341892ec84dfd68f802b2c094b76c780178c6 at heights 12846899 with 1 transactions in 3ms
[2024-02-13T03:14:20Z INFO  concordium_kpi_tracker] Processed blocks f4d54bd0d8af35490a5fef73c4117354d5985f754171536d2fbfecc21aa17ac0 at heights 12846900 with 7 transactions in 11ms
[2024-02-13T03:14:22Z INFO  concordium_kpi_tracker] Processed blocks 84acbd7a07f0530daafd90bcd1d3eb79c8265b57494fc547106104e421d266f1 at heights 12846901 with 5 transactions in 4ms
[2024-02-13T03:14:24Z INFO  concordium_kpi_tracker] Processed blocks 7554ee15fa80787a2b6cd7b4a1ac28a1e74e11a88d5826c79ddf46c8ddaee15e at heights 12846902 with 1 transactions in 3ms
[2024-02-13T03:14:26Z INFO  concordium_kpi_tracker] Processed blocks d5a109d2a626b6a3adc857bd8820bf7cd5be4ac77e71182922ad27309c342f83 at heights 12846903 with 7 transactions in 4ms
[2024-02-13T03:14:28Z INFO  concordium_kpi_tracker] Processed blocks 6ace56380d6343f43ce86a048cc94a8ffc15d7e3ce482ba434f7ba3c170704a1 at heights 12846904 with 3 transactions in 3ms
[2024-02-13T03:14:30Z INFO  concordium_kpi_tracker] Processed blocks 3789eef445b7bea86d9541e5c9b6b3d307b8737882069a665830729827f67bed at heights 12846905 with 3 transactions in 3ms
[2024-02-13T03:14:43Z WARN  concordium_kpi_tracker] Endpoint http://172.23.192.20:20000/ failed with error Could not get block info for block: 12846907. Trying next.
[2024-02-13T03:14:43Z INFO  concordium_kpi_tracker] Processing blocks from height 12846907 using node http://172.23.208.20:20000/
[2024-02-13T03:14:43Z INFO  concordium_kpi_tracker] Processed blocks 3b60652e8fa027fb5bd378dbd415ad29c91f915048565be456990ae2fbd869ca at heights 12846906 with 4 transactions in 4ms
[2024-02-13T03:14:43Z ERROR concordium_kpi_tracker] Database connection lost due to db error: ERROR: duplicate key value violates unique constraint "blocks_hash_key"
    DETAIL: Key (hash)=(\x3b60652e8fa027fb5bd378dbd415ad29c91f915048565be456990ae2fbd869ca) already exists.: ERROR: duplicate key value violates unique constraint "blocks_hash_key"
    DETAIL: Key (hash)=(\x3b60652e8fa027fb5bd378dbd415ad29c91f915048565be456990ae2fbd869ca) already exists.. Will attempt to reconnect in 1000ms.
[2024-02-13T03:14:44Z ERROR concordium_kpi_tracker] Database connection lost due to db error: ERROR: duplicate key value violates unique constraint "blocks_hash_key"
    DETAIL: Key (hash)=(\x3b60652e8fa027fb5bd378dbd415ad29c91f915048565be456990ae2fbd869ca) already exists.: ERROR: duplicate key value violates unique constraint "blocks_hash_key"
    DETAIL: Key (hash)=(\x3b60652e8fa027fb5bd378dbd415ad29c91f915048565be456990ae2fbd869ca) already exists.. Will attempt to reconnect in 2000ms.
[2024-02-13T03:14:46Z ERROR concordium_kpi_tracker] Database connection lost due to db error: ERROR: duplicate key value violates unique constraint "blocks_hash_key"
    DETAIL: Key (hash)=(\x3b60652e8fa027fb5bd378dbd415ad29c91f915048565be456990ae2fbd869ca) already exists.: ERROR: duplicate key value violates unique constraint "blocks_hash_key"
    DETAIL: Key (hash)=(\x3b60652e8fa027fb5bd378dbd415ad29c91f915048565be456990ae2fbd869ca) already exists.. Will attempt to reconnect in 4000ms.
[2024-02-13T03:14:50Z ERROR concordium_kpi_tracker] Database connection lost due to db error: ERROR: duplicate key value violates unique constraint "blocks_hash_key"
    DETAIL: Key (hash)=(\x3b60652e8fa027fb5bd378dbd415ad29c91f915048565be456990ae2fbd869ca) already exists.: ERROR: duplicate key value violates unique constraint "blocks_hash_key"
    DETAIL: Key (hash)=(\x3b60652e8fa027fb5bd378dbd415ad29c91f915048565be456990ae2fbd869ca) already exists.. Will attempt to reconnect in 8000ms.
[2024-02-13T03:14:58Z ERROR concordium_kpi_tracker] Database connection lost due to db error: ERROR: duplicate key value violates unique constraint "blocks_hash_key"
    DETAIL: Key (hash)=(\x3b60652e8fa027fb5bd378dbd415ad29c91f915048565be456990ae2fbd869ca) already exists.: ERROR: duplicate key value violates unique constraint "blocks_hash_key"
    DETAIL: Key (hash)=(\x3b60652e8fa027fb5bd378dbd415ad29c91f915048565be456990ae2fbd869ca) already exists.. Will attempt to reconnect in 16000ms.
[2024-02-13T03:15:14Z ERROR concordium_kpi_tracker] Database connection lost due to db error: ERROR: duplicate key value violates unique constraint "blocks_hash_key"

The block it is trying to insert again is at height 12846906.

Restarting fixed this, so the most likely cause is that the queue between the query thread and db thread somehow maintained the block after it was already inserted.

soerenbf commented 2 months ago

Can we just add conflict resolution to ignore duplicates at the DB level?

abizjak commented 2 months ago

That's not fixing the problem. There is a bug somewhere since we should never be trying to insert the same thing twice. And patching the symptom is not the way.