Observed a data mismatch between a restored datanode and core for liquidity provisions when running the vegatools datadiff tool.
Note: Its quite possible this is NOT a bug but some issue with the timing or the status of the data thats being compared, BUT raising the bug for someone to investigate.
vegatools datadiff output:
[2023-10-24T10:16:48.741Z] 2023-10-24 10:16:48.158:[INFO] [MainProcess] [MainThread] [vega_commands.py]:[datanode_vegatools_database_compare]:[ 981] /jenkins/workspace/common/system-tests-wrapper/networkdata/testnet/../vegatools difftool -s /jenkins/workspace/common/system-tests-wrapper/networkdata/testnet/vega/node5/state/node/snapshots/ -d 127.0.0.1:3067
[2023-10-24T10:16:48.741Z] 2023-10-24 10:16:48.276:[INFO] [MainProcess] [MainThread] [vega_commands.py]:[datanode_vegatools_database_compare]:[ 984] Datanode data diff output for /jenkins/workspace/common/system-tests-wrapper/networkdata/testnet/vega/node5: CompletedProcess(args=['/jenkins/workspace/common/system-tests-wrapper/networkdata/testnet/../vegatools', 'difftool', '-s', '/jenkins/workspace/common/system-tests-wrapper/networkdata/testnet/vega/node5/state/node/snapshots/', '-d', '127.0.0.1:3067'], returncode=1, stdout='found snapshot for block-height 658\nsnapshot payloads written to: /tmp/snapshot.dat\nmismatch between core and datanode: success=false, report:\nkey=accounts, matchResult=full match, coreLength=40, datanodeLength=40, coreResult=, datanodeResult=\nkey=orders, matchResult=full match, coreLength=6, datanodeLength=6, coreResult=, datanodeResult=\nkey=markets, matchResult=full match, coreLength=1, datanodeLength=1, coreResult=, datanodeResult=\nkey=parties, matchResult=full match, coreLength=12, datanodeLength=12, coreResult=, datanodeResult=\nkey=limits, matchResult=full match, coreLength=1, datanodeLength=1, coreResult=, datanodeResult=\nkey=assets, matchResult=full match, coreLength=3, datanodeLength=3, coreResult=, datanodeResult=\nkey=delegations, matchResult=full match, coreLength=2, datanodeLength=2, coreResult=, datanodeResult=\nkey=epoch, matchResult=mismatching values, coreLength=1, datanodeLength=1, coreResult=seq:60 timestamps:{start_time:1698142575551496000 expiry_time:1698142605551496000}, datanodeResult=seq:61 timestamps:{start_time:1698142607084435000 expiry_time:1698142637084435000 first_block:673}\nkey=vegaTime, matchResult=mismatching values, coreLength=1, datanodeLength=1, coreResult=1698142596113102000, datanodeResult=1698142607084435000\nkey=nodes, matchResult=full match, coreLength=2, datanodeLength=2, coreResult=, datanodeResult=\nkey=netparams, matchResult=full match, coreLength=171, datanodeLength=171, coreResult=, datanodeResult=\nkey=proposals, matchResult=full match, coreLength=3, datanodeLength=3, coreResult=, datanodeResult=\nkey=deposits, matchResult=full match, coreLength=0, datanodeLength=0, coreResult=, datanodeResult=\nkey=withdrawals, matchResult=full match, coreLength=0, datanodeLength=0, coreResult=, datanodeResult=\nkey=liquidityProvisions, matchResult=mismatching number of elements, coreLength=0, datanodeLength=1, coreResult=, datanodeResult=id:"57d4df199e34115268f71f0ebb070a7255d68b22509519fb27730a14ccb9be07" party_id:"b2eb6b56c6fea25c7d7bc9adb9adf091ca7455ee55b6e070843695f7f1f1c8d7" created_at:1698142528746872000 updated_at:1698142528746872000 market_id:"316dc01c774b7c6d2ca9e31a332dc4b0fcbcc9cc476950030aab22ea04bf67a3" commitment_amount:"390500000000000000000000" fee:"0.3" version:1 status:STATUS_ACTIVE reference:"1CADBF9626C848F"\n\nkey=stake, matchResult=full match, coreLength=3, datanodeLength=3, coreResult=, datanodeResult=\nkey=transfers, matchResult=full match, coreLength=0, datanodeLength=0, coreResult=, datanodeResult=\n\n', stderr='/tmp/\ncompleted diff for accounts with status 0\ncompleted diff for orders with status 0\ncompleted diff for markets with status 0\ncompleted diff for parties with status 0\ncompleted diff for limits with status 0\ncompleted diff for assets with status 0\ncompleted diff for delegations with status 0\ncompleted diff for epoch with status 2\ncompleted diff for vegaTime with status 2\ncompleted diff for nodes with status 0\ncompleted diff for netparams with status 0\ncompleted diff for proposals with status 0\ncompleted diff for deposits with status 0\ncompleted diff for withdrawals with status 0\ncompleted diff for liquidityProvisions with status 1\ncompleted diff for stake with status 0\ncompleted diff for transfers with status 0\ncompleted diff with success? false\nError: mismatch between core and datanode: success=false, report:\nkey=accounts, matchResult=full match, coreLength=40, datanodeLength=40, coreResult=, datanodeResult=\nkey=orders, matchResult=full match, coreLength=6, datanodeLength=6, coreResult=, datanodeResult=\nkey=markets, matchResult=full match, coreLength=1, datanodeLength=1, coreResult=, datanodeResult=\nkey=parties, matchResult=full match, coreLength=12, datanodeLength=12, coreResult=, datanodeResult=\nkey=limits, matchResult=full match, coreLength=1, datanodeLength=1, coreResult=, datanodeResult=\nkey=assets, matchResult=full match, coreLength=3, datanodeLength=3, coreResult=, datanodeResult=\nkey=delegations, matchResult=full match, coreLength=2, datanodeLength=2, coreResult=, datanodeResult=\nkey=epoch, matchResult=mismatching values, coreLength=1, datanodeLength=1, coreResult=seq:60 timestamps:{start_time:1698142575551496000 expiry_time:1698142605551496000}, datanodeResult=seq:61 timestamps:{start_time:1698142607084435000 expiry_time:1698142637084435000 first_block:673}\nkey=vegaTime, matchResult=mismatching values, coreLength=1, datanodeLength=1, coreResult=1698142596113102000, datanodeResult=1698142607084435000\nkey=nodes, matchResult=full match, coreLength=2, datanodeLength=2, coreResult=, datanodeResult=\nkey=netparams, matchResult=full match, coreLength=171, datanodeLength=171, coreResult=, datanodeResult=\nkey=proposals, matchResult=full match, coreLength=3, datanodeLength=3, coreResult=, datanodeResult=\nkey=deposits, matchResult=full match, coreLength=0, datanodeLength=0, coreResult=, datanodeResult=\nkey=withdrawals, matchResult=full match, coreLength=0, datanodeLength=0, coreResult=, datanodeResult=\nkey=liquidityProvisions, matchResult=mismatching number of elements, coreLength=0, datanodeLength=1, coreResult=, datanodeResult=id:"57d4df199e34115268f71f0ebb070a7255d68b22509519fb27730a14ccb9be07" party_id:"b2eb6b56c6fea25c7d7bc9adb9adf091ca7455ee55b6e070843695f7f1f1c8d7" created_at:1698142528746872000 updated_at:1698142528746872000 market_id:"316dc01c774b7c6d2ca9e31a332dc4b0fcbcc9cc476950030aab22ea04bf67a3" commitment_amount:"390500000000000000000000" fee:"0.3" version:1 status:STATUS_ACTIVE reference:"1CADBF9626C848F"\n\nkey=stake, matchResult=full match, coreLength=3, datanodeLength=3, coreResult=, datanodeResult=\nkey=transfers, matchResult=full match, coreLength=0, datanodeLength=0, coreResult=, datanodeResult=\n\nUsage:\n vegatools difftool [flags]\n\nFlags:\n -r, --block-height int block-height of the snapshot to dump\n -d, --datanode string datanode url\n -h, --help help for difftool\n -s, --snap-db-path string path to the goleveldb database folder\n\n')
Observed behaviour
Data difference reported by the vegatools datadiff tool
completed diff for liquidityProvisions with status 1
key=liquidityProvisions, matchResult=mismatching number of elements, coreLength=0, datanodeLength=1, coreResult=, datanodeResult=id:"57d4df199e34115268f71f0ebb070a7255d68b22509519fb27730a14ccb9be07" party_id:"b2eb6b56c6fea25c7d7bc9adb9adf091ca7455ee55b6e070843695f7f1f1c8d7" created_at:1698142528746872000 updated_at:1698142528746872000 market_id:"316dc01c774b7c6d2ca9e31a332dc4b0fcbcc9cc476950030aab22ea04bf67a3" commitment_amount:"390500000000000000000000" fee:"0.3" version:1 status:STATUS_ACTIVE reference:"1CADBF9626C848F"
Expected behaviour
No data difference between datanode and core.
Steps to reproduce
Run the below system test:
System Test Name: test_datanode_compare_restored_datanode_data_against_core
System Test Branch: enable_skipped_datanode_test
Problem encountered
Observed a data mismatch between a restored datanode and core for liquidity provisions when running the vegatools datadiff tool.
Note: Its quite possible this is NOT a bug but some issue with the timing or the status of the data thats being compared, BUT raising the bug for someone to investigate.
vegatools datadiff output:
Observed behaviour
Data difference reported by the vegatools datadiff tool
Expected behaviour
No data difference between datanode and core.
Steps to reproduce
Software version
vega - develop
Failing test
test_datanode_compare_restored_datanode_data_against_core
Jenkins run
https://jenkins.vega.rocks/job/common/job/system-tests-wrapper/25660/testReport/junit/tests.datanode/datanode_test/test_datanode_compare_restored_datanode_data_against_core/
Configuration used
No response
Relevant log output