search

vegaprotocol / vegatools

A go command line utility providing a bunch of tools to use with a Vega network
MIT License
3 stars 4 forks source link

[Bug]: Data mismatch between restored datanode and core #308

Closed MuthuVega closed 10 months ago

MuthuVega commented 10 months ago

Problem encountered

Observed a data mismatch between a restored datanode and core for liquidity provisions when running the vegatools datadiff tool.

Note: Its quite possible this is NOT a bug but some issue with the timing or the status of the data thats being compared, BUT raising the bug for someone to investigate.

vegatools datadiff output:

[2023-10-24T10:16:48.741Z] 2023-10-24 10:16:48.158:[INFO] [MainProcess] [MainThread] [vega_commands.py]:[datanode_vegatools_database_compare]:[ 981] /jenkins/workspace/common/system-tests-wrapper/networkdata/testnet/../vegatools difftool -s /jenkins/workspace/common/system-tests-wrapper/networkdata/testnet/vega/node5/state/node/snapshots/ -d 127.0.0.1:3067
[2023-10-24T10:16:48.741Z] 2023-10-24 10:16:48.276:[INFO] [MainProcess] [MainThread] [vega_commands.py]:[datanode_vegatools_database_compare]:[ 984] Datanode data diff output for /jenkins/workspace/common/system-tests-wrapper/networkdata/testnet/vega/node5: CompletedProcess(args=['/jenkins/workspace/common/system-tests-wrapper/networkdata/testnet/../vegatools', 'difftool', '-s', '/jenkins/workspace/common/system-tests-wrapper/networkdata/testnet/vega/node5/state/node/snapshots/', '-d', '127.0.0.1:3067'], returncode=1, stdout='found snapshot for block-height 658\nsnapshot payloads written to: /tmp/snapshot.dat\nmismatch between core and datanode: success=false, report:\nkey=accounts, matchResult=full match, coreLength=40, datanodeLength=40, coreResult=, datanodeResult=\nkey=orders, matchResult=full match, coreLength=6, datanodeLength=6, coreResult=, datanodeResult=\nkey=markets, matchResult=full match, coreLength=1, datanodeLength=1, coreResult=, datanodeResult=\nkey=parties, matchResult=full match, coreLength=12, datanodeLength=12, coreResult=, datanodeResult=\nkey=limits, matchResult=full match, coreLength=1, datanodeLength=1, coreResult=, datanodeResult=\nkey=assets, matchResult=full match, coreLength=3, datanodeLength=3, coreResult=, datanodeResult=\nkey=delegations, matchResult=full match, coreLength=2, datanodeLength=2, coreResult=, datanodeResult=\nkey=epoch, matchResult=mismatching values, coreLength=1, datanodeLength=1, coreResult=seq:60  timestamps:{start_time:1698142575551496000  expiry_time:1698142605551496000}, datanodeResult=seq:61  timestamps:{start_time:1698142607084435000  expiry_time:1698142637084435000  first_block:673}\nkey=vegaTime, matchResult=mismatching values, coreLength=1, datanodeLength=1, coreResult=1698142596113102000, datanodeResult=1698142607084435000\nkey=nodes, matchResult=full match, coreLength=2, datanodeLength=2, coreResult=, datanodeResult=\nkey=netparams, matchResult=full match, coreLength=171, datanodeLength=171, coreResult=, datanodeResult=\nkey=proposals, matchResult=full match, coreLength=3, datanodeLength=3, coreResult=, datanodeResult=\nkey=deposits, matchResult=full match, coreLength=0, datanodeLength=0, coreResult=, datanodeResult=\nkey=withdrawals, matchResult=full match, coreLength=0, datanodeLength=0, coreResult=, datanodeResult=\nkey=liquidityProvisions, matchResult=mismatching number of elements, coreLength=0, datanodeLength=1, coreResult=, datanodeResult=id:"57d4df199e34115268f71f0ebb070a7255d68b22509519fb27730a14ccb9be07"  party_id:"b2eb6b56c6fea25c7d7bc9adb9adf091ca7455ee55b6e070843695f7f1f1c8d7"  created_at:1698142528746872000  updated_at:1698142528746872000  market_id:"316dc01c774b7c6d2ca9e31a332dc4b0fcbcc9cc476950030aab22ea04bf67a3"  commitment_amount:"390500000000000000000000"  fee:"0.3"  version:1  status:STATUS_ACTIVE  reference:"1CADBF9626C848F"\n\nkey=stake, matchResult=full match, coreLength=3, datanodeLength=3, coreResult=, datanodeResult=\nkey=transfers, matchResult=full match, coreLength=0, datanodeLength=0, coreResult=, datanodeResult=\n\n', stderr='/tmp/\ncompleted diff for accounts with status 0\ncompleted diff for orders with status 0\ncompleted diff for markets with status 0\ncompleted diff for parties with status 0\ncompleted diff for limits with status 0\ncompleted diff for assets with status 0\ncompleted diff for delegations with status 0\ncompleted diff for epoch with status 2\ncompleted diff for vegaTime with status 2\ncompleted diff for nodes with status 0\ncompleted diff for netparams with status 0\ncompleted diff for proposals with status 0\ncompleted diff for deposits with status 0\ncompleted diff for withdrawals with status 0\ncompleted diff for liquidityProvisions with status 1\ncompleted diff for stake with status 0\ncompleted diff for transfers with status 0\ncompleted diff with success? false\nError: mismatch between core and datanode: success=false, report:\nkey=accounts, matchResult=full match, coreLength=40, datanodeLength=40, coreResult=, datanodeResult=\nkey=orders, matchResult=full match, coreLength=6, datanodeLength=6, coreResult=, datanodeResult=\nkey=markets, matchResult=full match, coreLength=1, datanodeLength=1, coreResult=, datanodeResult=\nkey=parties, matchResult=full match, coreLength=12, datanodeLength=12, coreResult=, datanodeResult=\nkey=limits, matchResult=full match, coreLength=1, datanodeLength=1, coreResult=, datanodeResult=\nkey=assets, matchResult=full match, coreLength=3, datanodeLength=3, coreResult=, datanodeResult=\nkey=delegations, matchResult=full match, coreLength=2, datanodeLength=2, coreResult=, datanodeResult=\nkey=epoch, matchResult=mismatching values, coreLength=1, datanodeLength=1, coreResult=seq:60  timestamps:{start_time:1698142575551496000  expiry_time:1698142605551496000}, datanodeResult=seq:61  timestamps:{start_time:1698142607084435000  expiry_time:1698142637084435000  first_block:673}\nkey=vegaTime, matchResult=mismatching values, coreLength=1, datanodeLength=1, coreResult=1698142596113102000, datanodeResult=1698142607084435000\nkey=nodes, matchResult=full match, coreLength=2, datanodeLength=2, coreResult=, datanodeResult=\nkey=netparams, matchResult=full match, coreLength=171, datanodeLength=171, coreResult=, datanodeResult=\nkey=proposals, matchResult=full match, coreLength=3, datanodeLength=3, coreResult=, datanodeResult=\nkey=deposits, matchResult=full match, coreLength=0, datanodeLength=0, coreResult=, datanodeResult=\nkey=withdrawals, matchResult=full match, coreLength=0, datanodeLength=0, coreResult=, datanodeResult=\nkey=liquidityProvisions, matchResult=mismatching number of elements, coreLength=0, datanodeLength=1, coreResult=, datanodeResult=id:"57d4df199e34115268f71f0ebb070a7255d68b22509519fb27730a14ccb9be07"  party_id:"b2eb6b56c6fea25c7d7bc9adb9adf091ca7455ee55b6e070843695f7f1f1c8d7"  created_at:1698142528746872000  updated_at:1698142528746872000  market_id:"316dc01c774b7c6d2ca9e31a332dc4b0fcbcc9cc476950030aab22ea04bf67a3"  commitment_amount:"390500000000000000000000"  fee:"0.3"  version:1  status:STATUS_ACTIVE  reference:"1CADBF9626C848F"\n\nkey=stake, matchResult=full match, coreLength=3, datanodeLength=3, coreResult=, datanodeResult=\nkey=transfers, matchResult=full match, coreLength=0, datanodeLength=0, coreResult=, datanodeResult=\n\nUsage:\n  vegatools difftool [flags]\n\nFlags:\n  -r, --block-height int      block-height of the snapshot to dump\n  -d, --datanode string       datanode url\n  -h, --help                  help for difftool\n  -s, --snap-db-path string   path to the goleveldb database folder\n\n')

Observed behaviour

Data difference reported by the vegatools datadiff tool

completed diff for liquidityProvisions with status 1
key=liquidityProvisions, matchResult=mismatching number of elements, coreLength=0, datanodeLength=1, coreResult=, datanodeResult=id:"57d4df199e34115268f71f0ebb070a7255d68b22509519fb27730a14ccb9be07"  party_id:"b2eb6b56c6fea25c7d7bc9adb9adf091ca7455ee55b6e070843695f7f1f1c8d7"  created_at:1698142528746872000  updated_at:1698142528746872000  market_id:"316dc01c774b7c6d2ca9e31a332dc4b0fcbcc9cc476950030aab22ea04bf67a3"  commitment_amount:"390500000000000000000000"  fee:"0.3"  version:1  status:STATUS_ACTIVE  reference:"1CADBF9626C848F"

Expected behaviour

No data difference between datanode and core.

Steps to reproduce

Run the below system test:

System Test Name: test_datanode_compare_restored_datanode_data_against_core
System Test Branch: enable_skipped_datanode_test

Software version

vega - develop

Failing test

test_datanode_compare_restored_datanode_data_against_core

Jenkins run

https://jenkins.vega.rocks/job/common/job/system-tests-wrapper/25660/testReport/junit/tests.datanode/datanode_test/test_datanode_compare_restored_datanode_data_against_core/

Configuration used

No response

Relevant log output

All relevant logs pasted above for reference.
wwestgarth commented 10 months ago

This is a problem with the difftool, when it reads from the snapshot it is only looking at the V1 payload and not the new V2 payload.