Investigate discrepancies between indexer state, oasis-scan, oasis-node

Andrew7234 commented 1 year ago

Indexer: https://index.oasislabs.com/v1/consensus/accounts/oasis1qzzwkrjmyrzey46hyste5xszue3ggy90ag7k3687 available: 375185453505

Oasisscan: https://api.oasisscan.com/mainnet/chain/account/info/oasis1qzzwkrjmyrzey46hyste5xszue3ggy90ag7k3687 available: 499.933455244

Notes: @aefhm found this account while searching for validators with few delegators.

aefhm commented 1 year ago

I think the escrow_balance_debonding math is probably wrong as seen on this account.

pro-wh commented 1 year ago

wrong oasisscan link. that one shows outgoing delegations, indexer shows incoming delegations

https://www.oasisscan.com/validators/detail/oasis1qzzwkrjmyrzey46hyste5xszue3ggy90ag7k3687

36,739,276.69 ROSE

mitjat commented 1 year ago

Looks like this ticket is not about any specific known inconsistency any more. I'm therefore mapping it to the cover ticket in clickup that deals with inconsistencies: https://app.clickup.com/t/3qtwdv5

mitjat commented 1 year ago

We're focusing on incorrect balance tracking, but AFAIK the issue of incorrect node registry state also hasn't been fixed yet. I ran bisection (over different heights) to figure out when indexer starts diverging from StateToGenesis output. The discrepancy is large-ish and happens all at once: Height 8050155 is GOOD (no discrepancy), 8050156 is BAD

                    Error:          Not equal:                                                                                                                                                                [4/45732]
                                expected: 130                                                                                                                                                 
                                actual  : 108                                                                                                                                                 
                Test:           TestGenesisFull                                                                                                                                               
                Messages:       wrong number of nodes                                                                                                                                         
    genesis_test.go:447: node g/gNUJ3L8NbJilg+ivA0JHXFoCpzXwedqSLO3Xd0FSM= expected, but not found                                                                                            
    genesis_test.go:447: node bsq05GjUhJU7CdNOGrI2satPmpAUjRmT0gAGbWyfhLw= expected, but not found                                                                                            
    genesis_test.go:447: node o+KFxCWnNyUstx2iCbMDvTsf+P/pDCC4lLxlljtOUeM= expected, but not found                                                                                            
    genesis_test.go:447: node bbkPZRl8KA//0joD1R9HPnucJ4GklCoAtUoJrincsoo= expected, but not found                                                                                            
    genesis_test.go:447: node DudLYGDVKMgh/OZ0rOsoxeCNzSxalm4bvQsrsE/H9XA= expected, but not found                                                                                            
    genesis_test.go:447: node J2vFhYaht7l1B++dBUTzqubT72WfOeeBkI8rOamrgGc= expected, but not found                                                                                            
    genesis_test.go:447: node ONNtkoEg0gKzrYd+ERdUz5Xq/D7S72IyCGbc6lJQcZk= expected, but not found                                                                                            
    genesis_test.go:447: node qHFe9LJXRDRERZgyJFCzbd4YSwqekUrrkkVb9AWaHXQ= expected, but not found                                                                                            
    genesis_test.go:447: node FFKV7PJ9eyTwGjAf5CjO/l0Ec1mfeE2ViXwxTYcwmPo= expected, but not found                                                                                            
    genesis_test.go:447: node HwsTYDWFD/V6+DFRWxSfmKffLjzxLJjO45BNHk1gvZw= expected, but not found                                                                                            
    genesis_test.go:447: node +zVbgQqOdY90Z2NQKXFByNT0OwLxj/Ho4j4qT5u2yKM= expected, but not found                                                                                            
    genesis_test.go:447: node +isN1pgldZUB/ayTb8e1rbD4vXWiOsFKxOjZlZKvcEU= expected, but not found                                                                                            
    genesis_test.go:447: node kilLHKVgCJC5T62styokgO0IHWmLClT85XzFDSBCjok= expected, but not found                                                                                            
    genesis_test.go:447: node axFry86SMrw5X4ITO+YHLLrVqW6N1vcaTgrTOJfQPCc= expected, but not found                                                                                            
    genesis_test.go:447: node Wb0BR+Ih7yuYgC+m4tFqtPP8G4jDH79iJaYkgNx6DY4= expected, but not found                                                                                            
    genesis_test.go:447: node 1Ofik7xZJ3Tb/JE+zHKNe3l8NbSK9h/92fjynrW7xw0= expected, but not found                                                                                            
    genesis_test.go:447: node Eeze8MB6KgnOs8rE1E90QX5i6JFq+z3e0dkTcxIzF5Y= expected, but not found                                                                                            
    genesis_test.go:447: node NwJb7kl/uJUbhnYoxJXWpcc2zmbRVW6Norps326dz+k= expected, but not found                                                                                            
    genesis_test.go:447: node skPNqlh8lnR83RyK2clbAscktu5jY/fWq5OzcSZL6Iw= expected, but not found                                                                                            
    genesis_test.go:447: node c7ycVhiEqHbQfOdeeaj965nuxN8Hsb4BxMPbaPIipRw= expected, but not found                                                                                            
    genesis_test.go:447: node CfdTYbSw5r7lgm2oNKIv8uHeJFmvl5q9iFcV1P5zFS0= expected, but not found                                                                                            
    genesis_test.go:447: node vVcKIFmjqLRSp/sjKeX3yPQ3R/jf3lU2hAzXYd+4wnk= expected, but not found

Will continue by looking at SQL statements, outputs from the node, etc.

pro-wh commented 1 year ago

which side is 'expected' and which side is 'actual' on that readout?

mitjat commented 1 year ago

Well spotted. 130 is actual, 108 is expected. We produce too many nodes.

Some more observations:

8050156 is the start of a new epoch
it is the first epoch-start since damask genesis in which node registrations expire
the nodes we're missing are exactly the nodes that we (correctly, IMO and based on what I know so far) deleted because they expired.

We test equality incorrectly: our usage has expected as the first arg, API has expected as the second arg. I fixed some of these elsewhere already but looks like we have more :|

oasisprotocol / nexus

Investigate discrepancies between indexer state, oasis-scan, oasis-node #211