Data Integrity Check

[bijakatlykkex]

Hello,

Is there any way to check if data in Azure in OK?

Actually in our deployment of QBit.Ninja we have started to get some inconsistencies in the results (Our QBit.Ninja code is up to date now). For example the url "balances/38DdqhVuqb36jjmxTbvBkiFPXKA8EmW9Ly/summary?colored=true&at=446308" returns transactionCount 7385 on our deployment but 1384 on api.qbit.ninja (the 1384 seems to be the correct one) also wrong value for one of the assets (AJPMQpygd8V9UCAxwFYYHYXLHJ7dUkQJ5w). [Block 446307 is fine, 446308 is the first one with difference].

Without "colored=true" things seems to be identical, also number of differences increases as the number of block increases.

A document on data structure may also be helpful.

Thank you

[NicolasDorier]

wow this is bad. The problem can come from the balanceSummary, can you list your transactions and check they match ? My guess is that Balance summary is not returning the right result, but balances and transactions are OK. Can you check?

[NicolasDorier]

btw I just checked. api.qbit.ninja returns 7385 as well.

[bijakatlykkex]

About 7835, I have attached what I get.

[bijakatlykkex]

About balances and transactions being OK, as I checked by eye yes they are fine and match to what api.qbit.ninja returns (Even with color info, our output has additional address field added during build, so non automatic check).

Balance summary seems to be OK but just without color info, when I append "?colored=true" to balance summary, things change, even transactionCount changes from 7384 to 7385.

Thank you very much for spending time, contributing to .Net Bitcoin.

[NicolasDorier]

Thanks a lot I will look into it. Can you copy paste me your result on http://api.qbit.ninja/balances/38DdqhVuqb36jjmxTbvBkiFPXKA8EmW9Ly/summary?colored=true&at=446308

But on your instance.

[bijakatlykkex]

Thanks, attached is what I get on our deployment (Besides transactionCount, asset AJPMQpygd8V9UCAxwFYYHYXLHJ7dUkQJ5w values are different):

{
  "confirmed": {
    "transactionCount": 7385,
    "amount": 5256806695,
    "received": 1083700678704,
    "assets": [
      {
        "asset": "AXkedGbAH1XGDpAypVzA5eyjegX4FaCnvM",
        "quantity": 4758381180000,
        "received": 7381768845650000
      },
      {
        "asset": "AJPMQpygd8V9UCAxwFYYHYXLHJ7dUkQJ5w",
        "quantity": 157577740000,
        "received": 60900118990000
      },
      {
        "asset": "AWm6LaxuJgUQqJ372qeiUxXhxRWTXfpzog",
        "quantity": 110524520000,
        "received": 96150819300000
      },
      {
        "asset": "ASzmrSxhHjioWMYivoawap9yY4cxAfAMxR",
        "quantity": 39064490000,
        "received": 9666142070000
      },
      {
        "asset": "AKi5F8zPm7Vn1FhLqQhvLdoWNvWqtwEaig",
        "quantity": 50918250000,
        "received": 5191367180000
      },
      {
        "asset": "Ab8mNRBmrPJCmghHDoMsq26GP5vxm7hZpP",
        "quantity": 4939780940000,
        "received": 217494542450000
      },
      {
        "asset": "AYeENupK7A9LZ5BsQiXnp22tHHquoASsFc",
        "quantity": 51000000,
        "received": 51000000
      },
      {
        "asset": "AWFDocA327hGp8PKGcTSPYnqaUtMjN3nvv",
        "quantity": 1990092,
        "received": 5990092
      },
      {
        "asset": "AJ7wAWbH8Rd5UEHnMC6EoBP5B1gUa7U3cw",
        "quantity": 3500042,
        "received": 697240125
      },
      {
        "asset": "ANEjN5ngJJefc8RjKZHeAc37uHKzpwdsc8",
        "quantity": 2493812,
        "received": 148248191
      },
      {
        "asset": "AHG1gqienL4Gj5kcJhqm5U5LFCMtt8Vfy6",
        "quantity": 4831379,
        "received": 396447152
      },
      {
        "asset": "AbjAzsV5s5ANrGK3Kf3qbKQVQYSS3McfUH",
        "quantity": 4827598,
        "received": 440469688
      },
      {
        "asset": "AUrMzvHTYJ8oQVECCEAXUU4dBM5PvhkP9Z",
        "quantity": 481474934,
        "received": 16841423971
      },
      {
        "asset": "AJZos2d6vHxAYWLcu9xFudm1MDxK6FTk7u",
        "quantity": 4933535,
        "received": 165630765
      }
    ]
  },
  "spendable": {
    "transactionCount": 7385,
    "amount": 5256806695,
    "received": 1083700678704,
    "assets": [
      {
        "asset": "AXkedGbAH1XGDpAypVzA5eyjegX4FaCnvM",
        "quantity": 4758381180000,
        "received": 7381768845650000
      },
      {
        "asset": "AJPMQpygd8V9UCAxwFYYHYXLHJ7dUkQJ5w",
        "quantity": 157577740000,
        "received": 60900118990000
      },
      {
        "asset": "AWm6LaxuJgUQqJ372qeiUxXhxRWTXfpzog",
        "quantity": 110524520000,
        "received": 96150819300000
      },
      {
        "asset": "ASzmrSxhHjioWMYivoawap9yY4cxAfAMxR",
        "quantity": 39064490000,
        "received": 9666142070000
      },
      {
        "asset": "AKi5F8zPm7Vn1FhLqQhvLdoWNvWqtwEaig",
        "quantity": 50918250000,
        "received": 5191367180000
      },
      {
        "asset": "Ab8mNRBmrPJCmghHDoMsq26GP5vxm7hZpP",
        "quantity": 4939780940000,
        "received": 217494542450000
      },
      {
        "asset": "AYeENupK7A9LZ5BsQiXnp22tHHquoASsFc",
        "quantity": 51000000,
        "received": 51000000
      },
      {
        "asset": "AWFDocA327hGp8PKGcTSPYnqaUtMjN3nvv",
        "quantity": 1990092,
        "received": 5990092
      },
      {
        "asset": "AJ7wAWbH8Rd5UEHnMC6EoBP5B1gUa7U3cw",
        "quantity": 3500042,
        "received": 697240125
      },
      {
        "asset": "ANEjN5ngJJefc8RjKZHeAc37uHKzpwdsc8",
        "quantity": 2493812,
        "received": 148248191
      },
      {
        "asset": "AHG1gqienL4Gj5kcJhqm5U5LFCMtt8Vfy6",
        "quantity": 4831379,
        "received": 396447152
      },
      {
        "asset": "AbjAzsV5s5ANrGK3Kf3qbKQVQYSS3McfUH",
        "quantity": 4827598,
        "received": 440469688
      },
      {
        "asset": "AUrMzvHTYJ8oQVECCEAXUU4dBM5PvhkP9Z",
        "quantity": 481474934,
        "received": 16841423971
      },
      {
        "asset": "AJZos2d6vHxAYWLcu9xFudm1MDxK6FTk7u",
        "quantity": 4933535,
        "received": 165630765
      }
    ]
  },
  "immature": {
    "transactionCount": 0,
    "amount": 0,
    "received": 0,
    "assets": []
  }
}

[NicolasDorier]

Meanwhile, if you are in a rush to get back to normal, you can can clear the table chainchache. This is where the balancesummary are stored. BalanceSummaries are lazily calculated then cached upon request, so you do not loose indexed data.

I think the indexed data is fine, but there is a concurrency issue with the calculation of balance summaries, and I think knowing where it comes from. Can you try that and confirm that balancesummary is back to normal? (your first request to the summary might be a bit long)

[bijakatlykkex]

Thanks very much for the information.

After clearing the table (Actually deleting it), I did not get wrong data anymore (For some reason /summary page returns 404 sometimes in initial loads).

Actually we have two code bases using the data, one old and one up to date, we are switching to the new code. I was thinking about just reading the data, not creating the cache.

I will update the issue after more tests.

Thanks

[NicolasDorier]

@bijakatlykkex if you delete the table you need to reboot your front end so it create it again. Reading the data is complicated, if you have 7000 transactions it will take a while without some sort of cache.

[bijakatlykkex]

Thanks for clarifications.

Actually I decided to delete the table because it had so many records (I went up to about 12000 record deletions, it was continuing).

[bijakatlykkex]

It seems the problem has been eliminated in these days by clearing the cache.

As stated previously: "I think the indexed data is fine, but there is a concurrency issue with the calculation of balance summaries, and I think knowing where it comes from."

So, I am not sure if this issue needs more followup or not, and this thread should remain open or be closed.

Thanks

[NicolasDorier]

keep it open. I think the underlying cause is not fixed.

[bijakatlykkex]

Of course. Thank you.