[Tracking issue] `UNKNOWN_TRANSACTION` response problem

khorolets commented 6 months ago

Two weeks ago (let's say approximately since 2024-03-15, when the first 1.37.x resharding happened), we started receiving complaints about transactions needing to be added on the ReadRPC side.

This means that queries like EXPERIMENTAL_tx_status or tx are responded to with UNKNOWN_TRANSACION.

We haven't handled this problem entirely, so the issue is ongoing.

This is related to the transaction_details stored in ScyllaDB
If we run tx-indexer around the blocks with "missing" transactions, they appear
We figured that when a transaction is missing, we actually have it in ScyllaDB. The error we experience is borsh-deserialization error
We added the hack to attempt to validate what we have stored (and to ensure that we store it so the service works as expected) #222
[WE ARE HERE ]We monitor the logs to narrow the problem and come up with the hypothesis
...

*The next updates will go in the comments**

kobayurii commented 5 months ago

Looks like that problem has been solved.

khorolets commented 5 months ago

Looks like that problem has been solved.

Yeah, but we have no idea why it happened and what has fixed it :(

khorolets commented 3 months ago

I am reopening the issue since in the last few days we've been reported about different transactions with this particular issue:

4KArr8xHHGr6oiFz19HCcNPu8xDY9HVmCKzeMrHe1S4P (121,949,312)
6Cu1WmLrCmcfN1Hb3zyzbGKXeTExcM8ihNjgfc9iXZVx (121,974,256)
mi2a1KwagRFZhpqBNKhKaCTkHVj98J8tZnxSr1NpxSQ (113,304,726)
4xr3nbEpneszHhpSf4FFWQjXqtACPKtUugCgi8vCHaHp (104,813,671)
6WwguuQPEd3c3HQtkF88VqrTESDZ7w5hC4oDGk4Eoaqc (121,857,753)

We have reindexed transaction around those blocks to make them work.

While I can find an explanation for the two of them (more than a few months ago) the majority of the transactions are new. It means they were indexed with this issue even though we have introduced the additional checks to prevent it. It doesn't make sense to me.

We double checked borsh and can confirm it has all necessary checks and guaranteed deterministic ser/de. The only suspect is ScyllaDB. I assume there's something happening during the data distribution on the write. We do only 3-4 checks to prevent them. Perhaps it's worth to:

[ ] Add metric to watch for the error rate for transactions saving
[ ] Increase the number of checks up to 10

near / read-rpc

[Tracking issue] `UNKNOWN_TRANSACTION` response problem #224