Data is lost in block 22633808

[telezhnaya]

While investigating https://github.com/telezhnaya/indexer_balances/issues/1, I've found the issue We have the receipt in this block, but we don't have execution_outcome

 select * from blocks where block_height = 22633808; – 1605411776828600726 68v7s5fg8inLQ2VVnFvZ375bbbx1akU4kbH8ezo6RyFg
 select * from account_changes where account_changes.changed_in_block_timestamp = 1605411776828600726;
 select * from receipts where receipt_id = '6QKTEHsLihDgVrdtWYbURA28zF29Xdw5dnfWbDVWHPv4';
 select * from execution_outcomes where receipt_id = '6QKTEHsLihDgVrdtWYbURA28zF29Xdw5dnfWbDVWHPv4';

Please check why receipt_execution_outcomes is empty for block 22633808

[khorolets]

@telezhnaya could you check please, I've made near-lake-indexer to run against that block again.

[frol]

@khorolets ideally, we should have looked at what was the state of those files before rewriting and why near-lake-framework-rs passed incomplete StreamMessage down the line 🧐

[telezhnaya]

@khorolets we still have this problem

[khorolets]

OK, I've checked. I want to pass it on to you @telezhnaya I've found the ExecutionOutcome for that Receipt in the block 22635169

May I ask you to find out why the heck the ExecutionOutcome for Receipt 6QKTEHsLihDgVrdtWYbURA28zF29Xdw5dnfWbDVWHPv4 is stored as executed_in_block_hash of 68v7s5fg8inLQ2VVnFvZ375bbbx1akU4kbH8ezo6RyFg in Indexer for Explore Database if it was executed in 5WEpxdCAouEwxtbPrBuVmztX9X4cbnbfgWrnTHbvUWM8? O_o

22635169 `StreamerMessage`

``` StreamerMessage { block: BlockView { author: AccountId( "accomplice.poolv1.near", ), header: BlockHeaderView { height: 22635169, prev_height: None, epoch_id: `7Y4LN7GgyYVXMdsEuDV6dWfsML7A9sn6pwnYbY3whn5H`, next_epoch_id: `6NYcV1hbnNiW1VAWSkKDiBAvAFgDZqMC29y3vjPogoBF`, hash: `5WEpxdCAouEwxtbPrBuVmztX9X4cbnbfgWrnTHbvUWM8`, prev_hash: `BEzouvv6xJ9FWkGGgKN98bqEUde3ZKJx51KiWUZbtdQU`, prev_state_root: `CxgT68o6sQ2nDEBdrLYdWgYAC5YkiBykgVHTK6xL4J5C`, chunk_receipts_root: `98jPBYaQEMxGzDUJJHwoLoGHpi61LGfbCoebTG6LEH3v`, chunk_headers_root: `14DCgPqsMSZbLziSHbvjCZakweD2NVeeo5jUzRkKBf6c`, chunk_tx_root: `7tkzFg8RHBmMw1ncRJZCCZAizgq4rwCftTKYLce8RU8t`, outcome_root: `8zYQs2R2ENytfHVj5C2Y4CzPBqrenNPzfZ6Fx8z97vXW`, chunks_included: 1, challenges_root: `11111111111111111111111111111111`, timestamp: 1605412974411419407, timestamp_nanosec: 1605412974411419407, random_value: `RhWqciryRzrGxtg9WYeLRy1a8coh2SqkxpfSxNFerUB`, validator_proposals: [], chunk_mask: [ true, ], gas_price: 100000000, block_ordinal: None, rent_paid: 0, validator_reward: 0, total_supply: 1004251838545238298658346767796377, challenges_result: [], last_final_block: `CXHmNbYaaf5N3fATTqTuC8Whb7GfvGpqkWxyM8t8hL43`, last_ds_final_block: `BEzouvv6xJ9FWkGGgKN98bqEUde3ZKJx51KiWUZbtdQU`, next_bp_hash: `5QmKcsZ7KwUg6579QNTwi9YV4EdWjw3F7F5fB2jiLCWq`, block_merkle_root: `8f924XiPysYmjofqBWTFypbm2Z6tkdda66zXWzoXHena`, epoch_sync_data_hash: None, approvals: [ Some( ed25519:4WnTJLRAcExdnzvP93ht3YskBmzsZo8mYJksRKsnMPGWL4TjvFyLqyrHUxbgvVukF5Vhh9wpXBozqxYWpetywyjF, ), Some( ed25519:3JzkAAod6tAx2UZTj5i23JSCm9L4V8ZF1uamW5rX7fWdcrAZUMz8Sd56MA4rWoFUAup6ELZ3vpDZvH7KbKcsbTCF, ), Some( ed25519:fQWxxjq81un7Yr2YhWA97FReH5kz35rtRXMTDbUJyZsB6Y33rQNCqxEFTGVdTeZNNZ9CBwLhfXbPLDdtxw3CJpg, ), Some( ed25519:2cPEpz1a1k8Y5n41npygNcB79syJRRpqSkVrbcRANioqBMxDb6hsQBegXkkEAq3HKY2DtQ8TEggzAPbcKnL6EBEo, ), Some( ed25519:2H22imgbVWiduzQ2GTBLhh4yBuqR6wDFPe3qbA6783oc6nmfWHUxhxzAGCQ9afwZcLPktUNwfhvZ7Vv55zRUioPu, ), Some( ed25519:4fEGEdi82vMsgGpTQYGJQHC1xcwPq8eRsTaPKB9ZDyisSeW2CGb3ia6yy7FC1heUzX9hoEeUkETSmFk3Np6TjJTr, ), Some( ed25519:2tMShKrnSwPuain8agTC19A5WWG1RxPzo8wXHJsdUMMCZrUjpwK2Pphg3Fu5Fnj1riEBFjmVaWdbmJuGVvkYobEC, ), Some( ed25519:2znAzLdJDTnfdUT8p7YvL8UmLju3XnnoRULDdDdMXB4NwFvdPeE9EXEn63tsis2TkqFudScyVU849SbHDWHY8nLV, ), Some( ed25519:2KxpsQGtHw7S5qYc5KjJ8DDVBo8PmUgak5h49jjTkGazgDFkmxRXrK26cVjUYsWEcS1g4UA2x4mcAWLE9U3SgAU3, ), Some( ed25519:3vd7p9vtVjzjhgDvgZNLfBoH3CgwVo3xuNyz4WNrgCjoAzH5yrpsCKn2imr7K53uaXnV6LvEb9TXemAzQUGN94sH, ), None, None, Some( ed25519:f6QcTVUsScxbYLCxGSyAonP4FPbcDAkW9n8WNBgKnVciouMYrfWUTCqiFHrDDKrpnmqUq1Yg4aXv7WDunSfgoKg, ), Some( ed25519:4yKymfmAJQeDBFjCG4a6iE5uFzfiL7RGiWVARTwZnWKR7Qxse2VLq1RBneCrigCpPNAiKJ5tL1qv9xRe2xbcmxSM, ), Some( ed25519:4paLHjhAj5GechpoeCpeFMvkjQ7yN5SSYAQKv8xz5xGmKSWtc8DJVm53DcC9Vo4fQ265pZe8Rky94D8iadyK598F, ), Some( ed25519:3FKE1wRkiJrYyFdspspjCtgRcGvbaJnZ7duS1e1EuW5VUMo4iKKSpTDUpujwxrGcHo1ye7zsX3Ujtbj3r2nk34fT, ), Some( ed25519:2p6M7FSvWFeJj9oy83i3de3CHFMWV7yhxj8cagtpPAoXLiwHxLwBKNQ2UGDxGG5GKKxU4n5V4NdZudkFBp18hwzT, ), Some( ed25519:zrCSN3Z1Y2kyrNhuSz2FXUx6Vy5csJKG4mGRjWxi7MRXLA3dntfDExgFRGaYnvh44tjyt9crit75UVVXy8JB8YA, ), Some( ed25519:26ZkT4d8S5TAYgAFbGyV5tnvam9myPQKqVm9nReE2eap1Z9AFrUnV2AgC1sarV1q7gkbL5rSMXyM7KGJEtythCbR, ), Some( ed25519:3zijHgW7SEfhtroRq8Ghp6VpKrm612V5wkw5tkYQhFKJmDC5XaAYFrVsoh8ZUaZmAoFC2eqfgZMNgzuPZ91HfDsQ, ), None, Some( ed25519:4tRifKr36e9FRzC1ffHdq37vEF61v7PeDkdjPFFNw72Gr8YtmS2RnDhWuQyUoD4cjjmEMCgLYiczTW3nw7guiH2A, ), None, Some( ed25519:abEsVu1enV3DRPYvYkVNgVS8Gn6hdj9MuTEHBZ7fL2hgwhzf3JdfnVDKQtuLuxRTofhMncChPgd1uTVLfrm8uZ7, ), Some( ed25519:5cZwog43zRrLJjL6mhuecPTCQ71iaBSGLhMKihk7DVDS6Uekr4ZijgLqtLEjLboNGhGYPPHpTQu7QSjJUXJNSH9e, ), Some( ed25519:2NP8JyzTZWhEvysr4cbirzCeeXSeQfDjKsfj1v9L8HLgqfJ1hbjKWbB7r75ouqoos4fb3QGk6iuP82XFm6A2qhc6, ), Some( ed25519:5sMFwQzDy4o38d9NcXhsX96XFtADb2nz2qRhY1QyZ1CHHTBP8FHc4SwxpjB8ua54zufoyDU6SiKfQUE659xd8Bkn, ), Some( ed25519:5ahbyrmzKozgJ3LVHAk5NZz2EhwPeM78jU23PkXiNtcP3gRJXWw16YjXpZxE8cBVNFS5D3ikcZRWk2Kpio3bWTLb, ), Some( ed25519:Ye4C6J3zdB87a5Uyb3vkfU4z4Bnj8ukaKhrxqQ8FxDZ6NvJMguB9UHU6ix1oUW1N5ZPFYCScVtXtW6HE4m8En39, ), Some( ed25519:2X4oiBzVZSHdutDQCpqosZ88fxZP5xx4oKnocmSRRgH9wsTsWQJ8j4B2vsCnRZnwrPpdW9VnHGTnwb5bfUe9BWzY, ), Some( ed25519:3hV7RhPb19co78TZBan5zXzRFszCwTosr39DpoarqiFm8gFNYhG8x8QCGBEygbrq9QJLuajQXwT6ogbTaA7vdXV5, ), Some( ed25519:3Qg7LVWPiHpzADt22kcXAntwq5sKVbgcJDisT9WjYBa1kE9su84G7mE1VMXUnUQBwyfrYxPzbXg5w4L6p1TSDAez, ), None, Some( ed25519:2JqXppXe231PaH5rJHQimSE3DDmsQk4vKbu2SwuabxUpxpiGHFa8puBvGHcE3zMD46f13j8y4hBT5bZGkZtBZmSm, ), Some( ed25519:TiZkve8kHT8itVfpdPhvnYka3cdX7c6PrK8TuNfMEENtSSXAyAQRA9XtcBKYdEroKMqqfNs4BKds6FvDxxXEj5j, ), None, None, Some( ed25519:4xWFYqiqJJ3fZbHE3NZPnRkSK2wzekH9Bhqc3GvQVk3fKjnjH6jvdjBYcUAGDLaubao3XHwS2Se1KGcNn8pubdDC, ), None, Some( ed25519:DjFm9UNQPQPaAAGXkhxr5VjCzfqEH3iUMeNN1pwYhrHwhRWy1rFqKaxxYKBWqSoJBiSnv4evvrWdWND8osFdgST, ), Some( ed25519:3VmECv2HQrX8Zp2mATVcaDuXgd2bhW9hU97NnACPPjJr63FoqLhL3mVC9JqzeLxaCRApK4ayKJbAMYxjMNmyNena, ), Some( ed25519:2tBtz1kSag9vMrromqdNTj1sU3mC6RFAWdsmfrvJiBLLHnUr7NGssfWkZ3Pb2yEArguc3hgPXqKtozzp68PtiST4, ), None, Some( ed25519:TPKthq5gv5Uy32C3YNEpXXahUNxN4AVW6WKXU1qC2iJrVMWfYrDcnc5fknRY59MRPb6ZteRaTqhAVQr2eK6YUK3, ), Some( ed25519:2VbPs89GRUous4cSqbaeQqnTn2JWN4TuvuTiZatVWzvwPC9rNu8GjtXBi7Qz6zyvnGUaewMtWF7qu6TFJCYN9d4N, ), Some( ed25519:4crAGqzng3GzxF4kVQzdDbzbsB3PfrvXXt4JNoQyyyjJsHiALkCMxV2yHe7u5Ch5oBr5xayQL9pSGW6juPa73MQv, ), Some( ed25519:5CEr16Arxua7MCJjf17JsEWDGuv3WJPApRkCYs1tbouwZJSnE3JEsoNFWcj1fNykqyfNtvwbu3ztJ76C6uUAaEK9, ), Some( ed25519:43UwrxBfn1H9GhayskoSnUEd5m8viCUixiU3cWQx8EgDA3FnbpbbDV67uJDkBHyKJK2FwfqBFoQE3w4at8CL2TJE, ), Some( ed25519:52uPyxbceVRMNme1nacxUtqvEUNpAjacKRBUGJ7vh4irZ93fmdqj9k6vRAVA5hAvkZx8eG6VXfy2XcvKBF8hneiY, ), Some( ed25519:3ZT1bK4gsDaavUyB9ozjqDjvu6gUtoNWCRXTWojqAWsBudaeuooL9EcXGEDAxQ6uv4jekTzZy8oCeNqH2GLBvQDH, ), Some( ed25519:4eBCQceLonuJjmDG5cjDNds2bBo65DFRzETi7iN4t4QDy266j5QN2qoUbHhzRdwJtZYF2piSP74kmZdbTaBqxg2w, ), Some( ed25519:4YA5HYQQeutE6p2G2PVFQd6ftC1CRCrgaFxAvQrxNt4N7HLik62WG5a9zp9hGEJ7NzoRThM6s9DYG4KYgTegwwBf, ), Some( ed25519:4tXcpYYPZ6tVccenaGC29yk9aoRTknhwGwEB8yrVcb4qJK9S6FRKu8NhCfvGptyozQe4fYiY3vBAFGf4zvnbCSkH, ), Some( ed25519:4s5BPRFRKyvGZhkAa7RowppK7s8r3YguN2XLicGxk3PrpfDTurFZxAJursEstD6QwBHFZGbNED2KxRM5JtbcHdDM, ), ], signature: ed25519:28F72kjLTn7B8hZqfF214y77uLunq6jWw7v7FbjDFoJ3WXrW9QpWwanaFNyKUZPf6YwvUcXJ9hb1hN7WbsUFKt93, latest_protocol_version: 36, }, chunks: [ ChunkHeaderView { chunk_hash: `HmDUWTVtHLf9uuuxiasU2ZCTwxEapGqVky9A26kmeTJT`, prev_block_hash: `BEzouvv6xJ9FWkGGgKN98bqEUde3ZKJx51KiWUZbtdQU`, outcome_root: `6MgWFeiCPzKRJPqF8FPMzkexXb1wGDLbFdmmZe5eKu6L`, prev_state_root: `41mGs9fEarHH8t6rgWSSatZ7xPvzhzfwwb91XyhMJMEP`, encoded_merkle_root: `Dg6eoP833WPmAtMpisU1oVDZWQDoV9cCTirz9Z8nKpkp`, encoded_length: 344, height_created: 22635169, height_included: 22635169, shard_id: 0, gas_used: 105525848812409, gas_limit: 1000000000000000, rent_paid: 0, validator_reward: 0, balance_burnt: 7477318296645900000000, outgoing_receipts_root: `76BH1tHiQ8UyBQMHY2xtxPfUby38Uf5UkePw4ZhGp2zy`, tx_root: `11111111111111111111111111111111`, validator_proposals: [], signature: ed25519:5zYQVo8oafyArE1xAgMBuf79D5XcnoaxZxc9TuhztvrG4hCuFnTmrzzTDWVxtWkgyGWajAqr2fSRYYpQvX89hRaL, }, ], }, shards: [ IndexerShard { shard_id: 0, chunk: Some( IndexerChunkView { author: AccountId( "accomplice.poolv1.near", ), header: ChunkHeaderView { chunk_hash: `HmDUWTVtHLf9uuuxiasU2ZCTwxEapGqVky9A26kmeTJT`, prev_block_hash: `BEzouvv6xJ9FWkGGgKN98bqEUde3ZKJx51KiWUZbtdQU`, outcome_root: `6MgWFeiCPzKRJPqF8FPMzkexXb1wGDLbFdmmZe5eKu6L`, prev_state_root: `41mGs9fEarHH8t6rgWSSatZ7xPvzhzfwwb91XyhMJMEP`, encoded_merkle_root: `Dg6eoP833WPmAtMpisU1oVDZWQDoV9cCTirz9Z8nKpkp`, encoded_length: 344, height_created: 22635169, height_included: 22635169, shard_id: 0, gas_used: 105525848812409, gas_limit: 1000000000000000, rent_paid: 0, validator_reward: 0, balance_burnt: 7477318296645900000000, outgoing_receipts_root: `76BH1tHiQ8UyBQMHY2xtxPfUby38Uf5UkePw4ZhGp2zy`, tx_root: `11111111111111111111111111111111`, validator_proposals: [], signature: ed25519:5zYQVo8oafyArE1xAgMBuf79D5XcnoaxZxc9TuhztvrG4hCuFnTmrzzTDWVxtWkgyGWajAqr2fSRYYpQvX89hRaL, }, transactions: [], receipts: [ ReceiptView { predecessor_id: AccountId( "system", ), receiver_id: AccountId( "aspaceodyssey2001.near", ), receipt_id: `6QKTEHsLihDgVrdtWYbURA28zF29Xdw5dnfWbDVWHPv4`, receipt: Action { signer_id: AccountId( "system", ), signer_public_key: ed25519:11111111111111111111111111111111, gas_price: 0, output_data_receivers: [], input_data_ids: [], actions: [ Transfer { deposit: 3690308308890300000000000000, }, ], }, }, ReceiptView { predecessor_id: AccountId( "system", ), receiver_id: AccountId( "relay20201112.bridge.near", ), receipt_id: `7hvy4PwoSxWM3BsesEJCGUUsNnncruYCqbmpKtAG7qKu`, receipt: Action { signer_id: AccountId( "relay20201112.bridge.near", ), signer_public_key: ed25519:CmAGytdaDYfmrB85ArFfLdRepDTSTPdEnp8trUdh8U3L, gas_price: 0, output_data_receivers: [], input_data_ids: [], actions: [ Transfer { deposit: 178576843726431094307760, }, ], }, }, ], }, ), receipt_execution_outcomes: [ IndexerExecutionOutcomeWithReceipt { execution_outcome: ExecutionOutcomeWithIdView { proof: [ MerklePathItem { hash: `5cA8GQnRkTdvvUf1KBZxJpH76XQkYbgaVDHudXKC5bn6`, direction: Right, }, ], block_hash: `5WEpxdCAouEwxtbPrBuVmztX9X4cbnbfgWrnTHbvUWM8`, id: `6QKTEHsLihDgVrdtWYbURA28zF29Xdw5dnfWbDVWHPv4`, outcome: ExecutionOutcomeView { logs: [], receipt_ids: [], gas_burnt: 0, tokens_burnt: 0, executor_id: AccountId( "aspaceodyssey2001.near", ), status: SuccessValue(``), metadata: ExecutionMetadataView { version: 1, gas_profile: None, }, }, }, receipt: ReceiptView { predecessor_id: AccountId( "system", ), receiver_id: AccountId( "aspaceodyssey2001.near", ), receipt_id: `6QKTEHsLihDgVrdtWYbURA28zF29Xdw5dnfWbDVWHPv4`, receipt: Action { signer_id: AccountId( "system", ), signer_public_key: ed25519:11111111111111111111111111111111, gas_price: 0, output_data_receivers: [], input_data_ids: [], actions: [ Transfer { deposit: 3087754972079500200000000, }, ], }, }, }, IndexerExecutionOutcomeWithReceipt { execution_outcome: ExecutionOutcomeWithIdView { proof: [ MerklePathItem { hash: `9ZJuzK1o7ob3tPs67z7iUk2fX99A6hVoX6T29q8A29t5`, direction: Left, }, ], block_hash: `5WEpxdCAouEwxtbPrBuVmztX9X4cbnbfgWrnTHbvUWM8`, id: `7hvy4PwoSxWM3BsesEJCGUUsNnncruYCqbmpKtAG7qKu`, outcome: ExecutionOutcomeView { logs: [], receipt_ids: [], gas_burnt: 0, tokens_burnt: 0, executor_id: AccountId( "relay20201112.bridge.near", ), status: SuccessValue(``), metadata: ExecutionMetadataView { version: 1, gas_profile: None, }, }, }, receipt: ReceiptView { predecessor_id: AccountId( "system", ), receiver_id: AccountId( "relay20201112.bridge.near", ), receipt_id: `7hvy4PwoSxWM3BsesEJCGUUsNnncruYCqbmpKtAG7qKu`, receipt: Action { signer_id: AccountId( "relay20201112.bridge.near", ), signer_public_key: ed25519:CmAGytdaDYfmrB85ArFfLdRepDTSTPdEnp8trUdh8U3L, gas_price: 0, output_data_receivers: [], input_data_ids: [], actions: [ Transfer { deposit: 178576843726431094307760, }, ], }, }, }, ], state_changes: [ StateChangeWithCauseView { cause: ReceiptProcessing { receipt_hash: `6QKTEHsLihDgVrdtWYbURA28zF29Xdw5dnfWbDVWHPv4`, }, value: AccountUpdate { account_id: AccountId( "aspaceodyssey2001.near", ), account: AccountView { amount: 3695396021799638242400000000, locked: 0, code_hash: `11111111111111111111111111111111`, storage_usage: 333, storage_paid_at: 0, }, }, }, StateChangeWithCauseView { cause: ReceiptProcessing { receipt_hash: `7hvy4PwoSxWM3BsesEJCGUUsNnncruYCqbmpKtAG7qKu`, }, value: AccountUpdate { account_id: AccountId( "relay20201112.bridge.near", ), account: AccountView { amount: 439119056546033518936080338, locked: 0, code_hash: `11111111111111111111111111111111`, storage_usage: 182, storage_paid_at: 0, }, }, }, ], }, ], } ```

[telezhnaya]

Are you sure it's for me? We just listen to the blockchain, we don't create this data from nowhere Maybe we need to summon someone from the core team?

[khorolets]

@telezhnaya My main concern is why we have stored that piece of data in indexer for explorer db the way we have. Let's not bother protocol team unless we have any evidence.

I am going to check the archival indexer data to narrow down the questions who's mistake is that and will update you.

[khorolets]

My checks on the archival node with Indexer Framework prove that ExecutionOutcome for that receipt is included in block 22635169.

22633808.log 22635169.log

Since the data reflected by Lake Indexer is correct I am closing this issue and I am going to open an issue on Indexer for Explorer project to figure out the reason the record in the database is incorrect.

[telezhnaya]

I feel it's still an issue for this project. Indexer balances, where I've found this issue initially, has no dependency to indexer-for-explorer. I mentioned indexer-for-explorer here just because it's the convenient way to look at the data.

Inside lake, we have json with block 22633808, where one field explicitly says "hey, we have the other useful data in the other field", and the second field is empty

[khorolets]

I disagree. Lake indexer doesn't say we have execution outcome for this receipt in that block. Have a look at my comment with attached logs from Indexer Framework. The block you refer to just doesn't contain ExecutionOutcome.

[telezhnaya]

At json about block 22633808, we see this

StateChangeWithCauseView {
                    cause: ReceiptProcessing {
                        receipt_hash: `6QKTEHsLihDgVrdtWYbURA28zF29Xdw5dnfWbDVWHPv4`,
                    },
                    value: AccountUpdate {
                        account_id: AccountId(
                            "aspaceodyssey2001.near",
                        ),
                        account: AccountView {
                            amount: 5087754972079500200000000,
                            locked: 0,
                            code_hash: `11111111111111111111111111111111`,
                            storage_usage: 182,
                            storage_paid_at: 0,
                        },
                    },
                },

The fee was taken. As I understand the blockchain logic, it means that the receipt should be executed in the same block.

[telezhnaya]

Wow, it looks like it's not the fee. I'm naive and I still believe indexer-for-explorer data According to that, 3.08 NEAR came to aspaceodyssey2001.near at block 22633808. So I'm totally lost. Do you have any idea why may we have such balance change? I need to handle such situations at https://github.com/near/near-indexer-balances The more confusing for me is why such a situation is so unique

[telezhnaya]

For future me: 3.08 is found by writing the query (the timestamps picked up randomly)

select * from account_changes 
where account_changes.affected_account_id = 'aspaceodyssey2001.near' and
account_changes.changed_in_block_timestamp > 1604411776828600726
and account_changes.changed_in_block_timestamp <= 1605411776828600726;

plus some calc (5087754972079500200000000 - 2000000000000000000000000) / 10**24

[frol]

@khorolets @telezhnaya This was an instance of duplicate transaction execution. Blocks #22633807 and #22635168 contain transaction J4CZZQrZK6kYPVLkrdbTEpcqhUNZiRxktbMzHviqeGgf. The implicit account was initially created by a transfer call at block #22633373, then it was deleted with a transaction at block #22633807 (unfortunately, Indexer for Explorer Postgres does not reflect it, so you will have to make a chunks RPC call, see below), and then the account was re-created at block #22634878 with a new transfer, and then the previous account deletion transaction got re-played (due to the nonce tracking issue; after the account was re-created, the nonce got reset to zero and now exactly the same transaction became valid again as its nonce is greater than the current one [zero]) and was executed again at block #22635168, which deleted the implicit account again and transferred all the tokens to aspaceodyssey2001.near (I can only hope that the tokens anyway were designated to be transferred to that account as otherwise it would lead to someone transferring more tokens than intended)

http post https://archival-rpc.mainnet.near.org jsonrpc=2.0 method=chunk params:='{"block_id": 22633807, "shard_id": 0}' id=dontcare

http post https://archival-rpc.mainnet.near.org jsonrpc=2.0 method=chunk params:='{"block_id": 22635168, "shard_id": 0}' id=dontcare

[khorolets]

then it was deleted with a transaction at block #22633807 (unfortunately, Indexer for Explorer Postgres does not reflect it, so you will have to make a chunks RPC call, see below)

Here are the transaction and the receipt you're referring to, I believe: https://explorer.near.org/blocks/CVMvbRjcTJ4cbJsn2fNqWQgXmUffWsJs2pg2chRRnSDf

Thank you @frol for digging into it. Your input is very useful!

We're trying to understand the data flow behind these actions. Talking about blocks #22633807 and #22633808. We see the:

1. Transaction with DeleteAccount in #22633807
2. Transaction is converted into Receipt FA9zcm7WkWxdjkub7WFiKkQdnnQrcEmBht94VFzXfkm1
3. The Receipt FA9zcm7WkWxdjkub7WFiKkQdnnQrcEmBht94VFzXfkm1 is immediately executed in the same block. And it contains status: SuccessValue(``) and receipt_ids: [ '6QKTEHsLihDgVrdtWYbURA28zF29Xdw5dnfWbDVWHPv4' ]
4. This Receipt 6QKTEHsLihDgVrdtWYbURA28zF29Xdw5dnfWbDVWHPv4 is included in the very next block #22633808.

But receipt_execution_outcomes: [] is empty for the block #22633808, though StateChangeWithCauseView with cause ReceiptProcessing for the Receipt 6QKTEHsLihDgVrdtWYbURA28zF29Xdw5dnfWbDVWHPv4 is present.

At the same time, Indexer for Explorer says the Receipt 6QKTEHsLihDgVrdtWYbURA28zF29Xdw5dnfWbDVWHPv4 is executed in block #22633808. I've managed to find the ExecutionOutcome with Lake and Indexer Frameworks in block #22635169.

So my main assumption is that we might have introduced some changes to the Indexer Framework that we are no longer able to collect the Receipt 6QKTEHsLihDgVrdtWYbURA28zF29Xdw5dnfWbDVWHPv4 for block #22633808 while it should be there (it used to be there where we were indexing that block with Indexer for Explorer).

What do you think?

[frol]

So my main assumption is that we might have introduced some changes to the Indexer Framework that we are no longer able to collect the Receipt 6QKTEHsLihDgVrdtWYbURA28zF29Xdw5dnfWbDVWHPv4 for block #22633808 while it should be there (it used to be there where we were indexing that block with Indexer for Explorer).

This is an unpleasant mystery. I have no better guesses, but I don't see how that could have happened.