search

stampchain-io / btc_stamps

Bitcoin Stamps Indexer
https://stampchain.io
GNU Affero General Public License v3.0
9 stars 2 forks source link

1.8.5 txlist_hash mismatches #329

Closed reinamora137 closed 1 day ago

reinamora137 commented 5 days ago

On some nodes, seeing a mismatch here:

indexer_1 | index_core.check.ConsensusError: Incorrect txlist_hash consensus hash for block 835000. Calculated a9640b735b40535b3c3aed5bf83692c3befb12db6674ecf58b9022ed5cac8973 but expected 7cb2ce944ff66f7d016ebbcac53412c080bd2b384e313d98e1578c9080c43e31 indexer_1 | 2024/06/28 03:29:21 Command exited with error: exit status 1

second independent index, same mismatch (openstamp)

index_core.check.ConsensusError: Incorrect txlist_hash consensus hash for block 835000. Calculated a9640b735b40535b3c3aed5bf83692c3befb12db6674ecf58b9022ed5cac8973 but expected 7cb2ce944ff66f7d016ebbcac53412c080bd2b384e313d98e1578c9080c43e31

attached is a db dump of the stampchain production (and matching dev) database for comparison and debug to isolate the exact block for mismatch output.csv

To Note: txlist_hash is a hash of the string derived from:

valid_stamps_in_block (List[ValidStamp])

class ValidStamp(TypedDict): stamp_number: int tx_hash: str cpid: str is_btc_stamp: bool is_valid_base64: bool stamp_base64: str is_cursed: bool src_data: str

TODO: Find the exact block of mismatch, and compare the valid_stamps_in_block values

This is the function that takes the valid_stamps_in_block, converts it to a string and computes the hash

https://github.com/stampchain-io/btc_stamps/blob/6b89dc40ab26c3f4845b5b385a247a2b5e81f0e4/indexer/src/index_core/blocks.py#L560

It's possible we need to sort the string (by stamp_number?) that derives the txlist_content if there is some multithreading going on somewhere?

For block 835000 this is the txlist_content string @ stampchain which is used to calculate the hash.

[{'stamp_number': 376734, 'tx_hash': 'ae9c3c44d660e246a9c819ecb78a9e1cdde073842b6eefabd2a7320a1d3173c5', 'cpid': 'JhIJBhMAdJLyk25O5gNB', 'is_btc_stamp': True, 'is_valid_base64': True, 'stamp_base64': '', 'is_cursed': False, 'src_data': ''}, 
{'stamp_number': 376735, 'tx_hash': '9e712fa154ba6927e15abf4f0b470e5daa972fb3a82428b5d578df80414e3fdf', 'cpid': 'dyxXuenW63hDDlupG8ni', 'is_btc_stamp': True, 'is_valid_base64': True, 'stamp_base64': '', 'is_cursed': False, 'src_data': ''}, 
{'stamp_number': 376736, 'tx_hash': '859065ffb252027a6cccb067ed34f66dbeef6ce584548de5ff8bbad0702023e1', 'cpid': 'KYNfbfJX3nhDISa35h8a', 'is_btc_stamp': True, 'is_valid_base64': True, 'stamp_base64': '', 'is_cursed': False, 'src_data': ''}]

branch created to test the sorting theory: https://github.com/stampchain-io/btc_stamps/tree/reinamora/txlist_content

n4kashu commented 5 days ago

Compared the CSV to my local (referenced in https://github.com/stampchain-io/btc_stamps/issues/330 ) DB is up 834999 and all hashes do match 1:1 output2.csv

btcopenstamp commented 5 days ago

DB is up 834999 and txlist_hashe is inconsistent starting from block height 834608.

For block 834608 this is the txlist_content string which is used to calculate the hash.

[{'stamp_number': 368478, 'tx_hash': '6771c0fa5939e65e534bfc98769a1f419931ef0cca59b3a8b3c7d88f5db9a24a', 'cpid': 'QIpQqX2GwkcqbIxeB74V', 'is_btc_stamp': True, 'is_valid_base64': True, 'stamp_base64': '', 'is_cursed': False, 'src_data': ''}, 
{'stamp_number': 368479, 'tx_hash': '69762c76d3c7d79ca9d7c08fd80e15f679b477b27843eea9f5e919e052a2b0ca', 'cpid': 'A2195739247552813300', 'is_btc_stamp': True, 'is_valid_base64': True, 'stamp_base64': 'IWZ1bmN0aW9uKHQpe3ZhciBlPWZ1bmN0aW9uKCl7cmV0dXJuIG5ldyBlLmZuLmluaXR9O2UuZm49ZS5wcm90b3R5cGU9e2NvbnN0cnVjdG9yOmUsaW5pdDpmdW5jdGlvbigpe3JldHVybiB0aGlzfSxhZGRHWjphc3luYyBmdW5jdGlvbih0KXtsZXQgZT1uZXcgRGVjb21wcmVzc2lvblN0cmVhbSgiZ3ppcCIpLG49KGF3YWl0IGZldGNoKHQpKS5ib2R5LnBpcGVUaHJvdWdoKGUpO3JldHVybiBjb25zb2xlLmxvZyh0LCItIGd6IiksYXdhaXQoYXdhaXQgbmV3IFJlc3BvbnNlKG4pKS50ZXh0KCl9LGFkZFRYVDphc3luYyBmdW5jdGlvbih0KXtsZXQgZT1hd2FpdCBmZXRjaCh0KTtyZXR1cm4gY29uc29sZS5sb2codCwiLSB0eHQiKSxhd2FpdCBlLnRleHQoKX0sZXJyb3I6ZnVuY3Rpb24odCxlKXtkb2N1bWVudC53cml0ZSh0KSxjb25zb2xlLmxvZyhlKX0sYWRkQzphc3luYyBmdW5jdGlvbih0LGUpe3ZhciBuPSIiO2ZvcihsZXQgaT0wO2k8dC5sZW5ndGg7aSsrKXtsZXQgYT0iL3MvIjsyPT1lJiYoYT0iZGF0YTphcHBsaWNhdGlvbi9vY3RldC1zdHJlYW07YmFzZTY0LCIpO2xldCBvPTA7ZG97dHJ5e3N3aXRjaChlKXtjYXNlIDI6ZGVmYXVsdDpuKz1hd2FpdCB0aGlzLmFkZEdaKGErdFtpXSk7YnJlYWs7Y2FzZSAxOm4rPWF3YWl0IHRoaXMuYWRkVFhUKGErdFtpXSl9bz01fWNhdGNoKGUpe2NvbnNvbGUubG9nKCJGYWlsZWQgdG8gbG9hZCFcblJldHJ5aW5nIHJlc291cmNlOiAiK2ErdFtpXSsiIC0gdHJ5ICIrKG8rMSkpLG8rK319d2hpbGUobzwzKTtpZigzPT09byl0aHJvdyBuZXcgRXJyb3IoIkZhaWxlZCB0byBsb2FkIHJlc291cmNlOiAiK2ErdFtpXSl9cmV0dXJuIG59LGFkZDpmdW5jdGlvbih0LGUpe2xldCBuPWRvY3VtZW50LmNyZWF0ZUVsZW1lbnQoZSk7bi5pbm5lckhUTUw9dCxkb2N1bWVudC5ib2R5LmFwcGVuZENoaWxkKG4pfSxqczphc3luYyBmdW5jdGlvbih0LGUpe3RoaXMuYWRkKGF3YWl0IHRoaXMuYWRkQyh0LGUpLCJzY3JpcHQiKX0sY3NzOmFzeW5jIGZ1bmN0aW9uKHQsZSl7dGhpcy5hZGQoYXdhaXQgdGhpcy5hZGRDKHQsZSksInN0eWxlIil9LGh0bWw6YXN5bmMgZnVuY3Rpb24odCxlKXt0aGlzLmFkZChhd2FpdCB0aGlzLmFkZEModCxlKSwiZGl2Iil9fSxlLmZuLmluaXQucHJvdG90eXBlPWUuZm4sInVuZGVmaW5lZCIhPXR5cGVvZiBkZWZpbmUmJmRlZmluZSBpbnN0YW5jZW9mIEZ1bmN0aW9uJiZudWxsIT1kZWZpbmUuYW1kP2RlZmluZSgoZnVuY3Rpb24oKXtyZXR1cm4gZX0pKToidW5kZWZpbmVkIiE9dHlwZW9mIG1vZHVsZSYmbW9kdWxlLmV4cG9ydHM/bW9kdWxlLmV4cG9ydHM9ZTpudWxsIT10JiYodC5BcHBlbmQ9ZSl9KHRoaXN8fHt9KSxhcHBlbmRDQigpOw==', 'is_cursed': False, 'src_data': ''},
{'stamp_number': 368480, 'tx_hash': '06014a5767b895d393c97d2070ca3c7ee3615d5fa568d8f5ca5cdae02fb27639', 'cpid': 'AFpRko5DQceWCLhDbj2p', 'is_btc_stamp': True, 'is_valid_base64': True, 'stamp_base64': '', 'is_cursed': False, 'src_data': ''}, 
{'stamp_number': 368481, 'tx_hash': '3e95ffbe548b6c849e9b2c805fb985a7522d50efb8446094140a1c10981dc1e5', 'cpid': '39GvcUpdE9zypHfUy2Ry', 'is_btc_stamp': True, 'is_valid_base64': True, 'stamp_base64': '', 'is_cursed': False, 'src_data': ''}, 
{'stamp_number': 368482, 'tx_hash': 'ffb6b1dd56d6c844d29f749cb97bf8c901aaf104d49285bdc16cac4db3112ee9', 'cpid': 'SWTIVcuXEiqoxwjBUt2J', 'is_btc_stamp': True, 'is_valid_base64': True, 'stamp_base64': '', 'is_cursed': False, 'src_data': ''}]

My server info: aws ec2 c6a.xlarge ubuntu-jammy-22.04-amd64-server

reinamora137 commented 5 days ago

DB is up 834999 and txlist_hashe is inconsistent starting from block height 834608.

For block 834608 this is the txlist_content string which is used to calculate the hash.

[{'stamp_number': 368478, 'tx_hash': '6771c0fa5939e65e534bfc98769a1f419931ef0cca59b3a8b3c7d88f5db9a24a', 'cpid': 'QIpQqX2GwkcqbIxeB74V', 'is_btc_stamp': True, 'is_valid_base64': True, 'stamp_base64': '', 'is_cursed': False, 'src_data': ''}, 
{'stamp_number': 368479, 'tx_hash': '69762c76d3c7d79ca9d7c08fd80e15f679b477b27843eea9f5e919e052a2b0ca', 'cpid': 'A2195739247552813300', 'is_btc_stamp': True, 'is_valid_base64': True, 'stamp_base64': 'IWZ1bmN0aW9uKHQpe3ZhciBlPWZ1bmN0aW9uKCl7cmV0dXJuIG5ldyBlLmZuLmluaXR9O2UuZm49ZS5wcm90b3R5cGU9e2NvbnN0cnVjdG9yOmUsaW5pdDpmdW5jdGlvbigpe3JldHVybiB0aGlzfSxhZGRHWjphc3luYyBmdW5jdGlvbih0KXtsZXQgZT1uZXcgRGVjb21wcmVzc2lvblN0cmVhbSgiZ3ppcCIpLG49KGF3YWl0IGZldGNoKHQpKS5ib2R5LnBpcGVUaHJvdWdoKGUpO3JldHVybiBjb25zb2xlLmxvZyh0LCItIGd6IiksYXdhaXQoYXdhaXQgbmV3IFJlc3BvbnNlKG4pKS50ZXh0KCl9LGFkZFRYVDphc3luYyBmdW5jdGlvbih0KXtsZXQgZT1hd2FpdCBmZXRjaCh0KTtyZXR1cm4gY29uc29sZS5sb2codCwiLSB0eHQiKSxhd2FpdCBlLnRleHQoKX0sZXJyb3I6ZnVuY3Rpb24odCxlKXtkb2N1bWVudC53cml0ZSh0KSxjb25zb2xlLmxvZyhlKX0sYWRkQzphc3luYyBmdW5jdGlvbih0LGUpe3ZhciBuPSIiO2ZvcihsZXQgaT0wO2k8dC5sZW5ndGg7aSsrKXtsZXQgYT0iL3MvIjsyPT1lJiYoYT0iZGF0YTphcHBsaWNhdGlvbi9vY3RldC1zdHJlYW07YmFzZTY0LCIpO2xldCBvPTA7ZG97dHJ5e3N3aXRjaChlKXtjYXNlIDI6ZGVmYXVsdDpuKz1hd2FpdCB0aGlzLmFkZEdaKGErdFtpXSk7YnJlYWs7Y2FzZSAxOm4rPWF3YWl0IHRoaXMuYWRkVFhUKGErdFtpXSl9bz01fWNhdGNoKGUpe2NvbnNvbGUubG9nKCJGYWlsZWQgdG8gbG9hZCFcblJldHJ5aW5nIHJlc291cmNlOiAiK2ErdFtpXSsiIC0gdHJ5ICIrKG8rMSkpLG8rK319d2hpbGUobzwzKTtpZigzPT09byl0aHJvdyBuZXcgRXJyb3IoIkZhaWxlZCB0byBsb2FkIHJlc291cmNlOiAiK2ErdFtpXSl9cmV0dXJuIG59LGFkZDpmdW5jdGlvbih0LGUpe2xldCBuPWRvY3VtZW50LmNyZWF0ZUVsZW1lbnQoZSk7bi5pbm5lckhUTUw9dCxkb2N1bWVudC5ib2R5LmFwcGVuZENoaWxkKG4pfSxqczphc3luYyBmdW5jdGlvbih0LGUpe3RoaXMuYWRkKGF3YWl0IHRoaXMuYWRkQyh0LGUpLCJzY3JpcHQiKX0sY3NzOmFzeW5jIGZ1bmN0aW9uKHQsZSl7dGhpcy5hZGQoYXdhaXQgdGhpcy5hZGRDKHQsZSksInN0eWxlIil9LGh0bWw6YXN5bmMgZnVuY3Rpb24odCxlKXt0aGlzLmFkZChhd2FpdCB0aGlzLmFkZEModCxlKSwiZGl2Iil9fSxlLmZuLmluaXQucHJvdG90eXBlPWUuZm4sInVuZGVmaW5lZCIhPXR5cGVvZiBkZWZpbmUmJmRlZmluZSBpbnN0YW5jZW9mIEZ1bmN0aW9uJiZudWxsIT1kZWZpbmUuYW1kP2RlZmluZSgoZnVuY3Rpb24oKXtyZXR1cm4gZX0pKToidW5kZWZpbmVkIiE9dHlwZW9mIG1vZHVsZSYmbW9kdWxlLmV4cG9ydHM/bW9kdWxlLmV4cG9ydHM9ZTpudWxsIT10JiYodC5BcHBlbmQ9ZSl9KHRoaXN8fHt9KSxhcHBlbmRDQigpOw==', 'is_cursed': False, 'src_data': ''},
{'stamp_number': 368480, 'tx_hash': '06014a5767b895d393c97d2070ca3c7ee3615d5fa568d8f5ca5cdae02fb27639', 'cpid': 'AFpRko5DQceWCLhDbj2p', 'is_btc_stamp': True, 'is_valid_base64': True, 'stamp_base64': '', 'is_cursed': False, 'src_data': ''}, 
{'stamp_number': 368481, 'tx_hash': '3e95ffbe548b6c849e9b2c805fb985a7522d50efb8446094140a1c10981dc1e5', 'cpid': '39GvcUpdE9zypHfUy2Ry', 'is_btc_stamp': True, 'is_valid_base64': True, 'stamp_base64': '', 'is_cursed': False, 'src_data': ''}, 
{'stamp_number': 368482, 'tx_hash': 'ffb6b1dd56d6c844d29f749cb97bf8c901aaf104d49285bdc16cac4db3112ee9', 'cpid': 'SWTIVcuXEiqoxwjBUt2J', 'is_btc_stamp': True, 'is_valid_base64': True, 'stamp_base64': '', 'is_cursed': False, 'src_data': ''}]

My server info: aws ec2 c6a.xlarge ubuntu-jammy-22.04-amd64-server

It appears this tx is showing as cursed in the production data:

-1779 | 69762c76d3c7d79ca9d7c08fd80e15f679b477b27843eea9f5e919e052a2b0ca | A2195739247552813300 |

It also appears this was a cursed stamp in version 1.8.0 as well (due to it being a JS file)

n4kashu commented 5 days ago

image

reinamora137 commented 4 days ago

This stamp should indeed be labeled as cursed (negative number) since it is a JS file:

https://stampchain.io/stamps/69762c76d3c7d79ca9d7c08fd80e15f679b477b27843eea9f5e919e052a2b0ca.js

Now we need to determine why some runs of the indexer are flagging this as valid.

reinamora137 commented 4 days ago

I have made this modification to ensure consistent regex behavior across python versions when detecting JS

https://github.com/stampchain-io/btc_stamps/commit/17f599094c108d6a8019f7a9afad1db5c0f95a82

will need to reindex from block 834608. the following can be used to clear the tables and then re-execute poetry run indexer

SET @block_index = 834608;
SET FOREIGN_KEY_CHECKS = 0;
DELETE FROM transactions WHERE block_index >= @block_index;
DELETE FROM blocks WHERE block_index >= @block_index;
DELETE FROM StampTableV4 WHERE block_index >= @block_index;
DELETE FROM SRC20 WHERE block_index >= @block_index;
DELETE FROM SRC20Valid WHERE block_index >= @block_index;
DELETE FROM balances;
SET FOREIGN_KEY_CHECKS = 1;