Closed rdlrt closed 1 month ago
Indeed, that's a bug that probablly affects all Byron consumed entries. We can prepare a tag that fixes it.
Indeed, that's a bug that probablly affects all Byron consumed entries. We can prepare a tag that fixes it.
Thanks for the quick turnaround and fix , and getting a tag or minor release - I am happy with the change and we can resync from genesis (as 1-time resync and then snapshot to replicate to others is simple enough).
Just noting from tag/release notes pov that this is breaking and not [atleast yet] handled by migration, mentioning this since I have seen responses to queries that dbsync does not require resync, while there have been re-sync required between the tags - atleast if not using config options. I think we had one at 13.2.x , and ever since - have stuck to re-synching from genesis, unless the delta is quite small or new data is inserted w/o changes to existing columns.
This has been fixed and tag 13.4.0.2 doesn't have the problem. It require a resync to fix older data as no migration is provided
In general the numbering of the releases follow changes to the default schema. If an upgrade caused wrong old values, that was probably due to a bug in the migration process.
Tested the version of fix with migration: https://github.com/IntersectMBO/cardano-db-sync/pull/1827 So far haven't noticed any issues :green_circle:
This fix is now 13.4.1.0. Some edge cases of the fix:
Fixed 5201000/5174749 entries
, since more entries are updating than necessary.I guess this is the warning message which initially I wanted to report but then realized that it is expected in case of stopping in the middle of fixing and starting service again:
[db-sync-node:Info:95] [2024-08-27 10:09:33.79 UTC] Fixed 2003511/2065090 entries
[db-sync-node:Warning:95] [2024-08-27 10:10:33.85 UTC] While fixing block ByronBlock {byronBlockRaw = ABOBBlock (ABlock {blockHeader = AHeader {aHeaderProtocolMagicId = Annotated {unAnnotated = ProtocolMagicId {unProtocolMagicId = 764824073}, annotation = "\SUB-\150J\t"}, aHeaderPrevHash = Annotated {unAnnotated = 2aa5103f79c14d5d2cb704223965f7a3f23bdfeb0105737fffda54ad422d1965, annotation = "X *\165\DLE?y\193M],\183\EOT\"9e\247\163\242;\223\235\SOH\ENQs\DEL\255\218T\173B-\EMe"}, aHeaderSlot = Annotated {unAnnotated = SlotNumber {unSlotNumber = 2541752}, annotation = "\130\CANu\EM8\216"}, aHeaderDifficulty = Annotated {unAnnotated = ChainDifficulty {unChainDifficulty = 2540244}, annotation = "\129\SUB\NUL&\194\212"}, headerProtocolVersion = 0.2.0, headerSoftwareVersion = cardano-sl:1, aHeaderProof = Annotated {unAnnotated = Proof {proofUTxO = TxProof {txpNumber = 1, txpRoot = MerkleRoot {getMerkleRoot = f5d2d9331f61f100e486295d5a2adc8fccb39f467e20f07915e49af65b9e1e9a}, txpWitnessesHash = 7a469f952e7eaa5c5337794680e0565b3d47629f9efbea98cdfd6d7868136a81}, proofSsc = SscProof, proofDelegation = afc0da64183bf2664f3d4eec7238d524ba607faeeab24fc100eb861dba69971b, proofUpdate = 4e66280cd94d591072349bec0a3090a53aa945562efb6d08d56e53654b0e4098}, annotation = "\132\131\SOHX \245\210\217\&3\USa\241\NUL\228\134)]Z*\220\143\204\179\159F~ \240y\NAK\228\154\246[\158\RS\154X zF\159\149.~\170\\S7yF\128\224V[=Gb\159\158\251\234\152\205\253mxh\DC3j\129\130\ETXX \211j&\EM\166rIF\EOT\225\ESC\180G\203\207R1\233\242\186%\194\SYN\145w\237\201A\189P\173lX \175\192\218d\CAN;\242fO=N\236r8\213$\186`\DEL\174\234\178O\193\NUL\235\134\GS\186i\151\ESCX Nf(\f\217MY\DLEr4\155\236\n0\144\165:\169EV.\251m\b\213nSeK\SO@\152"}, headerGenesisKey = VerificationKey {unVerificationKey = XPub {xpubPublicKey = "\v\219\US^\243\217\148\ETXu\147\242&bU\241\&4\165de\139\178\223\129K;\156\239\185m\163O\169", xpubChaincode = ChainCode "\200\136Y\FS\133\183p\253\&6rm_=\153\FSf\136(\175\252{\190\br\253i\145\&6\230d\217\216"}}, headerSignature = ABlockSignature {delegationCertificate = UnsafeACertificate {aEpoch = Annotated {unAnnotated = EpochNumber {getEpochNumber = 0}, annotation = "\NUL"}, issuerVK = VerificationKey {unVerificationKey = XPub {xpubPublicKey = "\v\219\US^\243\217\148\ETXu\147\242&bU\241\&4\165de\139\178\223\129K;\156\239\185m\163O\169", xpubChaincode = ChainCode "\200\136Y\FS\133\183p\253\&6rm_=\153\FSf\136(\175\252{\190\br\253i\145\&6\230d\217\216"}}, delegateVK = VerificationKey {unVerificationKey = XPub {xpubPublicKey = "_\221\238\218\222'\DC4\214\219/\158\DC1\EOTt=-\141\129\142\205\220\&0n\ETBa\b\219\DC4\202\173\212A", xpubChaincode = ChainCode "\180W\213\132\f`\248\132\v\153\200\247\140)\n\226)\212\248C\RSg\139\167\165E\195V\a\185M\219"}}, signature = Signature (XSignature {unXSignature = "U'A\247(\EMnb\242\CAN\EOT{\148K$\206M7C\NUL\208K\155(\DC4&\245Z\160\NUL\213=\237f\152\154\213\234\t\b\230\255d\146\NUL\US\241\142\206lp@\169\&4\ACK\aY\233\174\t\134;\242\ETX"}), annotation = "\132\NULX@\v\219\US^\243\217\148\ETXu\147\242&bU\241\&4\165de\139\178\223\129K;\156\239\185m\163O\169\200\136Y\FS\133\183p\253\&6rm_=\153\FSf\136(\175\252{\190\br\253i\145\&6\230d\217\216X@_\221\238\218\222'\DC4\214\219/\158\DC1\EOTt=-\141\129\142\205\220\&0n\ETBa\b\219\DC4\202\173\212A\180W\213\132\f`\248\132\v\153\200\247\140)\n\226)\212\248C\RSg\139\167\165E\195V\a\185M\219X@U'A\247(\EMnb\242\CAN\EOT{\148K$\206M7C\NUL\208K\155(\DC4&\245Z\160\NUL\213=\237f\152\154\213\234\t\b\230\255d\146\NUL\US\241\142\206lp@\169\&4\ACK\aY\233\174\t\134;\242\ETX"}, signature = Signature (XSignature {unXSignature = "\187\171\157S*\255\228\194\148\234\173\CAN^\129\155\204\156#\ESC\215\157C\169L\242O\SI!\n\210\SOa/\182\SO\212\ENQ\191x\226\160\158<H\218L1i\129\170\172\166\GS\138\210P\160@\233I\128\t\FS\ACK"})}, headerAnnotation = "\133\SUB-\150J\tX *\165\DLE?y\193M],\183\EOT\"9e\247\163\242;\223\235\SOH\ENQs\DEL\255\218T\173B-\EMe\132\131\SOHX \245\210\217\&3\USa\241\NUL\228\134)]Z*\220\143\204\179\159F~ \240y\NAK\228\154\246[\158\RS\154X zF\159\149.~\170\\S7yF\128\224V[=Gb\159\158\251\234\152\205\253mxh\DC3j\129\130\ETXX \211j&\EM\166rIF\EOT\225\ESC\180G\203\207R1\233\242\186%\194\SYN\145w\237\201A\189P\173lX \175\192\218d\CAN;\242fO=N\236r8\213$\186`\DEL\174\234\178O\193\NUL\235\134\GS\186i\151\ESCX Nf(\f\217MY\DLEr4\155\236\n0\144\165:\169EV.\251m\b\213nSeK\SO@\152\132\130\CANu\EM8\216X@\v\219\US^\243\217\148\ETXu\147\242&bU\241\&4\165de\139\178\223\129K;\156\239\185m\163O\169\200\136Y\FS\133\183p\253\&6rm_=\153\FSf\136(\175\252{\190\br\253i\145\&6\230d\217\216\129\SUB\NUL&\194\212\130\STX\130\132\NULX@\v\219\US^\243\217\148\ETXu\147\242&bU\241\&4\165de\139\178\223\129K;\156\239\185m\163O\169\200\136Y\FS\133\183p\253\&6rm_=\153\FSf\136(\175\252{\190\br\253i\145\&6\230d\217\216X@_\221\238\218\222'\DC4\214\219/\158\DC1\EOTt=-\141\129\142\205\220\&0n\ETBa\b\219\DC4\202\173\212A\180W\213\132\f`\248\132\v\153\200\247\140)\n\226)\212\248C\RSg\139\167\165E\195V\a\185M\219X@U'A\247(\EMnb\242\CAN\EOT{\148K$\206M7C\NUL\208K\155(\DC4&\245Z\160\NUL\213=\237f\152\154\213\234\t\b\230\255d\146\NUL\US\241\142\206lp@\169\&4\ACK\aY\233\174\t\134;\242\ETXX@\187\171\157S*\255\228\194\148\234\173\CAN^\129\155\204\156#\ESC\215\157C\169L\242O\SI!\n\210\SOa/\182\SO\212\ENQ\191x\226\160\158<H\218L1i\129\170\172\166\GS\138\210P\160@\233I\128\t\FS\ACK\132\131\NUL\STX\NUL\130jcardano-sl\SOH\160X K\169*\163 \198\n\204\154\215\185\166O.\218U\196\210\236(\230\EOT\250\241\134p\139O\fN\142\223", headerExtraAnnotation = "\132\131\NUL\STX\NUL\130jcardano-sl\SOH\160X K\169*\163 \198\n\204\154\215\185\166O.\218U\196\210\236(\230\EOT\250\241\134p\139O\fN\142\223"}, blockBody = ABody {bodyTxPayload = ATxPayload {aUnTxPayload = [ATxAux {aTaTx = Annotated {unAnnotated = UnsafeTx {txInputs = TxInUtxo 637279e61188c847f991743c5530c6d83e167f47a57000bdf52f3076367c2491 0 :| [], txOutputs = TxOut {txOutAddress = Address {addrRoot = 782ed82aad1439d33a20c8c739425b52b1944ea8b5d91cc8a3cf900f, addrAttributes = Attributes { data_ = AddrAttributes {aaVKDerivationPath = Just (HDAddressPayload {getHDAddressPayload = "\202>U<\156c\197\195\STX\140\212C\128H\190\193t\133\159\134\DEL\US\ETB\206\143g\164e"}), aaNetworkMagic = NetworkMainOrStage} }, addrType = ATVerKey}, txOutValue = Lovelace 2783038327581} :| [TxOut {txOutAddress = Address {addrRoot = 6789efcd83371bc0eb2fcaaddf53f78535c6e2bcbbcb40f2af8d455e, addrAttributes = Attributes { data_ = AddrAttributes {aaVKDerivationPath = Just (HDAddressPayload {getHDAddressPayload = "\ENQ\182\164\201\FS\236\242\r.\139\232o\238\227\239O5\132\253\DEL/\194w\242\235\221\186\230"}), aaNetworkMagic = NetworkMainOrStage} }, addrType = ATVerKey}, txOutValue = Lovelace 290903095}], txAttributes = Attributes { data_ = () }}, annotation = "\131\159\130\NUL\216\CANX$\130X cry\230\DC1\136\200G\249\145t<U0\198\216>\SYN\DELG\165p\NUL\189\245/0v6|$\145\NUL\255\159\130\130\216\CANXB\131X\FSx.\216*\173\DC49\211: \200\199\&9B[R\177\148N\168\181\217\FS\200\163\207\144\SI\161\SOHX\RSX\FS\202>U<\156c\197\195\STX\140\212C\128H\190\193t\133\159\134\DEL\US\ETB\206\143g\164e\NUL\SUB\182\DC3\ETB\n\ESC\NUL\NUL\STX\135\250\STX\203\GS\130\130\216\CANXB\131X\FSg\137\239\205\131\&7\ESC\192\235/\202\173\223S\247\133\&5\198\226\188\187\203@\242\175\141E^\161\SOHX\RSX\FS\ENQ\182\164\201\FS\236\242\r.\139\232o\238\227\239O5\132\253\DEL/\194w\242\235\221\186\230\NUL\SUB\146\217^\175\SUB\DC1V\212\&7\255\160"}, aTaWitness = Annotated {unAnnotated = [VKWitness (VerificationKey {unVerificationKey = XPub {xpubPublicKey = "\ENQ\237rlwj>\157\206SIB-\162XO\203\149\165\t,\162\153k\NAK+:\161j\239\RS\DC3", xpubChaincode = ChainCode "\162\ESC=uW\243\&6\154\204\216/\204\r(\FS-O\210\149\186,\192eGuj\205\&6<\189\b\195"}}) (Signature (XSignature {unXSignature = "\155\141\r\162_+\182\215\ETX\ETX\177\241\219\174\147\221\f\SO\178\230(\153:q\233\189-!/\219H\249G\199i\DC3\147\247$k\181\201\175\142\&2\168Km]RF\CAN\225\214#\189\193\STX~H\235\147\227\ENQ"}))], annotation = "\129\130\NUL\216\CANX\133\130X@\ENQ\237rlwj>\157\206SIB-\162XO\203\149\165\t,\162\153k\NAK+:\161j\239\RS\DC3\162\ESC=uW\243\&6\154\204\216/\204\r(\FS-O\210\149\186,\192eGuj\205\&6<\189\b\195X@\155\141\r\162_+\182\215\ETX\ETX\177\241\219\174\147\221\f\SO\178\230(\153:q\233\189-!/\219H\249G\199i\DC3\147\247$k\181\201\175\142\&2\168Km]RF\CAN\225\214#\189\193\STX~H\235\147\227\ENQ"}, aTaAnnotation = "\130\131\159\130\NUL\216\CANX$\130X cry\230\DC1\136\200G\249\145t<U0\198\216>\SYN\DELG\165p\NUL\189\245/0v6|$\145\NUL\255\159\130\130\216\CANXB\131X\FSx.\216*\173\DC49\211: \200\199\&9B[R\177\148N\168\181\217\FS\200\163\207\144\SI\161\SOHX\RSX\FS\202>U<\156c\197\195\STX\140\212C\128H\190\193t\133\159\134\DEL\US\ETB\206\143g\164e\NUL\SUB\182\DC3\ETB\n\ESC\NUL\NUL\STX\135\250\STX\203\GS\130\130\216\CANXB\131X\FSg\137\239\205\131\&7\ESC\192\235/\202\173\223S\247\133\&5\198\226\188\187\203@\242\175\141E^\161\SOHX\RSX\FS\ENQ\182\164\201\FS\236\242\r.\139\232o\238\227\239O5\132\253\DEL/\194w\242\235\221\186\230\NUL\SUB\146\217^\175\SUB\DC1V\212\&7\255\160\129\130\NUL\216\CANX\133\130X@\ENQ\237rlwj>\157\206SIB-\162XO\203\149\165\t,\162\153k\NAK+:\161j\239\RS\DC3\162\ESC=uW\243\&6\154\204\216/\204\r(\FS-O\210\149\186,\192eGuj\205\&6<\189\b\195X@\155\141\r\162_+\182\215\ETX\ETX\177\241\219\174\147\221\f\SO\178\230(\153:q\233\189-!/\219H\249G\199i\DC3\147\247$k\181\201\175\142\&2\168Km]RF\CAN\225\214#\189\193\STX~H\235\147\227\ENQ"}]}, bodySscPayload = SscPayload, bodyDlgPayload = UnsafeAPayload {getPayload = [], getAnnotation = "\159\255"}, bodyUpdatePayload = APayload {payloadProposal = Nothing, payloadVotes = [], payloadAnnotation = "\130\128\159\255"}}, blockAnnotation = "\131\133\SUB-\150J\tX *\165\DLE?y\193M],\183\EOT\"9e\247\163\242;\223\235\SOH\ENQs\DEL\255\218T\173B-\EMe\132\131\SOHX \245\210\217\&3\USa\241\NUL\228\134)]Z*\220\143\204\179\159F~ \240y\NAK\228\154\246[\158\RS\154X zF\159\149.~\170\\S7yF\128\224V[=Gb\159\158\251\234\152\205\253mxh\DC3j\129\130\ETXX \211j&\EM\166rIF\EOT\225\ESC\180G\203\207R1\233\242\186%\194\SYN\145w\237\201A\189P\173lX \175\192\218d\CAN;\242fO=N\236r8\213$\186`\DEL\174\234\178O\193\NUL\235\134\GS\186i\151\ESCX Nf(\f\217MY\DLEr4\155\236\n0\144\165:\169EV.\251m\b\213nSeK\SO@\152\132\130\CANu\EM8\216X@\v\219\US^\243\217\148\ETXu\147\242&bU\241\&4\165de\139\178\223\129K;\156\239\185m\163O\169\200\136Y\FS\133\183p\253\&6rm_=\153\FSf\136(\175\252{\190\br\253i\145\&6\230d\217\216\129\SUB\NUL&\194\212\130\STX\130\132\NULX@\v\219\US^\243\217\148\ETXu\147\242&bU\241\&4\165de\139\178\223\129K;\156\239\185m\163O\169\200\136Y\FS\133\183p\253\&6rm_=\153\FSf\136(\175\252{\190\br\253i\145\&6\230d\217\216X@_\221\238\218\222'\DC4\214\219/\158\DC1\EOTt=-\141\129\142\205\220\&0n\ETBa\b\219\DC4\202\173\212A\180W\213\132\f`\248\132\v\153\200\247\140)\n\226)\212\248C\RSg\139\167\165E\195V\a\185M\219X@U'A\247(\EMnb\242\CAN\EOT{\148K$\206M7C\NUL\208K\155(\DC4&\245Z\160\NUL\213=\237f\152\154\213\234\t\b\230\255d\146\NUL\US\241\142\206lp@\169\&4\ACK\aY\233\174\t\134;\242\ETXX@\187\171\157S*\255\228\194\148\234\173\CAN^\129\155\204\156#\ESC\215\157C\169L\242O\SI!\n\210\SOa/\182\SO\212\ENQ\191x\226\160\158<H\218L1i\129\170\172\166\GS\138\210P\160@\233I\128\t\FS\ACK\132\131\NUL\STX\NUL\130jcardano-sl\SOH\160X K\169*\163 \198\n\204\154\215\185\166O.\218U\196\210\236(\230\EOT\250\241\134p\139O\fN\142\223\132\159\130\131\159\130\NUL\216\CANX$\130X cry\230\DC1\136\200G\249\145t<U0\198\216>\SYN\DELG\165p\NUL\189\245/0v6|$\145\NUL\255\159\130\130\216\CANXB\131X\FSx.\216*\173\DC49\211: \200\199\&9B[R\177\148N\168\181\217\FS\200\163\207\144\SI\161\SOHX\RSX\FS\202>U<\156c\197\195\STX\140\212C\128H\190\193t\133\159\134\DEL\US\ETB\206\143g\164e\NUL\SUB\182\DC3\ETB\n\ESC\NUL\NUL\STX\135\250\STX\203\GS\130\130\216\CANXB\131X\FSg\137\239\205\131\&7\ESC\192\235/\202\173\223S\247\133\&5\198\226\188\187\203@\242\175\141E^\161\SOHX\RSX\FS\ENQ\182\164\201\FS\236\242\r.\139\232o\238\227\239O5\132\253\DEL/\194w\242\235\221\186\230\NUL\SUB\146\217^\175\SUB\DC1V\212\&7\255\160\129\130\NUL\216\CANX\133\130X@\ENQ\237rlwj>\157\206SIB-\162XO\203\149\165\t,\162\153k\NAK+:\161j\239\RS\DC3\162\ESC=uW\243\&6\154\204\216/\204\r(\FS-O\210\149\186,\192eGuj\205\&6<\189\b\195X@\155\141\r\162_+\182\215\ETX\ETX\177\241\219\174\147\221\f\SO\178\230(\153:q\233\189-!/\219H\249G\199i\DC3\147\247$k\181\201\175\142\&2\168Km]RF\CAN\225\214#\189\193\STX~H\235\147\227\ENQ\255\130\ETX\217\SOH\STX\128\159\255\130\128\159\255\129\160"}), byronBlockSlotNo = SlotNo 2541752, byronBlockHash = ByronHash {unByronHash = eae3d3f8a772d17f2e3e543874b5a411f008682e859407b490ad08e10997d3ce}}, encountered error Error SNErrDefault: "resolving tx tx hash \"cf4b6c26a260cae1235d8025128598eba06a52089cc97c62942c2c783bfa2b4a\""
[db-sync-node:Info:95] [2024-08-27 10:10:34.01 UTC] Fixed (2065090,Nothing) consumed_by_tx_id wrong entries
What was initially worrying was the end of warning message:
encountered error Error SNErrDefault: "resolving tx tx hash \"cf4b6c26a260cae1235d8025128598eba06a52089cc97c62942c2c783bfa2b4a\"
but after starting service and checking data things looked OK.
Maybe Information about those edge cases of the fix should be added to the changelog of pre-release ?
OS Your OS: Linux (Debian 12)
Versions The
db-sync
version (egcardano-db-sync --version
): cardano-db-sync 13.4.0.1 PostgreSQL version: 16.3Build/Install Method The method you use to build or install
cardano-db-sync
: cabalRun method The method you used to run
cardano-db-sync
(eg Nix/Docker/systemd/none): systemdAdditional context We are using these config options for dbsync , particularly
tx_out
hasconsumed
value set. These are applied across Koios instances (i.e. ~17 core instances + community ones), and we share snapshots with these options hosted ourselves. Thus, for the data integrity issue below, we're hoping to fix the values and re-host a snapshot with correct history.Problem Report
For Byron tx_out entries, the
tx_out
=>consumed_by_tx_id
is incorrect. Consider transactions for addressDdzFFzCqrhsvgjTLbbZNqacNLQpacoVnHM3AB5r8MiCHSnHx9ebH7Gjizg1KCt8Ba7hDmjA9q1gexs4yq787AQ2kCznVVqyAZNLxaMHr
(notice how consumed_by and tx_id are same):Thus, we're not able to check where this address is used as input:
For those who are not using
consumed
flag, thetx_in
did contain right transactions (eg: see adastat showing 4 transactions here ).Ideally, we'd want this tagged (even if not released) so that the binary built can be used consistently across instances and snapshots we end up creating with fixed data are compatible.
We verified that this is true only for Byron-era:
Thanks to @hodlonaut for testing and providing query results