search

IntersectMBO / cardano-ledger

The ledger implementation and specifications of the Cardano blockchain.
Apache License 2.0
258 stars 158 forks source link

(more) serialization issues #2961

Closed waalge closed 2 years ago

waalge commented 2 years ago

I'm trying to bump node version from 1.34.x to 1.35.2.

I have a bunch of code that depends on plutus-apps, cardano-api, and cardano-ledger.

TBH I'm pretty confused about the overlap of typing between modules/libs, and I'm not sure what/where is responsible for the serialization into CBOR of which bits. The output of 

     .... 
     return $ Just $ (decodeUtf8 . B16.encode . serialiseToCBOR) (makeSignedTransaction [] body)

used to be acceptable to a nami wallet. Since bumping, no longer.

An example output that is invalid according to the CSL

84a70082825820273e58697276ac3a4dbf4ba2e3e846aaaa78d5e3cf13da8c75b447017eda2d6801825820540586d4f9a28308f7d43e7429031c9b02ba86cc01b12ec36c04787e04b3554f000d81825820433251afcbbc89c1460da9796c564fe6d6de2093411530a3d62ca1b0ce7c5dc3000183a200583900dada5e7c74f6c15d1c7899d6620dd17963e8a3e0697f631cd2877ea1bf1a835671aabc836d93cc1e031c2d3a09ce4151e6216b88ebfd85d3011a172db33ea200583900dada5e7c74f6c15d1c7899d6620dd17963e8a3e0697f631cd2877ea1bf1a835671aabc836d93cc1e031c2d3a09ce4151e6216b88ebfd85d3011a000eedc2a300581d700575adec1f876c0fbc5ec8b55adea417a9fe9ca78b65023ef1a07bcc01821a00989680a2581c1762147dcbd338c331debba20ecbbd4f32f69091d14a8d8ff98f3255a145434849505301581c9d452b7878b20559e5779f5cba2e59b1039d7593c1990e361bde11bba1581c0575adec1f876c0fbc5ec8b55adea417a9fe9ca78b65023ef1a07bcc010282005820923918e403bf43c34b4ef6b48eb2ee04babed17320d8d1b9ff9ad086e86f44ec021a0005a17a0e81581cdada5e7c74f6c15d1c7899d6620dd17963e8a3e0697f631cd2877ea109a1581c9d452b7878b20559e5779f5cba2e59b1039d7593c1990e361bde11bba1581c0575adec1f876c0fbc5ec8b55adea417a9fe9ca78b65023ef1a07bcc010b58203244d523013da3320050c5d55cbbb41f7fd43f1b946a685e61c16118b0de474da3038159091159090e0100003323322332232323232323232323232323232323322323232323232232223232533533223500822350042322350062232325335002100110285335333200150260040291330114910253380035003222222222233355301c120013232123300122333500522002002001002350012200112330012253350021035100103225335333573466e3c04cd400488d4008880080cc0c84ccd5cd19b8701235001223500222001033032103200a1028330104910253370032323232333573466e1cc00c0080040ac0a8cd4ccc800540a00180b1200148008d4010888888888801cc8004d540ac88cd400520002235002225335333573466e3c00801c0b80b44c02c0044c01800cd402088cccd40048c98c8098cd5ce2481024c680002620012326320263357389201024c68000262326320263357389201024c680002632001355026223350014800088d4008894cd4ccd5cd19b8f00200702902810011300600330050043333573466e1cd55cea8012400046644246600200600464646464646464646464646666ae68cdc39aab9d500a480008cccccccccc888888888848cccccccccc00402c02802402001c01801401000c008cd40588c8c8cccd5cd19b8735573aa004900011991091980080180118109aba15002301b357426ae8940088c98c80accd5ce01601581489aab9e5001137540026ae854028cd405805cd5d0a804999aa80cbae501835742a010666aa032eb94060d5d0a80399a80b0109aba15006335016335502402275a6ae854014c8c8c8cccd5cd19b8735573aa00490001199109198008018011919191999ab9a3370e6aae754009200023322123300100300233502775a6ae854008c0a0d5d09aba2500223263202f33573806005e05a26aae7940044dd50009aba150023232323333573466e1cd55cea8012400046644246600200600466a04eeb4d5d0a80118141aba135744a004464c6405e66ae700c00bc0b44d55cf280089baa001357426ae8940088c98c80accd5ce01601581489aab9e5001137540026ae854010cd4059d71aba15003335016335502475c40026ae854008c078d5d09aba2500223263202733573805004e04a26ae8940044d5d1280089aba25001135744a00226ae8940044d5d1280089aba25001135744a00226aae7940044dd50009aba150023232323333573466e1d400520062321222230040053019357426aae79400c8cccd5cd19b875002480108c848888c008014c06cd5d09aab9e500423333573466e1d400d20022321222230010053017357426aae7940148cccd5cd19b875004480008c848888c00c014dd71aba135573ca00c464c6404466ae7008c08808007c0780744d55cea80089baa001357426ae8940088c98c806ccd5ce00e00d80c880d09931900d19ab9c491035054350001a135573ca00226ea8004cd54049d71191999ab9a3370ea0029001100f11999ab9a3370ea0049000100f11931900c99ab9c01a019017016135573a6ea8004894cd400440644cd5ce00100c190009aa80c9108911299a80089a80191000910999a802910011802001199aa9803890008028020008919118011bac001320013550192233335573e0024a012466a01060086ae84008c00cd5d100100a11919191999ab9a3370e6aae75400d20002333222123330010040030023232323333573466e1cd55cea80124000466442466002006004602a6ae854008cd4038050d5d09aba2500223263201933573803403202e26aae7940044dd50009aba150033335500775ca00c6ae854008cd4029d71aba135744a004464c6402a66ae7005805404c4d5d1280089aab9e500113754002266aa002eb9d6889119118011bab00132001355017223233335573e0044a010466a00e66aa024600c6aae754008c014d55cf280118021aba200301313574200222440042442446600200800624464646666ae68cdc3a800a40004642446004006600a6ae84d55cf280191999ab9a3370ea0049001109100091931900899ab9c01201100f00e135573aa00226ea80048c8c8cccd5cd19b8735573aa004900011991091980080180118029aba15002375a6ae84d5d1280111931900719ab9c00f00e00c135573ca00226ea80048c8cccd5cd19b8735573aa002900011bae357426aae7940088c98c8030cd5ce00680600509baa00112232323333573466e1d400520042122200123333573466e1d40092002232122230030043006357426aae7940108cccd5cd19b87500348000848880088c98c803ccd5ce00800780680600589aab9d5001137540024646666ae68cdc3a800a4004401c46666ae68cdc3a80124000401c464c6401666ae7003002c0240204d55ce9baa001232323232323333573466e1d4005200c21222222200323333573466e1d4009200a21222222200423333573466e1d400d2008233221222222233001009008375c6ae854014dd69aba135744a00a46666ae68cdc3a8022400c4664424444444660040120106eb8d5d0a8039bae357426ae89401c8cccd5cd19b875005480108cc8848888888cc018024020c030d5d0a8049bae357426ae8940248cccd5cd19b875006480088c848888888c01c020c034d5d09aab9e500b23333573466e1d401d2000232122222223005008300e357426aae7940308c98c804ccd5ce00a00980880800780700680600589aab9d5004135573ca00626aae7940084d55cf280089baa0012323232323333573466e1d400520022333222122333001005004003375a6ae854010dd69aba15003375a6ae84d5d1280191999ab9a3370ea0049000119091180100198041aba135573ca00c464c6401866ae700340300280244d55cea80189aba25001135573ca00226ea80048c8c8cccd5cd19b875001480088c8488c00400cdd71aba135573ca00646666ae68cdc3a8012400046424460040066eb8d5d09aab9e500423263200933573801401200e00c26aae7540044dd50008891119191999ab9a3370e6aae754009200023355007300635742a004600a6ae84d5d1280111931900499ab9c00a009007135573ca00226ea8004448848cc00400c009261200149103505431001225335002153350011004100315335001100310041220021220011220021220011123230010012233003300200200133512233002488120273e58697276ac3a4dbf4ba2e3e846aaaa78d5e3cf13da8c75b447017eda2d6800480088848cc00400c00880050481d879800581840100d8799f581c0575adec1f876c0fbc5ec8b55adea417a9fe9ca78b65023ef1a07bccd87980ff821a000c96ba1a0f010660f5f6

A hand modified version deemed valid

84a70082825820273e58697276ac3a4dbf4ba2e3e846aaaa78d5e3cf13da8c75b447017eda2d6801825820540586d4f9a28308f7d43e7429031c9b02ba86cc01b12ec36c04787e04b3554f000d81825820433251afcbbc89c1460da9796c564fe6d6de2093411530a3d62ca1b0ce7c5dc300018382583900dada5e7c74f6c15d1c7899d6620dd17963e8a3e0697f631cd2877ea1bf1a835671aabc836d93cc1e031c2d3a09ce4151e6216b88ebfd85d31a172db33e82583900dada5e7c74f6c15d1c7899d6620dd17963e8a3e0697f631cd2877ea1bf1a835671aabc836d93cc1e031c2d3a09ce4151e6216b88ebfd85d31a000eedc283581d700575adec1f876c0fbc5ec8b55adea417a9fe9ca78b65023ef1a07bcc821a00989680a2581c1762147dcbd338c331debba20ecbbd4f32f69091d14a8d8ff98f3255a145434849505301581c9d452b7878b20559e5779f5cba2e59b1039d7593c1990e361bde11bba1581c0575adec1f876c0fbc5ec8b55adea417a9fe9ca78b65023ef1a07bcc015820923918e403bf43c34b4ef6b48eb2ee04babed17320d8d1b9ff9ad086e86f44ec021a0005a17a0e81581cdada5e7c74f6c15d1c7899d6620dd17963e8a3e0697f631cd2877ea109a1581c9d452b7878b20559e5779f5cba2e59b1039d7593c1990e361bde11bba1581c0575adec1f876c0fbc5ec8b55adea417a9fe9ca78b65023ef1a07bcc010b58203244d523013da3320050c5d55cbbb41f7fd43f1b946a685e61c16118b0de474da3038159091159090e0100003323322332232323232323232323232323232323322323232323232232223232533533223500822350042322350062232325335002100110285335333200150260040291330114910253380035003222222222233355301c120013232123300122333500522002002001002350012200112330012253350021035100103225335333573466e3c04cd400488d4008880080cc0c84ccd5cd19b8701235001223500222001033032103200a1028330104910253370032323232333573466e1cc00c0080040ac0a8cd4ccc800540a00180b1200148008d4010888888888801cc8004d540ac88cd400520002235002225335333573466e3c00801c0b80b44c02c0044c01800cd402088cccd40048c98c8098cd5ce2481024c680002620012326320263357389201024c68000262326320263357389201024c680002632001355026223350014800088d4008894cd4ccd5cd19b8f00200702902810011300600330050043333573466e1cd55cea8012400046644246600200600464646464646464646464646666ae68cdc39aab9d500a480008cccccccccc888888888848cccccccccc00402c02802402001c01801401000c008cd40588c8c8cccd5cd19b8735573aa004900011991091980080180118109aba15002301b357426ae8940088c98c80accd5ce01601581489aab9e5001137540026ae854028cd405805cd5d0a804999aa80cbae501835742a010666aa032eb94060d5d0a80399a80b0109aba15006335016335502402275a6ae854014c8c8c8cccd5cd19b8735573aa00490001199109198008018011919191999ab9a3370e6aae754009200023322123300100300233502775a6ae854008c0a0d5d09aba2500223263202f33573806005e05a26aae7940044dd50009aba150023232323333573466e1cd55cea8012400046644246600200600466a04eeb4d5d0a80118141aba135744a004464c6405e66ae700c00bc0b44d55cf280089baa001357426ae8940088c98c80accd5ce01601581489aab9e5001137540026ae854010cd4059d71aba15003335016335502475c40026ae854008c078d5d09aba2500223263202733573805004e04a26ae8940044d5d1280089aba25001135744a00226ae8940044d5d1280089aba25001135744a00226aae7940044dd50009aba150023232323333573466e1d400520062321222230040053019357426aae79400c8cccd5cd19b875002480108c848888c008014c06cd5d09aab9e500423333573466e1d400d20022321222230010053017357426aae7940148cccd5cd19b875004480008c848888c00c014dd71aba135573ca00c464c6404466ae7008c08808007c0780744d55cea80089baa001357426ae8940088c98c806ccd5ce00e00d80c880d09931900d19ab9c491035054350001a135573ca00226ea8004cd54049d71191999ab9a3370ea0029001100f11999ab9a3370ea0049000100f11931900c99ab9c01a019017016135573a6ea8004894cd400440644cd5ce00100c190009aa80c9108911299a80089a80191000910999a802910011802001199aa9803890008028020008919118011bac001320013550192233335573e0024a012466a01060086ae84008c00cd5d100100a11919191999ab9a3370e6aae75400d20002333222123330010040030023232323333573466e1cd55cea80124000466442466002006004602a6ae854008cd4038050d5d09aba2500223263201933573803403202e26aae7940044dd50009aba150033335500775ca00c6ae854008cd4029d71aba135744a004464c6402a66ae7005805404c4d5d1280089aab9e500113754002266aa002eb9d6889119118011bab00132001355017223233335573e0044a010466a00e66aa024600c6aae754008c014d55cf280118021aba200301313574200222440042442446600200800624464646666ae68cdc3a800a40004642446004006600a6ae84d55cf280191999ab9a3370ea0049001109100091931900899ab9c01201100f00e135573aa00226ea80048c8c8cccd5cd19b8735573aa004900011991091980080180118029aba15002375a6ae84d5d1280111931900719ab9c00f00e00c135573ca00226ea80048c8cccd5cd19b8735573aa002900011bae357426aae7940088c98c8030cd5ce00680600509baa00112232323333573466e1d400520042122200123333573466e1d40092002232122230030043006357426aae7940108cccd5cd19b87500348000848880088c98c803ccd5ce00800780680600589aab9d5001137540024646666ae68cdc3a800a4004401c46666ae68cdc3a80124000401c464c6401666ae7003002c0240204d55ce9baa001232323232323333573466e1d4005200c21222222200323333573466e1d4009200a21222222200423333573466e1d400d2008233221222222233001009008375c6ae854014dd69aba135744a00a46666ae68cdc3a8022400c4664424444444660040120106eb8d5d0a8039bae357426ae89401c8cccd5cd19b875005480108cc8848888888cc018024020c030d5d0a8049bae357426ae8940248cccd5cd19b875006480088c848888888c01c020c034d5d09aab9e500b23333573466e1d401d2000232122222223005008300e357426aae7940308c98c804ccd5ce00a00980880800780700680600589aab9d5004135573ca00626aae7940084d55cf280089baa0012323232323333573466e1d400520022333222122333001005004003375a6ae854010dd69aba15003375a6ae84d5d1280191999ab9a3370ea0049000119091180100198041aba135573ca00c464c6401866ae700340300280244d55cea80189aba25001135573ca00226ea80048c8c8cccd5cd19b875001480088c8488c00400cdd71aba135573ca00646666ae68cdc3a8012400046424460040066eb8d5d09aab9e500423263200933573801401200e00c26aae7540044dd50008891119191999ab9a3370e6aae754009200023355007300635742a004600a6ae84d5d1280111931900499ab9c00a009007135573ca00226ea8004448848cc00400c009261200149103505431001225335002153350011004100315335001100310041220021220011220021220011123230010012233003300200200133512233002488120273e58697276ac3a4dbf4ba2e3e846aaaa78d5e3cf13da8c75b447017eda2d6800480088848cc00400c00880050481d879800581840100d8799f581c0575adec1f876c0fbc5ec8b55adea417a9fe9ca78b65023ef1a07bccd87980ff821a000c96ba1a0f010660f5f6

The eventual tx: https://testnet.cardanoscan.io/transaction/658fbd94a22bc38e56071ff9f8bc362417c3007277887c6200829e429c0bd760

The TxOuts are of the body are now maps, rather than arrays. I don't know if this is a babbage thing, but the cddl of babbage looks to be just an extension of alonzo.

Since I cannot see this issue elsewhere, I'm presuming the error is mine. But given the weeks I've cumulatively lost to issues of serialization, I thought I'd try to ask the room.

waalge commented 2 years ago

cddl of babbage looks to be just an extension of alonzo.

I was looking at an out dated version of the cddl. Its clearly not an extension of alonzo any longer.

JaredCorduan commented 2 years ago

Its clearly not an extension of alonzo any longer.

It actually is backwards compatible. In particular, for transaction outputs:

https://github.com/input-output-hk/cardano-ledger/blob/275f90cfcd580d5102206953e69ae099aa45f216/eras/babbage/test-suite/cddl-files/babbage.cddl#L75

Where legacy_transaction_output uses a list and post_alonzo_transaction_output uses a map.

I'm not sure what/where is responsible for the serialization into CBOR of which bits.

The ledger (this repo) is responsible for the serialization of transactions. The CLI does have an intermediate serialization for the "transaction body" (in quotes since it is a structure somewhere between what the ledger calls a transaction body and a transaction), which has caused confusion in the past.

An example output that is invalid according to the CSL

By CSL, you meant the "Cardano serialization library", right? ie the rust library provided by dcSpark? I have not used this library myself, could it be that this library is not also still supporting the legacy_transaction_output?

waalge commented 2 years ago

It actually is backwards compatible

Sorry - yes, you're right. What I meant is that I've yet to figure out how to coerce the api to choose the alonzo-style encoding. Both nami and eternl seem unhappy with the new TxOut format. My version of nami is likely too old, but that is transparently dependent on the CSL (it was emurgo back then). Eternl is new but... is a black box.

not also still supporting the legacy_transaction_output

From my attempts, they are only supporting the legacy version. They cannot parse the map

JaredCorduan commented 2 years ago

What I meant is that I've yet to figure out how to coerce the api to choose the alonzo-style encoding. Both nami and eternl seem unhappy with the new TxOut format. My version of nami is likely too old, but that is transparently dependent on the CSL (it was emurgo back then). Eternl is new but... is a black box.

I see, that helps a lot, thank you for the context. Unfortunately, that is out of my wheelhouse (as are the rest of the folks that regularly work in this repo).

From my attempts, they are only supporting the legacy version. They cannot parse the map

That definitely explains it.

Let me know if/how you think I can help, but it sounds like you just need babbage support in CSL.

waalge commented 2 years ago

I appreciate your engagement with this nonetheless

you just need babbage support in CSL.

Alas, such time is a luxury.

Someone has recently added an EraCast module to the cardano-api. That might work here, but its not made it the 1.35.x branches yet which I'm currently dependent on.

JaredCorduan commented 2 years ago

you just need babbage support in CSL.

Alas, such time is a luxury.

I hope I did not dismissive or unappreciative of how much effort this involves! I totally understand, I meant "just update" in the sense that we've properly identified your problem.