Testing txid to txref tool

[ChristopherA]

We have a prototype C tool submitted by @shannona for converting traditional transaction ids (aka txid) to the proposal by @velesavs & @jonasschnelli for bech32 encoded txref's. The code is in this repo at https://github.com/WebOfTrustInfo/btcr-hackathon/tree/master/txid2bech32 until we a more solid version.

We haven't been able to fully test it as we don't have any testnet vectors (see https://github.com/jonasschnelli/bitcoin_txref_code/issues/1 ).

It also was a quick hack and thus it has pretty clunky code and it really needs review and cleanup. Longer term I hope we can use a light client tool rather than a bitcoin full node with -txindex turned on. Currently that is 167 GB.

[ChristopherA]

I just tried it on my own linode testnet full node txindex and I'm getting segfaults — so something is different from @shannona's config and mine. Only thing notable is mine is based on bitcoin 14.2 and his is 14.1, but I doubt that will be it. I'll try reverting to 14.1, otherwise revisit the latest in c debuggers ;-)

[shannona]

I just uploaded a new version with better error reporting. It'll show where your error is rather than segfaulting.

However, the error is almost absolutely that you're working with a txid not in your wallet, and 'get transaction' only works on things in your wallet. This needs to be rewritten quite a bit to use 'getrawtransaction', which also require painful digging through janson arrays to figure out the block index.

[shannona]

There's now a new version that correctly uses getrawtransaction. I've tested it on things both on and off my wallet.

[ChristopherA]

I was eventually able to get the tool to work, but it only worked on a transaction that I had sent. This despite that I have a full node with txindex which means it should have worked for any valid transaction. I think the problem is that there may need to be more RPC calls for transactions not previously known by bitcoind, even thought it has txindex, for this code to function.

In the meantime, I have good testnet based DID with a pointer to a DDO functioning:

You can see the transaction at https://live.blockcypher.com/btc-testnet/tx/f8cdaff3ebd9e862ed5885f8975489090595abe1470397f79780ead1c7528107/

The op_return points to a 69 character value: https://raw.githubusercontent.com/ChristopherA/self/master/ddo.jsonld

From bitcoind here is the transaction:

$ bitcoin-cli -testnet gettransaction f8cdaff3ebd9e862ed5885f8975489090595abe1470397f79780ead1c7528107

{
  "amount": 0.00000000,
  "fee": -0.05000000,
  "confirmations": 3,
  "blockhash": "00000000b3487880b2814da8c0a6b545453d88945dc29a7b700f653cd7e9cdc7",
  "blockindex": 1,
  "blocktime": 1499502050,
  "txid": "f8cdaff3ebd9e862ed5885f8975489090595abe1470397f79780ead1c7528107",
  "walletconflicts": [
  ],
  "time": 1499501000,
  "timereceived": 1499501000,
  "bip125-replaceable": "no",
  "details": [
    {
      "account": "",
      "category": "send",
      "amount": 0.00000000,
      "vout": 0,
      "fee": -0.05000000,
      "abandoned": false
    }
  ],
  "hex": "02000000016a3fbcf558d0a7138e45d5eaaf437c36cdcca00d2638f158e7f814382861cba2000000006a4730440220214449ff0d7808f80af8c625a36407500d47b1b7f6d785da8b01915198ff242a0220213263eac922fab71798088b15a4541703f02a10673f9e68be0f3d791dd351d6012102b97c30de767f084ce3080168ee293053ba33b235d7116a3263d29f1450936b71ffffffff020000000000000000476a4568747470733a2f2f7261772e67697468756275736572636f6e74656e742e636f6d2f4368726973746f70686572412f73656c662f6d61737465722f64646f2e6a736f6e6c6440597307000000001976a914a4eef2e89ae20a4bd4e13eda8dc7ad4f4804cad288ac00000000"
}

Converting the txid to a txref gives me:

./txid2bech32 f8cdaff3ebd9e862ed5885f8975489090595abe1470397f79780ead1c7528107
tx1-xxyv-xxxx-fpmf-u0

Still working on the DDO part. I can't find a tool to generate a pubkey from a testnet encoded base58check private key, or to extract one from a signature. I don't have a tool that will sign a JSON-LD properly with a testnet key yet.

But it will look something like:

{
  "@context": [
    "https://schema.org/",
    "https://w3id.org/security/v1"
  ],
  "id": "DID:BTCR:TX1XXYVXXXXFPMFU0",
  "type": [
    "Credential",
    "WebOfTrustCredential"
  ],
  "issuer": "DID:BTCR:TX1XXYVXXXXFPMFU0",
  "claim": {
    "id": "DID:BTCR:TX1XXYVXXXXFPMFU0",
    "alternatename": "ChristopherA",
    "url": "https://raw.githubusercontent.com/christophera/self/master/ddo.jsonld",
    "control": "ecdsa-koblitz-pubkey:?????",
    "owner": "ecdsa-koblitz-pubkey-hash:mvZ3MyLgsvYr87GGSbsPBWEDduLRptfzEU"
  },
  "signature": {
    "type": "EcdsaKoblitzSignature2016",
    "created": "2017-07-08T00:21:53Z",
    "creator": "ecdsa-koblitz-pubkey:?????",
    "signatureValue": "H/2e5KiopVH3glkoDf/yP/dwScEP0HPZioxNffvalkyUFYDX7OJ/4uKs6gPqjCISpR3DUe1PKTOVE6MSSTS8AeE="
  }
}

[shannona]

Should now work for things you didn't send, with the newest code here in the repo, which uses 'getrawtransaction' instead of 'gettransaction'.

[ChristopherA]

@shannona Thank you Shannon! I tested on my testnet instance, and both my transaction and your test vector transaction both work properly.

[ChristopherA]

Some possible next steps for this tool would be for it to confirm that the transaction is a P2PKH or PW2WPKH function, extract the public key as the control key, then it could construct the default unsigned DDO that is valid for any transaction without a op_return (using values from the block for date, the txout for hash of the owner key), and see if the DID has been revoked (the owner key address has been spent).

[jonasschnelli]

I just merged the txref code into libbtc (https://github.com/libbtc/libbtc). There is now a tool in libbtc called bitcoin-txref.

You can encode a txref by bitcoin-txref 100 1 (block height 100, pos 1). Testnet: bitcoin-txref -t 100 1. Decode: bitcoin-txref tx1-rgxq-qyqq-wutf-dp.

[kimdhamilton]

Goal: all tools give same answers

[shannona]

That would be nice, @kimdhamilton ! Have you been able to update yours to the newest version of Jonas' code? If so, which answer are you getting for f8cdaff3ebd9e862ed5885f8975489090595abe1470397f79780ead1c7528107?

Currently bitcoin-txref from @jonasschnelli gives txtest1-xyv2-xzyq-qqm5-tyke, which I assume is correct while my updated txid2bech32 gives txtest1-xxyv-xzxz-qqmq-tyke, which is oddly close.

[shannona]

Digging into this more, it looks like the hyphenation is the variance that I'm seeing.

Here's mine:

Before Hyphens: txtest1xyv2xzyqqqm5tyke
After Hyphens: txtest1-xxyv-xzxz-qqmq-tyke

Here's Jonas'

Before Hyphens: txtest1xyv2xzyqqqm5tyke
After Hyphens: txtest1-xyv2-xzyq-qqm5-tyke

(Obviously, Jonas' is hyphenating correctly.)

[shannona]

And this is where the variance occurs. Mine:

txtest1xyv2xzyqqqm5tyketyke
txtest1xyv2xzyqqqmqqmqetyke

Jonas':

txtest1xyv2xzyqqqm5tyketyke
txtest1xyv2xzyqqqmqqm5etyke

Which is this line of code, under the nonstandard hyphenation section:

memcpy(output+olen-5, output+olen-8, 4);

[shannona]

OK, so here's how that line works:

The output+olen pointer is at the start of the second tyke. That means that output+olen-8 with a length of 4 should be qqm5 and it should copy to output+olen-5, which is 5tyk. Obviously the right result is Jonas' qqmqqm5e. On mine I instead get qqmqqmqe which is like there was a timing error or something and it picked up the final q after that had already been filled in by the memcpy, instead of the 5 that was there originally.

So, @jonasschnelli , @kimdhamilton , anyone else, any idea why the memcpy might be getting the wrong results when it partially copies over itself? I've doubled checked that:

(1) My output char* array is constructed exactly like Jonas' [This is different from what I have in the repository right now] (2) My pointer to the output remains consistent through these copies (3) My strlen matches what I see when I run it through Jonas' tester

I'm at wit's end on this one, but it's been a long time since I was dealing with precise memory details like this in C, so I could be missing something easy.

[shannona]

Alright, so the answer seems to be that Jonas' library works differently depending on if you compile with the -O2 flag or not.

$ cc -O2 txid2bech32.c txref_code.c segwit_addr.c -lbitcoinrpc -ljansson -o txid2bech32
$ ./txid2bech32 f8cdaff3ebd9e862ed5885f8975489090595abe1470397f79780ead1c7528107
Txref: txtest1-xyv2-xzyq-qqm5-tyke

Versus:

$ cc txid2bech32.c txref_code.c segwit_addr.c -lbitcoinrpc -ljansson -o txid2bech32
$ ./txid2bech32 f8cdaff3ebd9e862ed5885f8975489090595abe1470397f79780ead1c7528107
Txref: txtest1-xxyv-xzxz-qqmq-tyke

So, we can get consistently right results if we compile it consistently, but that's very fragile, and I presume a bug.

I'm going cleanup my copy of this code, reupload it, and update the explanation with a stern warning on compiling.

[shannona]

I've added to the README.md the fact that this needs to be compiled with the -O2 flag. I'm still got a TODO of pushing out a version that has a nice JSON array of the relevant information, rather than the somewhat haphazard output currently. Should be out on Monday or Tuesday.

[kimdhamilton]

@ChristopherA @shannona our answers match now, so we can close this whenever @shannona is ready.

[kimdhamilton]

Thanks for all the help, @jonasschnelli. Except for the issue @shannona raised above (needing to compile with the -O2 flag), everything looks good. We've been able to match your results in 2 prototype implementations:

• c implementation in this repo, uses bitcoin.d: [https://github.com/WebOfTrustInfo/btcr-hackathon/tree/master/txid2bech32](https://github.com/WebOfTrustInfo/btcr-hackathon/tree/master/txid2bech32
• javascript implementation that uses the Blockcypher API https://github.com/WebOfTrustInfo/txref-conversion-js

In the javascript implementation, I ported your txref encoding/decoding unit tests and confirmed they match.

[shannona]

Closed.