search

blockchain-etl / bitcoin-etl

ETL scripts for Bitcoin, Litecoin, Dash, Zcash, Doge, Bitcoin Cash. Available in Google BigQuery https://goo.gl/oY5BCQ
https://twitter.com/BlockchainETL
MIT License
411 stars 123 forks source link

can't decode BTC address #55

Open 2W-12 opened 3 years ago

2W-12 commented 3 years ago

Hello, anyone can help to fix this:

x
Out[178]: <bitcoinetl.domain.block.BtcBlock at 0x7f71d337b780>
x.transactions[1].outputs[1].addresses
Out[179]: ['nonstandardafad2456e5fd6732199d77546c9526255fa66b77']

But bitcoin-cli returns everything correct

root@b:~# bitcoin-cli getrawtransaction edd9212daaeb9ac51ce6f5a8b7ff096c74d4aada71615beb9cd1e0858d5ebe60 1
{
  "txid": "edd9212daaeb9ac51ce6f5a8b7ff096c74d4aada71615beb9cd1e0858d5ebe60",
  "hash": "5361e2dd2475f86433fd7cf409ca96884a9f53bfcd20bda16cdcdeaf23d74ca0",
  "version": 1,
  "size": 382,
  "vsize": 192,
  "weight": 766,
  "locktime": 0,
  "vin": [
    {
      "txid": "e960fdabe8e301846d64b924766ffc4aba94a8fb056c096e7e6c9b631f5e344e",
      "vout": 0,
      "scriptSig": {
        "asm": "",
        "hex": ""
      },
      "txinwitness": [
        "",
        "304402200efe981c24a248b5ee2957a06d592220999a33eed783f93384f06b059e5613fd0220085b445a1638ded282aad9dd49120df2d954a4d307266967fbcad2e54f57328c01",
        "30440220452e0f0bfbba23e93cc265b04fd9c96cf88d9d1ccc49370444b46ef35f6a199802202507471dc999f2bd8bf4ea16d63a71c02153398002b685365228cf9e99d9c81501",
        "52210279d1f38c1c80d47cb00ddbbe2915a60d5706e1ef66056a169150f083b288eb952102cb7d02b654f8616bfc5ab017b7a3ec9092e466381af0f552b7efcd8d920453672103c96d495bfdd5ba4145e3e046fee45e84a8a48ad05bd8dbb395c011a32cf9f88053ae"
      ],
      "sequence": 4294967295
    }
  ],
  "vout": [
    {
      "value": 0.00913448,
      "n": 0,
      "scriptPubKey": {
        "asm": "OP_DUP OP_HASH160 1683db0ca249360942d1a9dabe5d0cced0c5a903 OP_EQUALVERIFY OP_CHECKSIG",
        "hex": "76a9141683db0ca249360942d1a9dabe5d0cced0c5a90388ac",
        "address": "1343piJzM5DB4N3gDE7VF17nnRD5FVj4pr",
        "type": "pubkeyhash"
      }
    },
    {
      "value": 0.00682552,
      "n": 1,
      "scriptPubKey": {
        "asm": "0 701a8d401c84fb13e6baf169d59684e17abd9fa216c8cc5b9fc63d622ff8c58d",
        "hex": "0020701a8d401c84fb13e6baf169d59684e17abd9fa216c8cc5b9fc63d622ff8c58d",
        "address": "bc1qwqdg6squsna38e46795at95yu9atm8azzmyvckulcc7kytlcckxswvvzej",
        "type": "witness_v0_scripthash"
      }
    }
  ],
  "hex": "010000000001014e345e1f639b6c7e6e096c05fba894ba4afc6f7624b9646d8401e3e8abfd60e90000000000ffffffff0228f00d00000000001976a9141683db0ca249360942d1a9dabe5d0cced0c5a90388ac386a0a0000000000220020701a8d401c84fb13e6baf169d59684e17abd9fa216c8cc5b9fc63d622ff8c58d040047304402200efe981c24a248b5ee2957a06d592220999a33eed783f93384f06b059e5613fd0220085b445a1638ded282aad9dd49120df2d954a4d307266967fbcad2e54f57328c014730440220452e0f0bfbba23e93cc265b04fd9c96cf88d9d1ccc49370444b46ef35f6a199802202507471dc999f2bd8bf4ea16d63a71c02153398002b685365228cf9e99d9c815016952210279d1f38c1c80d47cb00ddbbe2915a60d5706e1ef66056a169150f083b288eb952102cb7d02b654f8616bfc5ab017b7a3ec9092e466381af0f552b7efcd8d920453672103c96d495bfdd5ba4145e3e046fee45e84a8a48ad05bd8dbb395c011a32cf9f88053ae00000000",
  "blockhash": "000000000000000000086a2cff995563e22ed9ed046dc8b56690130893d6175c",
  "confirmations": 1,
  "time": 1637151616,
  "blocktime": 1637151616
}

so correct address for this transaction should be: 1343piJzM5DB4N3gDE7VF17nnRD5FVj4pr

jingli85 commented 3 years ago

After the bitcoin core update to release 22.0, we started to see all BTC addresses are converted to nonstandard by bitcoin-etl.

From the 22.0 release note https://bitcoincore.org/en/releases/22.0/, we see The following RPCs: gettxout, getrawtransaction, decoderawtransaction, decodescript, gettransaction, and REST endpoints: /rest/tx, /rest/getutxos, /rest/block deprecated the following fields (which are no longer returned in the responses by default): addresses, reqSigs. The -deprecatedrpc=addresses flag must be passed for these fields to be included in the RPC response. This flag/option will be available only for this major release, after which the deprecation will be removed entirely. Note that these fields are attributes of the scriptPubKey object returned in the RPC response. However, in the response of decodescript these fields are top-level attributes, and included again as attributes of the scriptPubKey object. (#20286)

The root cause of this issue is that core 22.0 changed the raw json structure. Pre 22.0, it was: "scriptPubKey": { "asm": "OP_DUP OP_HASH160 b2f7ff4dc4bb0402aec4664a683c4c0650b1f915 OP_EQUALVERIFY OP_CHECKSIG", "hex": "76a914b2f7ff4dc4bb0402aec4664a683c4c0650b1f91588ac", "reqSigs": 1, "type": "pubkeyhash", "addresses": [ "1HKJNJLdgXh79WPgR325fq4wb6opqV5Nqr" ] } With core 22.0, it is "scriptPubKey": { "asm": "OP_DUP OP_HASH160 b2f7ff4dc4bb0402aec4664a683c4c0650b1f915 OP_EQUALVERIFY OP_CHECKSIG", "hex": "76a914b2f7ff4dc4bb0402aec4664a683c4c0650b1f91588ac", "address": "1HKJNJLdgXh79WPgR325fq4wb6opqV5Nqr", "type": "pubkeyhash" } The array of "addresses" is replaced with "address"

To fix the problem, we modified 1, the line at this https://github.com/blockchain-etl/bitcoin-etl/blob/master/bitcoinetl/mappers/transaction_output_mapper.py#L49, from output.addresses = script_pub_key.get('addresses') to output.addresses = [script_pub_key.get('address')] in transaction_output_mapper.py 2, the set nonstandard address at here https://github.com/blockchain-etl/bitcoin-etl/blob/2d0012f776366b8682563c80ba2cb4820a826582/bitcoinetl/service/btc_service.py#L158 accordingly.

We are testing the change right now to see if the problem is fixed or not.

vu-opstech commented 2 years ago

I got same the issue! Still waiting for correct data :D

akhorshidi commented 2 years ago

The same issue exists here and the simple workaround for me was to directly send JSON-RPC calls to the web server started by bitcoind and then do some parsing for each type of the sctipts. For example for scriptPubKey of type P2PK we have: 

def script_pub_key_to_pub_key_hash(script_pub_key: str, type: str = 'pubkey'):
    """
    Create a public key hash from scriptPubKey hex

    :param script_pub_key: scriptPubKey hex in hexadecimal notation
    :type script_pub_key: str
    :param type: type of the scriptPubKey
    :type type: str
    :return: hash of public key for P2PK address
    :rtype: bytes
    """
    try:
        if type == 'pubkey':
            # Extract Public Key from ScriptPubKey
            pubkey_hex = script_pub_key[2:-2]
            # The number of bytes in the Public Key
            print(f'The length of public key is {int("0x" + script_pub_key_hex[:2], base=16)} bytes.')
            # Convert Public Key hex too binary
            pubkey_bin = binascii.unhexlify(pubkey_hex)
            # Create SHA-256 hash from Public Key binary
            pub_key_hash = hashlib.sha256(pubkey_bin).digest()
            # Compute RIPEMD-160 hash value/digest
            ripemd160 = hashlib.new('ripemd160')
            ripemd160.update(pub_key_hash)
            pub_key_double_hash = ripemd160.digest()

            return pub_key_double_hash

    except Exception as e:
        logging.info("Exception occurred: ", e, exc_info=True)

def pub_key_hash_to_addr(pubkeyhash: bytes, version_prefix: bytes = b'\x00'):
    """
    Create a Base58Check-encoded address from public key hash

    :param pubkeyhash: hash of public key
    :type pubkeyhash: bytes
    :param version_prefix: a version byte added to hash
    :type version_prefix: bytes
    :return: An bitcoin address in Base58Check format
    :rtype: bytes
    """
    try:
        # First add version byte to get a padded hash
        hash_versioned = version_prefix + pubkeyhash
        # Apply the SHA256 hash algorithm twice
        hash_first = hashlib.sha256(hash_versioned).digest()
        hash_second = hashlib.sha256(hash_first).digest()
        # Add the first four bytes as checksum
        hash_checksum = hash_versioned + hash_second[:4]
        # Encode in Base58Check
        base58check_encoded_address = base58.b58encode(hash_checksum)

        return base58check_encoded_address

    except Exception as e:
        logging.info("Exception occurred: ", e, exc_info=True)

if __name__ == "__main__":
    script_pub_key_hex = "410400d0ade32217e076945e0946ef7bed72d9aea035aa8891e4bf0749ae6e24f8a7d3ea56efafe472ac3943dbed3af7c093729720ac9ab04e8eba09286e3a00fe41ac"
    utxo_hash = script_pub_key_to_pub_key_hash(script_pub_key_hex)
    utxo_addr = pub_key_hash_to_addr(utxo_hash)
    print(utxo_addr)

HTH

medvedev1088 commented 2 years ago

This issue should be fixed in the latest version https://github.com/blockchain-etl/bitcoin-etl/pull/57/files

akhorshidi commented 2 years ago

This issue should be fixed in the latest version https://github.com/blockchain-etl/bitcoin-etl/pull/57/files

I'm using the latest version committed here and get the same output yet: https://github.com/blockchain-etl/bitcoin-etl/commit/77f1b469c46fa803a74963ff814154a0199cfe57

medvedev1088 commented 2 years ago

@akhorshidi do you mind adding steps to reproduce? It works correctly for edd9212daaeb9ac51ce6f5a8b7ff096c74d4aada71615beb9cd1e0858d5ebe60 posted above

akhorshidi commented 2 years ago

410400d0ade32217e076945e0946ef7bed72d9aea035aa8891e4bf0749ae6e24f8a7d3ea56efafe472ac3943dbed3af7c093729720ac9ab04e8eba09286e3a00fe41ac

How to reproduce the bug: export_blocks_and_transactions --provider-uri http://user:pass@bitcoind:8332 --chain bitcoin --start-block 119965 --end-block 119965 --blocks-output output/blocks.json --transactions-output output/transactions.json

Output Log: {"hash": "d70ae1131d433b655d0faeae1db4efa15bb4138f1e38e60a53073a58ea1dcc34", "size": 135, "virtual_size": 135, "version": 1, "lock_time": 0, "block_number": 119965, "block_hash": "00000000000069cf03c847d7d1d58b44474021b31dc65a8e83fd3a08e60a768c", "block_timestamp": 1303671399, "is_coinbase": true, "index": 0, "inputs": [], "outputs": [{"index": 0, "script_asm": "0400d0ade32217e076945e0946ef7bed72d9aea035aa8891e4bf0749ae6e24f8a7d3ea56efafe472ac3943dbed3af7c093729720ac9ab04e8eba09286e3a00fe41 OP_CHECKSIG", "script_hex": "410400d0ade32217e076945e0946ef7bed72d9aea035aa8891e4bf0749ae6e24f8a7d3ea56efafe472ac3943dbed3af7c093729720ac9ab04e8eba09286e3a00fe41ac", "required_signatures": null, "type": "nonstandard", "addresses": ["nonstandard2214070acbb06e77ad479bb181885c2c4702d758"], "value": 5012000000}], "input_count": 0, "output_count": 1, "input_value": 0, "output_value": 5012000000, "fee": 0}

As you can see, it labels all the addresses of type PubKey as nonstandard, but bitcoin-cli reports the correct public key which then can be used to generate the corresponding UTXO address, same as ones the we'll see in the explorers like this

bitcoin-cli getrawtransaction d70ae1131d433b655d0faeae1db4efa15bb4138f1e38e60a53073a58ea1dcc34 1 

"vout": [
    {
      "value": 50.12000000,
      "n": 0,
      "scriptPubKey": {
        "asm": "0400d0ade32217e076945e0946ef7bed72d9aea035aa8891e4bf0749ae6e24f8a7d3ea56efafe472ac3943dbed3af7c093729720ac9ab04e8eba09286e3a00fe41 OP_CHECKSIG",
        "hex": "410400d0ade32217e076945e0946ef7bed72d9aea035aa8891e4bf0749ae6e24f8a7d3ea56efafe472ac3943dbed3af7c093729720ac9ab04e8eba09286e3a00fe41ac",
        "type": "pubkey"
      }
    }
  ],
medvedev1088 commented 2 years ago

Ah I see, thanks for the steps.

bitcoin-etl doesn't currently parse scripts and only relies on addresses returned by bitcoin-cli. There is a feature request for this https://github.com/blockchain-etl/bitcoin-etl/issues/43