search

comit-network / comit-rs

Reference implementation of COMIT, an open protocol facilitating trustless cross-blockchain applications.
GNU General Public License v3.0
192 stars 33 forks source link

Create online bitcoin blocksource connector #1247

Closed bonomat closed 5 years ago

bonomat commented 5 years ago

We should add a blocksource connector in combination to #1131 which makes use of an online service so that the user does not need to run a full node anymore.

bonomat commented 5 years ago

For the online source we need a service that supports:

bonomat commented 5 years ago

blockchain.info/blockchain.com (they got merged in 2018):

Not clear what SLA.

API documentation states:

Please limit your queries to a maximum of 1 every 10 seconds. https://www.blockchain.info/api/q (29/08/2019)

Get latest block

https://blockchain.info/latestblock

Response

Use Curl command in terminal to get a response sh curl https://blockchain.info/latestblock | jq .

{
    "hash":"0000000000000000000774e87b6ef2665feaffd6e8caf790edcd2768728ba6bf",
    "time":1566432812,
    "block_index":1779505,
    "height":591162,
    "txIndexes":[482289893,482288465,482288759,...]
 }

Get block by ID including transaction

https://blockchain.info/rawblock/bid

Response

Use Curl command in terminal to get a response 

{
   "hash":"0000000000000bae09a7a393a8acded75aa67e46cb81f7acaa5ad94f9eacd103",
   "ver":1,
   "prev_block":"00000000000007d0f98d9edca880a6c124e25095712df8952e0439ac7409738a",
   "next_block":[
      "0000000000000981c0f836cc249fb18744fd33458b85d00de3e7f8995f4543ec"
   ],
   "mrkl_root":"935aa0ed2e29a4b81e0c995c39e06995ecce7ddbebb26ed32d550a72e8200bf5",
   "time":1322131230,
   "bits":437129626,
   "fee":200000,
   "nonce":2964215930,
   "n_tx":22,
   "size":9195,
   "block_index":169566,
   "main_chain":true,
   "height":154595,
   "received_time":1322131230,
   "relayed_by":"0.0.0.0",
   "tx":[
      {
         "lock_time":0,
         "ver":1,
         "size":168,
         "inputs":[
            {
               "sequence":4294967295,
               "witness":"",
               "script":"70736a049a110e1a04b099a417522cfabe6d6d4e6988c831bb48c551eea50f87b3c6461ade476fe15c98bed7c6a574aca4ff3501000000000000004d696e656420627920425443204775696c6420ac1eeeed88"
            }
         ],
         "weight":672,
         "time":1322131230,
         "tx_index":1939116,
         "vin_sz":1,
         "hash":"5b09bbb8d3cb2f8d4edbcf30664419fb7c9deaeeb1f62cb432e7741c80dbe5ba",
         "vout_sz":1,
         "relayed_by":"0.0.0.0",
         "out":[
            {
               "spent":true,
               "spending_outpoints":[
                  {
                     "tx_index":1944177,
                     "n":0
                  }
               ],
               "tx_index":1939116,
               "type":0,
               "addr":"1KUCp7YP5FP8ViRxhfszSUJCTAajK6viGy",
               "value":5000200000,
               "n":0,
               "script":"76a914ca975b00a8c203b8692f5a18d92dc5c2d2ebc57b88ac"
            }
         ]
      },

   ]
}

Note that the API in only available under blockchain.info not under the blockchain.com domain (the explorers seem to have been merged under the same domain, but the API is not).

bonomat commented 5 years ago

blockcypher.com

https://www.blockcypher.com/dev/bitcoin/#blockcypher-supported-language-sdks

Get latest block (without transactions)

https://api.blockcypher.com/v1/btc/main

Response
Use Curl command in terminal to get a response ```sh curl https://api.blockcypher.com/v1/btc/main | jq . ``` ``` { "name": "BTC.main", "height": 591163, "hash": "0000000000000000001561224b8d71cc0f7a864edb014458cf49bb477683832f", "time": "2019-08-22T00:40:32.667588748Z", "latest_url": "https://api.blockcypher.com/v1/btc/main/blocks/0000000000000000001561224b8d71cc0f7a864edb014458cf49bb477683832f", "previous_hash": "0000000000000000000774e87b6ef2665feaffd6e8caf790edcd2768728ba6bf", "previous_url": "https://api.blockcypher.com/v1/btc/main/blocks/0000000000000000000774e87b6ef2665feaffd6e8caf790edcd2768728ba6bf", "peer_count": 1044, "unconfirmed_count": 5398, "high_fee_per_kb": 39908, "medium_fee_per_kb": 25000, "low_fee_per_kb": 15000, "last_fork_height": 591032, "last_fork_hash": "000000000000000000171c72115855ea326707b50797624904f4597630551160" } ```

Get block by id

https://api.blockcypher.com/v1/btc/main/blocks/blockhash

Response
Use Curl command in terminal to get a response ```sh curl https://api.blockcypher.com/v1/btc/main/blocks/0000000000000000189bba3564a63772107b5673c940c16f12662b3e8546b412 | jq . ``` ``` { "hash": "0000000000000000001561224b8d71cc0f7a864edb014458cf49bb477683832f", "height": 591163, "chain": "BTC.main", "total": 17861502117169, "fees": 58165082, "size": 895755, "ver": 536870912, "time": "2019-08-22T00:39:43Z", "received_time": "2019-08-22T00:39:43Z", "coinbase_addr": "", "relayed_by": "34.237.15.134:8333", "bits": 387687377, "nonce": 2323836540, "n_tx": 2978, "prev_block": "0000000000000000000774e87b6ef2665feaffd6e8caf790edcd2768728ba6bf", "mrkl_root": "8a708e47d9c817830b5ccfac76ab4cfeaf7aaebdf68531f8fe571e7ebdb10eaf", "txids": [ "83c0d4d1c5962ad96d69c8b3382841bf8c9bfa426ebf4dd6dab7aabdab7af12a",.. ], "depth": 0, "prev_block_url": "https://api.blockcypher.com/v1/btc/main/blocks/0000000000000000000774e87b6ef2665feaffd6e8caf790edcd2768728ba6bf", "tx_url": "https://api.blockcypher.com/v1/btc/main/txs/", "next_txids": "https://api.blockcypher.com/v1/btc/main/blocks/0000000000000000001561224b8d71cc0f7a864edb014458cf49bb477683832f?txstart=20\u0026limit=20" }% ```

Get transaction details by tx id

https://api.blockcypher.com/v1/btc/main/txs/txid

Response
Use Curl command in terminal to get a response ```sh curl https://api.blockcypher.com/v1/btc/main/txs/6a79cf4888fa84630860bf9ab324a8346cc3b14eb5062225b86e6b3d391b9d88 | jq . ``` ```json { "block_hash": "000000000000000000040ff9279c3787e451f408dca999072b044a791e8ddb97", "block_height": 591164, "block_index": 4, "hash": "6a79cf4888fa84630860bf9ab324a8346cc3b14eb5062225b86e6b3d391b9d88", "addresses": [ "15CDrCUmrn6XCsnzmPWzooQEfkrx3k95qN", "15GXVkQGEQoC2zeAJvnntszoVMYeiFTnaQ", "19LUgxZ7SVBwJ3HrvcS7VXUnqkCGcz5fN7", "1AScamgT8iziuEDpm4Ss4BdcYa8c6pxVpK" ], "total": 3709440921, "fees": 115940, "size": 373, "preference": "high", "relayed_by": "178.254.20.84:8333", "confirmed": "2019-08-22T00:45:59Z", "received": "2019-08-22T00:41:25.029Z", "ver": 1, "double_spend": false, "vin_sz": 2, "vout_sz": 2, "confirmations": 1, "confidence": 1, "inputs": [ { "prev_hash": "323a7a3760bf8d6eddfc002c7223501da51938090441fedd502644e4b32d9ed3", "output_index": 0, "script": "47304402206cd0993398e8843591a06981941cc69b49f2d482057291a14d39d535ce69c01f0220386de05eeb43c0c4f45d9b6346e17d19a589c05f9b9126c68ee2135cb307e4c5012103c554f7d913198d56b9229dee998d5c50266a040f44ec2f8f5992dd68bbe392b1", "output_value": 3700000000, "sequence": 4294967295, "addresses": [ "15GXVkQGEQoC2zeAJvnntszoVMYeiFTnaQ" ], "script_type": "pay-to-pubkey-hash", "age": 591163 }, { "prev_hash": "8c4dd5af7362b2149bfc7952b659c77012ef1ccadaae0e0c554d2a883f30c18c", "output_index": 0, "script": "483045022100a7a82fd83901ef12ac54a70012476306180b632e46d07ef46d80e45bb7383062022033b0410be64bda224d6e90e0a458207d2e585514efe3ab0ff6e8b20686073e9501210280d6ef2f4582ac231c48f564d8078dad1ac28a27cec5a82a9ba6ed653b097d4e", "output_value": 9556861, "sequence": 4294967295, "addresses": [ "1AScamgT8iziuEDpm4Ss4BdcYa8c6pxVpK" ], "script_type": "pay-to-pubkey-hash", "age": 590956 } ], "outputs": [ { "value": 9440921, "script": "76a9145b7005e9b41afe51fee4d5c8e3a06de981d6598a88ac", "addresses": [ "19LUgxZ7SVBwJ3HrvcS7VXUnqkCGcz5fN7" ], "script_type": "pay-to-pubkey-hash" }, { "value": 3700000000, "script": "76a9142dffcdcb6d5c558d18d377041d2245d8cbd3943b88ac", "addresses": [ "15CDrCUmrn6XCsnzmPWzooQEfkrx3k95qN" ], "script_type": "pay-to-pubkey-hash" } ] } ```
bonomat commented 5 years ago

https://btc.com/api-doc

Get latest block

https://chain.api.btc.com/v3/block/latest/

Details `curl https://chain.api.btc.com/v3/block/latest/ | jq .` ```json { "data": { "height": 591165, "version": 541065216, "mrkl_root": "7b710b3308952bb164b0ed9d796d9a8ee4eea1f3f9d7ea536b8d36f71d8f44b9", "timestamp": 1566435126, "bits": 387687377, "nonce": 2919550730, "hash": "00000000000000000007ee2320494577d33fad2f7f2cb86db7f0241abe33b10d", "prev_block_hash": "000000000000000000040ff9279c3787e451f408dca999072b044a791e8ddb97", "next_block_hash": "0000000000000000000000000000000000000000000000000000000000000000", "size": 1160041, "pool_difficulty": 35493419660261, "difficulty": 10183488432890, "tx_count": 2122, "reward_block": 1250000000, "reward_fees": 18047186, "created_at": 1566435218, "confirmations": 1, "is_orphan": false, "curr_max_timestamp": 1566435126, "is_sw_block": true, "stripped_size": 944459, "weight": 3993418, "extras": { "pool_name": "BTC.com", "pool_link": "https://pool.btc.com" } }, "err_no": 0, "err_msg": null } ```

Get block by ID

curl https://chain.api.btc.com/v3/block/00000000000000000007ee2320494577d33fad2f7f2cb86db7f0241abe33b10d/ | jq .

{
  "data": {
    "height": 591165,
    "version": 541065216,
    "mrkl_root": "7b710b3308952bb164b0ed9d796d9a8ee4eea1f3f9d7ea536b8d36f71d8f44b9",
    "timestamp": 1566435126,
    "bits": 387687377,
    "nonce": 2919550730,
    "hash": "00000000000000000007ee2320494577d33fad2f7f2cb86db7f0241abe33b10d",
    "prev_block_hash": "000000000000000000040ff9279c3787e451f408dca999072b044a791e8ddb97",
    "next_block_hash": "0000000000000000000000000000000000000000000000000000000000000000",
    "size": 1160041,
    "pool_difficulty": 35493419660261,
    "difficulty": 10183488432890,
    "tx_count": 2122,
    "reward_block": 1250000000,
    "reward_fees": 18047186,
    "created_at": 1566435218,
    "confirmations": 1,
    "is_orphan": false,
    "curr_max_timestamp": 1566435126,
    "is_sw_block": true,
    "stripped_size": 944459,
    "weight": 3993418,
    "extras": {
      "pool_name": "BTC.com",
      "pool_link": "https://pool.btc.com"
    }
  },
  "err_no": 0,
  "err_msg": null
}

Get latest block with transaction details

curl https://chain.api.btc.com/v3/block/latest/tx\?verbose\=3 | jq .

{
  "data": {
    "total_count": 2122,
    "page": 1,
    "pagesize": 50,
    "list": [
      {
        "confirmations": 1,
        "block_height": 591165,
        "block_hash": "00000000000000000007ee2320494577d33fad2f7f2cb86db7f0241abe33b10d",
        "block_time": 1566435126,
        "created_at": 1566435218,
        "fee": 18047186,
        "hash": "08054265d9fd54b006682f85fdd5d9755bef5a9312ca9b4eca6b57c05bf0eb04",
        "inputs_count": 1,
        "inputs_value": 0,
        "is_coinbase": true,
        "is_double_spend": false,
        "is_sw_tx": true,
        "weight": 1248,
        "vsize": 312,
        "witness_hash": "e91f87012f961a7b918fb6b3d4f7c744bf8572dd14218064877e300c3ec69d61",
        "lock_time": 0,
        "outputs_count": 4,
        "outputs_value": 1268047186,
        "size": 339,
        "sigops": 0,
        "version": 2,
        "inputs": [
          {
            "prev_addresses": [],
            "prev_position": -1,
            "prev_tx_hash": "0000000000000000000000000000000000000000000000000000000000000000",
            "prev_type": "NONSTANDARD",
            "prev_value": 0,
            "sequence": 4294967295
          }
        ],
        "outputs": [
          {
            "addresses": [
              "97cfc76442fe717f2a3f0cc9c175f7561b661997"
            ],
            "value": 1268047186,
            "type": "P2WPKH_V0",
            "spent_by_tx": null,
            "spent_by_tx_position": -1
          },
          {
            "addresses": [],
            "value": 0,
            "type": "NULL_DATA",
            "spent_by_tx": null,
            "spent_by_tx_position": -1
          },
          {
            "addresses": [],
            "value": 0,
            "type": "NULL_DATA",
            "spent_by_tx": null,
            "spent_by_tx_position": -1
          },
          {
            "addresses": [],
            "value": 0,
            "type": "NULL_DATA",
            "spent_by_tx": null,
            "spent_by_tx_position": -1
          }
        ]
      },
      ...
bonomat commented 5 years ago

https://bitcoinchain.com/api

get latest block

curl https://api-r.bitcoinchain.com/v1/status | jq .

{
  "height": 591170,
  "hash": "0000000000000000001ba386d5984c4d19c9db2178a4fff0bcb0d9dcbf318d9f",
  "prev_hash": "000000000000000000000665c9923e606d0eb2996222039a970e073a55f366a6",
  "difficulty": "10183488432890.000000",
  "reward": "12.5",
  "next_block_reward": 12.5,
  "time": 1566438291
}

get block with transaction ids

curl https://api-r.bitcoinchain.com/v1/blocks/591170 | jq .

[
  {
    "hash": "0000000000000000001ba386d5984c4d19c9db2178a4fff0bcb0d9dcbf318d9f",
    "height": 591170,
    "prev_block": "000000000000000000000665c9923e606d0eb2996222039a970e073a55f366a6",
    "next_block": "",
    "mrkl_root": "17a826ed6c37d486f65bddbf51536775e18ddf0f193656c7210caa81a0b4ce43",
    "tx": [
      "1cdbeac1c00e245beab2db32f5774ffefb8038c9fc4a4e345c90aad9df910f74",
      ...
    ],
    "tx_count": 1842,
    "reward": "12.50000000",
    "fee": "0.30014507",
    "vout_sum": "168569.76285422",
    "is_main": true,
    "version": "541065216",
    "difficulty": "10183488432890.000000",
    "size": 1287240,
    "bits": "387687377",
    "nonce": "519716272",
    "time": 1566438291
  }
]

get block with

curl https://api-r.bitcoinchain.com/v1/tx/1cdbeac1c00e245beab2db32f5774ffefb8038c9fc4a4e345c90aad9df910f74 | jq .

[
  {
    "block_time": 1566438291,
    "blocks": [
      "0000000000000000001ba386d5984c4d19c9db2178a4fff0bcb0d9dcbf318d9f"
    ],
    "coinbase": "034205090494f35d5d535a30322f4254432e434f4d2ffabe6d6d737f4de1ffca92f289fea1b3e6db11aaf958f6753dd5a66dc23bc4792d717eed0800000016ed7448074af3adb743020000000000",
    "fee": 0,
    "inputs": [
      null
    ],
    "ip": "255.255.255.255",
    "lock_time": 0,
    "outputs": [
      {
        "out_script": {
          "asm": "0 97cfc76442fe717f2a3f0cc9c175f7561b661997",
          "hex": "001497cfc76442fe717f2a3f0cc9c175f7561b661997",
          "reqSigs": 0,
          "type": "witness_v0_keyhash"
        },
        "receiver": "Unknown",
        "spent": false,
        "value": 12.80014507
      },
      {
        "out_script": {
          "asm": "OP_RETURN aa21a9ed714f0b8d6be6dd838813536f5cc5e1b8485b19c9914920e345c142a180321e6b",
          "hex": "6a24aa21a9ed714f0b8d6be6dd838813536f5cc5e1b8485b19c9914920e345c142a180321e6b",
          "reqSigs": 0,
          "type": "nulldata"
        },
        "receiver": "Unknown",
        "spent": false,
        "value": 0
      },
      {
        "out_script": {
          "asm": "OP_RETURN 52534b424c4f434b3af83d1119cb4456555e8aa0d850227f6e30054b3d45af8fc50cb6dfdd10ffdad5",
          "hex": "6a2952534b424c4f434b3af83d1119cb4456555e8aa0d850227f6e30054b3d45af8fc50cb6dfdd10ffdad5",
          "reqSigs": 0,
          "type": "nulldata"
        },
        "receiver": "Unknown",
        "spent": false,
        "value": 0
      },
      {
        "out_script": {
          "asm": "OP_RETURN b9e11b6ded54c49aa55dfb6dee283f184a42ba3a59b2840025933219609dd75efafacfa1",
          "hex": "6a24b9e11b6ded54c49aa55dfb6dee283f184a42ba3a59b2840025933219609dd75efafacfa1",
          "reqSigs": 0,
          "type": "nulldata"
        },
        "receiver": "Unknown",
        "spent": false,
        "value": 0
      }
    ],
    "rec_time": 1566438366,
    "reward": 12.5,
    "self_hash": "1cdbeac1c00e245beab2db32f5774ffefb8038c9fc4a4e345c90aad9df910f74",
    "total_input": 0,
    "total_output": 12.80014507,
    "total_spend_output": 0,
    "version": 2
  }
]
da-kami commented 5 years ago

Rather than parsing the JSON (and mapping) try yo use the hex encoded raw block data. Given the standard block encoding this can be directly mapped into bitcoin library's Block.

Example for returning hex block data for block on blockchain.info:

https://blockchain.info/rawblock/0000000000000000000fabd59170c77c4dbcc188839f5596d4196867e5e92139?format=hex

da-kami commented 5 years ago

Update: blockchain.info does not properly support the testnet, hence we cannot retrieve hex-encoded block data for the testnet.

Because of this we will have to fall back to processing the block as JSON again. The implementation for processing the hex-encoded block could remain for mainnet only.

We can achieve JSON block processing by either using one of the following APIs (mainnet and testnet support):

blockchain.info

The easiest would still be to build on blockchain.info, because they offer an endpoint to retrieve the block including all transactions. Steps when using blockchain.info:

Example JSON block (incl. tx) for mainnet: https://blockchain.info/rawblock/00000000000000000012f911220277bc4c6703bb60e78b21a33653cd2639a9d5 Example JSON block (incl. tx) for testnet: https://testnet.blockchain.info/rawblock/0000000000000115da753cf7f9cb74dbac0a94f5f88b0bb5fa814b8d1f7d07c3

blockchain.info Considerations:

blockstream

Alternatively we can use the blockstream API (could be another, additional blocksource): https://github.com/Blockstream/esplora/blob/master/API.md Steps:

Example API calls (for mainnet just remove /testnet from path):

blockstream Considerations:

da-kami commented 5 years ago

Update:

Details why blockchain.info does not work: blockchain.info returns the previous outputs (as part of TxIn) by TX-index only. But unfortunately there is no endpoint for retrieving the Tx by index for testnet. Since we need the previous outputs for our queries we cannot use blockchain.info as the testnet block TX it not satisfied.

thomaseizinger commented 5 years ago

At least partially done in #1305. New ticket to be created for blockstream API.