Question: Anchoring dids using bitcoind in regtest mode

This is not an issue, but rather asking for help with setting up a local development environment using regtest mode.

I am running dockerized version of the ION including a BTC node in regtest mode. I've got everything working fine, except that the ion-bitcoin node fails to write transactions with error not_enough_balance_for_write.

Following, I'll try to provide as much details as I can about how I am setting up the environment.

This is what I see in ion-core

2024-01-09 16:58:07 Getting blockchain time...
2024-01-09 16:58:07 Got latest blockchain time: {"time":444,"hash":"332cf4c22de4c10c2f0c1fc44748b9da7d83c4974680f9980fedacb7d2568d17"}
2024-01-09 16:58:07 Core cachedApproximateTime updated to: 444
2024-01-09 16:58:07 Wrote 12442 byte content as IPFS CID: Qmb9uDTmy77NFoEikKG2aNWx3EP15hZDJmC4U44ieeVMcg
2024-01-09 16:58:07 Wrote chunk file Qmb9uDTmy77NFoEikKG2aNWx3EP15hZDJmC4U44ieeVMcg to content addressable store.
2024-01-09 16:58:07 Wrote 93 byte content as IPFS CID: QmW9rLo57FViTMPJEYPMoJnjoptG3dzAXK87pxTeC6JRwJ
2024-01-09 16:58:07 Wrote provisional index file QmW9rLo57FViTMPJEYPMoJnjoptG3dzAXK87pxTeC6JRwJ to content addressable store.
2024-01-09 16:58:07 Wrote 6428 byte content as IPFS CID: QmUuhtpT8XSyewMmgBJF1u3X245rekMeWCGYv9GxyqovDG
2024-01-09 16:58:07 Wrote core index file QmUuhtpT8XSyewMmgBJF1u3X245rekMeWCGYv9GxyqovDG to content addressable store.
2024-01-09 16:58:07 Writing data to blockchain: 90.QmUuhtpT8XSyewMmgBJF1u3X245rekMeWCGYv9GxyqovDG with minimum fee of: 400
2024-01-09 16:58:07 CommandSucceededEvent {
2024-01-09 16:58:07   connectionId: 'mongo:27017',
2024-01-09 16:58:07   requestId: 1017,
2024-01-09 16:58:07   commandName: 'find',
2024-01-09 16:58:07   duration: 0,
2024-01-09 16:58:07   reply: {
2024-01-09 16:58:07     cursor: { firstBatch: [], id: 0, ns: 'ion-regtest-core.transactions' },
2024-01-09 16:58:07     ok: 1
2024-01-09 16:58:07   }
2024-01-09 16:58:07 }
2024-01-09 16:58:07 Fetching Sidetree transactions from blockchain service...
2024-01-09 16:58:07 Fetching URI 'http://regtest-ion-bitcoin:3002/transactions'...
2024-01-09 16:58:07 Event emitted: sidetree_batch_writer_loop_failure: {"code":"not_enough_balance_for_write"}
2024-01-09 16:58:07 End batch writing. Duration: 190 ms.

And this is the counterpart at ion-bitcoin

2024-01-09 16:59:07 Verifying block 444 (332cf4c22de4c10c2f0c1fc44748b9da7d83c4974680f9980fedacb7d2568d17)
2024-01-09 16:59:07 Getting current block height...
2024-01-09 16:59:07 Sending RPC request: {"method":"getblockcount","jsonrpc":"1.0","id":"3c2c68s02us"}
2024-01-09 16:59:07 Getting hash for block 444
2024-01-09 16:58:07 Error: Not enough satoshis to broadcast. Failed to broadcast anchor string 90.QmUuhtpT8XSyewMmgBJF1u3X245rekMeWCGYv9GxyqovDG
2024-01-09 16:58:07     at BitcoinProcessor.<anonymous> (/node_modules/@decentralized-identity/sidetree/dist/lib/bitcoin/BitcoinProcessor.js:424:31)
2024-01-09 16:58:07     at Generator.next (<anonymous>)
2024-01-09 16:58:07     at fulfilled (/node_modules/@decentralized-identity/sidetree/dist/lib/bitcoin/BitcoinProcessor.js:5:58)
2024-01-09 16:58:07     at runMicrotasks (<anonymous>)
2024-01-09 16:58:07     at processTicksAndRejections (node:internal/process/task_queues:96:5)
2024-01-09 16:59:07 Sending RPC request: {"method":"getblockhash","params":[444],"jsonrpc":"1.0","id":"1ht4bfck0s0"}

Of course the error is pretty self explanatory, whatever the bitcoinWalletOrImportString is it is not funded. What I am struggling with is getting the correct value for regtest mode for bitcoinWalletOrImportString.

The steps I am executing are described below, after this everything works as expected except for the anchoring part which fails as mentioned above.

1 - Create Regtest configuration file

I created a new file named docker-regtest-bitcoin-config.json as follow:

{
  "bitcoinDataDirectory": "/data/bitcoin/regtest",
  "bitcoinFeeSpendingCutoffPeriodInBlocks": 1,
  "bitcoinFeeSpendingCutoff": 0.001,
  "bitcoinPeerUri": "http://regtest-bitcoin-core:18443",
  "bitcoinRpcUsername": "user",
  "bitcoinRpcPassword": "password",
  "bitcoinWalletOrImportString": "[Value got from step 6.1]",
  "databaseName": "ion-regtest-bitcoin",
  // Since I am running in regtest mode I need to fund wallets first
  // and create some transactions so that fees can be estimated.
  // In my local node, at height 200 I have all what's needed to start
  // running all components
  "genesisBlockNumber": 200,
  "logRequestError": true,
  "mongoDbConnectionString": "mongodb://mongo:27017/",
  "port": 3002,
  "sidetreeTransactionFeeMarkupPercentage": 1,
  "sidetreeTransactionPrefix": "ion:regtest",
  "transactionPollPeriodInSeconds": 60,
  "valueTimeLockUpdateEnabled": false,
  "valueTimeLockAmountInBitcoins": 0,
  "valueTimeLockPollPeriodInSeconds": 600,
  "valueTimeLockTransactionFeesAmountInBitcoins": 0.0001
}

2. I created a new config file name `docker-regtest-core-config.json` as follow

{
  "batchingIntervalInSeconds": 600,
  "blockchainServiceUri": "http://regtest-ion-bitcoin:3002",
  "databaseName": "ion-regtest-core",
  "didMethodName": "ion:regtest",
  "ipfsHttpApiEndpointUri": "http://ipfs:5001",
  "maxConcurrentDownloads": 20,
  "mongoDbConnectionString": "mongodb://mongo:27017/",
  "observingIntervalInSeconds": 60,
  "port": 3000
}

3. I created a new docker-compose override file, named `docker-compose.regtest-override.yml`, as follow

services:
  bitcoin-core:
    container_name: regtest-bitcoin-core
    command: --port=18443 -printtoconsole -txindex -server -rpcuser=user -rpcpassword=password -rpcallowip=0.0.0.0/0 -rpcbind=0.0.0.0 -regtest=1 -fallbackfee=0.000001
    ports:
      - "18443:18443"

  ion-bitcoin:
    container_name: regtest-ion-bitcoin
    # I am rebuilding the docker image so that new configuration files
    # are included, I did change this in docker-compose file as well
    build:
      context: ..
      dockerfile: ./docker/dockerfile
    environment:
      - ION_BITCOIN_CONFIG_FILE_PATH=/config/docker-regtest-bitcoin-config.json
      - ION_BITCOIN_VERSIONING_CONFIG_FILE_PATH=/config/regtest-bitcoin-versioning.json

  ion-core:
    container_name: regtest-ion-core
    # I am rebuilding the docker image so that new configuration files
    # are included, I did change this in docker-compose file as well
    build:
      context: ..
      dockerfile: ./docker/dockerfile
    environment:
      - ION_CORE_CONFIG_FILE_PATH=/config/docker-regtest-core-config.json
      - ION_CORE_VERSIONING_CONFIG_FILE_PATH=/config/regtest-core-versioning.json

4. I run docker-compose

The first time I put to run all components but ion-core and ion-bitcoin fail, so I stop both, initialize bitcoin and once everything is ready I turn them both on again. In my local I am using docker-compose profiles so that I can chose which services to start.

docker-compose -f docker-compose.yml -f docker-compose.regtest-override.yml up -d

5. Bootstrap bitcoin

Create a new wallet

bitcoin-cli -regtest -rpcuser=user -rpcpassword=password createwallet "ion_wallet"

Create a new address

bitcoin-cli -regtest -rpcuser=user -rpcpassword=password -rpcwallet=ion_wallet getnewaddress
# Example Output = bcrt1qnnlxd7e9dahtltdgm0y3msrcqapfms283n0v95

Mine 1 block to get funds for that address

bitcoin-cli -regtest -rpcuser=user -rpcpassword=password generatetoaddress 1 bcrt1qnnlxd7e9dahtltdgm0y3msrcqapfms283n0v95

Mine 100 blocks to a random address so that those funds can be unlocked

bitcoin-cli -regtest -rpcuser=user -rpcpassword=password generatetoaddress 100 mtbZzVBwLnDmhH4pE9QynWAgh6H3aC1E6M

Generate some random transactions until fees can be estimated

Here I just run a script that mines some blocks with some transactions until bitcoin-cli -regtest -rpcuser=user -rpcpassword=password estimatesmartfee [some height] starts returning some feerate

6. Create a new address, get WIF and fund it

sidetreeDefaultWallet should be already loaded at this point, if not I do it manually

Create a new address

bitcoin-cli -regtest -rpcuser=user -rpcpassword=password -rpcwallet=sidetreeDefaultWallet getnewaddress
# Example Output = bcrt1qsnxmsd5f6gcx4j26m6l86eyxf6em60m4uz7uu2

Fund the address

bitcoin-cli -regtest -rpcuser=user -rpcpassword=password -rpcwallet=ion_wallet sendtoaddress "bcrt1qsnxmsd5f6gcx4j26m6l86eyxf6em60m4uz7uu2" 10.0

Get WIF from address

bitcoin-cli -regtest -rpcuser=user -rpcpassword=password -rpcwallet=sidetreeDefaultWallet dumpprivkey "bcrt1qsnxmsd5f6gcx4j26m6l86eyxf6em60m4uz7uu2"
# Example Output = cUwg7SMKozMFTq2dy3fid9Pu1UHKCgsWva5dk2QV4ovo92zj5HgS

Finally update the bitcoinWalletOrImportString in docker-regtest-bitcoin-config.json WIF value obtained from step above

7. Update configuration

At this step I just rebuild the solution (npm run build) so that new configuration files are included in dist, I delete the containers and images for ion-core and ion-bitcoin and rerun docker-compose with the regtest override file.

8. I see some Unspent satoshis for that addrses

bitcoin-cli -regtest -rpcuser=user -rpcpassword=password -rpcwallet=sidetreeDefaultWallet listunspent 1 443 "[\"bcrt1qsnxmsd5f6gcx4j26m6l86eyxf6em60m4uz7uu2\"]"

# Example Output
[
  {
    "txid": "d08ff9109cecaa9d15f1cb635f11358d76453805086b10f6f8b6cd25729b9502",
    "vout": 1,
    "address": "bcrt1qsnxmsd5f6gcx4j26m6l86eyxf6em60m4uz7uu2",
    "label": "",
    "scriptPubKey": "001484cdb83689d2306ac95adebe7d64864eb3bd3f75",
    "amount": 10.00000000,
    "confirmations": 107,
    "spendable": true,
    "solvable": true,
    "desc": "wpkh([1f113800/0'/0'/1']02e5acd26e21db98e75768a5e03c3e206961742ae8547317f78af6f35899f8ea08)#0xhptkqk",
    "safe": true
  },
  {
    "txid": "309b6d99f9b8495cf30417fada82aa2e6cd93a4fd902636c21dce23e11664e32",
    "vout": 0,
    "address": "bcrt1qsnxmsd5f6gcx4j26m6l86eyxf6em60m4uz7uu2",
    "label": "",
    "scriptPubKey": "001484cdb83689d2306ac95adebe7d64864eb3bd3f75",
    "amount": 12.50001340,
    "confirmations": 107,
    "spendable": true,
    "solvable": true,
    "desc": "wpkh([1f113800/0'/0'/1']02e5acd26e21db98e75768a5e03c3e206961742ae8547317f78af6f35899f8ea08)#0xhptkqk",
    "safe": true
  },
  {
    "txid": "81b8cec00ab25e4679df1edda0bd9a4e4036459672eeedaa92e596733104a75b",
    "vout": 1,
    "address": "bcrt1qsnxmsd5f6gcx4j26m6l86eyxf6em60m4uz7uu2",
    "label": "",
    "scriptPubKey": "001484cdb83689d2306ac95adebe7d64864eb3bd3f75",
    "amount": 10.00000000,
    "confirmations": 108,
    "spendable": true,
    "solvable": true,
    "desc": "wpkh([1f113800/0'/0'/1']02e5acd26e21db98e75768a5e03c3e206961742ae8547317f78af6f35899f8ea08)#0xhptkqk",
    "safe": true
  },
  {
    "txid": "1cdf9507b4fd7d8ef05991f95d7dff20e553dae0a82b65527cdcaffc0e98fa78",
    "vout": 0,
    "address": "bcrt1qsnxmsd5f6gcx4j26m6l86eyxf6em60m4uz7uu2",
    "label": "",
    "scriptPubKey": "001484cdb83689d2306ac95adebe7d64864eb3bd3f75",
    "amount": 12.50001340,
    "confirmations": 108,
    "spendable": true,
    "solvable": true,
    "desc": "wpkh([1f113800/0'/0'/1']02e5acd26e21db98e75768a5e03c3e206961742ae8547317f78af6f35899f8ea08)#0xhptkqk",
    "safe": true
  }
]

TBD54566975 / sidetree-ion