Feature: Transition to a config file format that's a bit more flexible

DeftNerd commented 7 years ago

The Problem

Bitcoin's config file is in a windows-oriented INI file format. There are some configuration options that could benefit from being stored as arrays or in subarrays. It might be worth considering alternatives.

The Basic Solution

JSON formatted config files are fairly common and understood by administrators. We could create a bitcoin.json with default values. The Bitcoin daemon could choose which config file parser to use based on the file extension of the config file that is specified on startup.

If no config file is specified and a default is picked, it would pick the default INI style .conf file for backwards compatibility. Changing the default to load the .json config file could change with a future major revision like v1.15

Downsides

It's hard to properly comment JSON files to document helpful information and options. XML is always an option, but it's a bit 'meh' especially since the JSON-RPC port in Bitcoin indicates support for JSON as a protocol choice over XML.

Creating a JSON schema document that resides alongside the json config file could answer a lot of questions.

Example with some modifications to variable names

{
  "_comment_chain": "can be mainnet, testnet (testnet3), testnet3, or regtest",
  "chain": "mainnet",
  "network": {
    "proxy": {
      "enabled": false,
      "protocol": "socks5",
      "address": "127.0.0.1",
      "port": 9050
    },
    "bindings": [
      {
        "_comment_address": "A null address indicates all available network interfaces",
        "address": null,
        "p2p": {
          "_comment_enable": "Enabling a p2p binding allows outside connections",
          "enabled": true,
          "port": 8333,
          "_comment_whitelist": "Enabling this makes any connections to this interface whitelisted",
          "whitelist": false,
          "_comment_connections": "Maximum number of connections this interface can have (or for each interface if no interface is specified) ",
          "connections": 10
        },
        "rpc": {
          "_comment_enabled": "enables the json-rpc interface on this address",
          "enabled": false,
          "port": 8332,
          "timeout": 30,
          "connections": 10,
          "_comment_allow": "Array of ip addresses or subnets allowed to access the JSON-RPC interface. Leave the array empty to disable this function or to let the OS handle this",
          "allow": []
        }
      },
      {
        "_comment_address": "A null address indicates all available network interfaces",
        "address": "127.0.0.1",
        "p2p": {
          "_comment_enable": "Enabling a p2p binding allows outside connections",
          "enabled": true,
          "port": 8333,
          "_comment_whitelist": "Enabling this makes any connections to this interface whitelisted",
          "whitelist": true,
          "_comment_connections": "Maximum number of connections this interface can have (or for each interface if no interface is specified) ",
          "connections": 10
        },
        "rpc": {
          "_comment_enabled": "enables the json-rpc interface on this address",
          "enabled": true,
          "port": 8332,
          "timeout": 30,
          "connections": 10,
          "_comment_allow": "Array of ip addresses or subnets allowed to access the JSON-RPC interface. Leave the array empty to disable this function or to let the OS handle this",
          "allow": [
            "127.0.0.1"
          ]
        }
      },
      {
        "_comment_address": "A null address indicates all available network interfaces",
        "address": "[::1]",
        "p2p": {
          "_comment_enable": "Enabling a p2p binding allows outside connections",
          "enabled": true,
          "port": 8333,
          "_comment_whitelist": "Enabling this makes any connections to this interface whitelisted",
          "whitelist": true,
          "_comment_connections": "Maximum number of connections this interface can have (or for each interface if no interface is specified) ",
          "connections": 10
        },
        "rpc": {
          "_comment_enabled": "enables the json-rpc interface on this address",
          "enabled": true,
          "port": 8332,
          "timeout": 30,
          "connections": 10,
          "_comment_allow": "Array of ip addresses or subnets allowed to access the JSON-RPC interface. Leave the array empty to disable this function or to let the OS handle this",
          "allow": [
            "[::1]"
          ]
        }
      }
    ]
  },
  "_comment_nodes": "P2P Nodes that you trust implicitely and wish to connect with on startup and stay connected with",
  "nodes": [
    {
      "_comment_enabled": "Set to false to ignore this entry",
      "enabled": false,
      "_comment_address": "IPv4 or IPv6 (if supported by your system)",
      "address": "69.164.218.197",
      "port": 8333,
      "_comment_getpeers": "Set to true if you wish to ask this node for lists of other suggested peers to connect to",
      "getpeers": true
    },
    {
      "_comment_enabled": "Set to false to ignore this entry",
      "enabled": true,
      "_comment_address": "IPv4 or IPv6 (if supported by your system)",
      "address": "10.0.0.2",
      "port": 8333,
      "_comment_getpeers": "Set to true if you wish to ask this node for lists of other suggested peers to connect to",
      "getpeers": false
    }
  ],
  "_comment_rpc": "Configuration options for any JSON-RPC services listening on interfaces",
  "rpc": {
    "_comment_users": "User authentication information for remote connections to the JSON-RPC port",
    "users": [
      {
        "enabled": false,
        "user": "alice",
        "password": "f7efda5c189b999524f151318c0c86$d5b51b3beffbc02b724e5d095828e0bc8b2456e9ac8757ae3211a5d9b16a22ae"
      },
      {
        "enabled": false,
        "user": "bob",
        "password": "b2dd077cb54591a2f3139e69a897ac$4e71f08d48b4347cf8eff3815c0e25ae2e9a4340474079f55705f40574f4ec99"
      }
    ]
  },
  "transactions": {
    "_comment_relaying": "Criteria for any seen transactions to be relayed to other P2P nodes",
    "relaying": {
      "_comment_min_fees": "Minimum total fee in Satoshi the transaction must have to be relayed",
      "min_total": 100,
      "_comment_min_per_byte": "Minimum satoshis per byte the transaction must have to be relayed",
      "min_per_byte": 10,
      "_comment_rbf_transactions": "Allow the relaying of Replace-By-Fee transactions",
      "rbf_transactions": false,
      "_comment_doublespends": "Allow the relaying of doublespend transactions. This might help merchants see doublespend attempts more quickly",
      "doublespends": true
    },
    "creating": {
      "fees": {
        "_comment_method": "Method to use to calculate fees for crafted transactions",
        "method": "dynamic",
        "_comment_dynamic": "Dynamic fees are calculated by analyzing fees of transactions in the mempool and recent blocks",
        "dynamic": {
          "_comment_per_byte_min": "Allows you to override the minimum satoshis per byte",
          "per_byte_min": 0,
          "_comment_per_byte_max": "Allows you to override the maximum satoshis per byte",
          "per_byte_max": 500,
          "_comment_block_priority": "Calculate fees to try to get your transaction included in the next X blocks",
          "block_priority": 1
        },
        "_comment_size": "Calculate the fee based entirely on a static number of satoshis given for every byte of size the transaction uses",
        "size": {
          "per_byte_fee": 10
        },
        "_static_size": "Set a static fee regardless of the size the transaction or how congested the network is",
        "static": {
          "fee": 10
        }
      }
    },
    "addresses": {
      "keypool": 100
    },
    "ui": {
      "enabled": true,
      "minimize": false,
      "tray": true
    }
  }
}

Polve commented 7 years ago

While a JSON format would definitely be an improvement it has the problem that it does not support comments.

I would suggest the YAML standard: it's easy to understand and well supported, has the same flexibilty of JSON.

DeftNerd commented 7 years ago

Frankly, we could bikeshed all the different formats (I agree that YAML might be the best format btw), but what I was hoping to do first was start a conversation on how people felt about changing the config file format and about supporting arrays of values and nested arrays for clarity and future flexibility.

It's also important to think about anything it might break or what it might allow.

For instance, I was about to do some research to see if there are any reasons to propose enhancing the RPC access permissions later. A more flexible config file format could enable the setting of specific ip ranges on a per-user level or to support alternative/additional authentication methods or to specify permissions users have to different functions.

I was being a bit idealistic in wanting JSON since JSON-RPC uses JSON too... but I agree that the lack of commenting ability is a big hindrance. Human JSON might be the best of both worlds if YAML isn't acceptable for some reason.

christophebiocca commented 7 years ago

IIRC one of the nice things about the current format is that command-line syntax and config syntax map trivially: if -option=value works on the command line, then option=value works in the config file.

Are there existing options that would benefit from list/dict support? It would help make this discussion more concrete.

DeftNerd commented 7 years ago

IIRC one of the nice things about the current format is that command-line syntax and config syntax map trivially

That is a nice benefit that would be lost. I'm sure it's used during testing and debugging, but I don't think it's used much in production.

I might be mistaken with some of these, but I believe that they are true:

Currently because of the command-line and config file limitations, you have to bind the RPC port to a specific interface or to all interfaces but are not able to bind it to a subset of interfaces. This makes it impossible to bind the RPC port to localhost and an internal network port or two, but not bind to the external network port.

It also prevents selection of different port numbers for different interfaces.

I can see a use case where someone might have a single server running Bitcoin and Bitcoin Cash. eth0 on an internal network, eth1 being 1.2.3.4 on the internet and eth2 being 1.2.3.5 on the internet. They might want to bind Bitcoin to eth0 on 8332 and eth1 on 8332. And they might want to bind Cash to eth0 on 4332 and 8332 on eth2.

There are ways around this with IPTables but it would be nice to be able to do it natively.

Same with RPC authentication granularity. It might be nice to be able to specify RPC Auth details that's different for different interfaces so machines accessing your Bitcoin server on the internal network use credentials that cannot be used on the external network interface. Not super useful until/unless there is the ability to restrict users to certain portions of the JSON-RPC api based on permissions.

You also can't set max connections on a granular level. You might want more connections on an internal network or IPv6 network than a different network.

Another feature that would be nice would be the ability to specify the proxy config just for certain interfaces or to exclude addresses. You don't need to go through Tor for local network connections, for instance.

JaredR26 commented 7 years ago

What about HOCON? It works for both things and it works as a superset of json rules. Also allows commenting.

IMO this isn't the kind of change we'd want to consider until mid next year.

jgarzik commented 7 years ago

I like hjson.

In terms of simplicity, we already support JSON; just need some code to pre-process the input lines and strip out comments matching /^\s*\/\//

Definitely a lower priority change, but agree with the motivation -- it creates a more readable and modular config file.

DeftNerd commented 7 years ago

It's not a big thing, to be honest, but I'm hoping to find ways to help us stand out from Bitcoin Core.

There are lots of ideas that Bitcoin Core didn't care about or messed up their business models. Now that it's plausible that this will become the dominant repo if the Segwit2X hashpower continues to hold, features can be added, replaced, removed, or improved. Core was stubborn about admitting mistakes or accepting improvements if it didn't come from within their camp.

Is there a new BIP process for BTC1 in motion yet? The Bitcoin-dev mailing list as a gateway to the /bitcoin/bips repo suffers from politicization and heavy-handed moderation. If not, I would like to offer to get involved in that. I have some ideas on how to make a more open but an idea exchange and proposal clearing house that would be useful to all cryptocurrencies, not just Bitcoin.

I know this is off-topic for this thread, but some examples of what I'm talking about

Easier configuration and more granular configuration options will help and impress sysadmins and devs
Pre-made docker containers, deployment scripts for cloud hosters, AWS AMI
Not just signing the binary, but signing a blockchain archive up to the fork block and releasing it (torrent?) for quicker installation.
Ability to pass a command line argument to use the fork checkpoint block as the point of block validation when setting up new nodes.
better database choices for improved use of HD space or optimized access speeds (maybe giving the user the choice of which metric is important during initial chain processing if those things contradict)
ability to prune blocks to secondary storage
adding the ability to get balances from an address to the JSON-RPC API (like bcoin does if the optional index-address database is built)
Adding child-pays-fee-for-parent transactions and/or finding a way to extend BIP70 (payment requests) so a merchant can include some kind of signed identifier string that helps a miner identify that the transaction is for the merchant. It gives merchants the ability to strike deals with miners to give their incoming transactions high priority without making the customer pay the higher transaction fee.
Undoing the fee-based prioritization of transactions and adding additional data points to come up with a better formula to score transactions.
Discussing adding an op_code to bitcoin to enable cross-chain fund locking and unlocking. This would allow users to (for example) send 1 BTC to an address or contract on the Ethereum blockchain and lock the funds on the BTC chain. The funds could be unlocked on the Eth side by a similar mechanism. It would allow for the purchase of Eth based assets with BTC and Eth based assets paying BTC out without an exchange having to get involved.

jgarzik commented 7 years ago

Definitely agreed, in general. I've been working on the docker angle myself, in fact...

christophebiocca commented 7 years ago

Is there a new BIP process for Bitcoin Cash in motion yet?

Just to be 100% clear this project (btc1) doesn't follow the Bitcoin Cash chain. In case anyone else reads this thread and gets confused.

DeftNerd commented 7 years ago

Just to be 100% clear this project (btc1) doesn't follow the Bitcoin Cash chain. In case anyone else reads this thread and gets confused.

Oh god, you're right. I'm so sorry. I'm going to make some edits to clarify. My intentions are identical for both Segwit2X and BCH though.... to help foster a new community open to change and growth and to consider previously politically "verboten" topics with technical merit.

nshopik commented 7 years ago

I like gobgp approach they are support JSON/YAML/TOML since they are easily converted, so it matter of your personal preference in what language you can write configs.

zquestz commented 7 years ago

Just so people are aware, there are Docker images for all Bitcoin implementations at https://github.com/amacneil/docker-bitcoin

I have personally contributed one for Segwit2X. Feel free to poach. =)

Here is where you can grab it on Docker Hub. https://hub.docker.com/r/zquestz/bitcoin-btc1/

zquestz commented 7 years ago

Also for the config files... I would love it to be UCL (https://github.com/vstakhov/libucl). This can read JSON, but allows for a user friendly syntax that doesn't care about quotes and commas. Also allows for imports. =)

argon2 commented 7 years ago

You can port the config.dat to use JSON using boost::property_tree in about 2 days of coding without any additional dependencies.

http://www.boost.org/doc/libs/1_61_0/doc/html/property_tree.html

btc1 / bitcoin