Move flags to config file

[kderme]

We have recently added a big number of options https://github.com/input-output-hk/cardano-db-sync/blob/master/doc/configuration.md. These mostly cannot change between separate executions, which means the possibly they need to be part of the config file.

[kderme]

How this config could look like

{
tx-out: (on) 
      / consumed  {force_tx_in : (off) / on } 
      / prune     {force_tx_in : (off) / on } 
      / bootstrap {force_tx_in : (off) / on } 
      / off,
ledger: (on) / off / dont-use,
shelley : (on) / off / <stake_address_list>,
multiasset: (on) / off / <policies_list>,
metadata: (on) / off / <keys_list>,
plutus: (on) / off / <script_hash_list>
gov: (on) / off,
offchain-pool-data : (on) / off,
json_type : (text) / jsonb / no,
}

[kderme]

Steps to take for this

• Decide the exact format: json or yaml, use the existing config or add a new one
• Define a InsertConfig (?) Haskell type, part of SyncNodeParams with an aeson instance. Most of the fields are optional and have default values.
• Translate the InsertConfig type to the existing types InsertOptions and SyncOptions. These need to be extended. Some of the options will remain unused for now, but it's good to have them.

[sgillespie]

This is basically what I'm thinking: extend the db-sync-config file with a new property. This will allow users to use whichever format they prefer. The new property can be called insert_options (or insert_config), with the below pseudo json-schema:

 {
   "properties": {
         "tx_out": {
             "enum": ["enable", "disable", "consumed", "prune", "bootstrap"]
         },
         "ledger": {
             "enum": ["enable", "disable", "dont-use"]
         },
         "shelley": {
             "type": "object",
             "properties": {
                 "enable": {
                     type: "boolean"
                 },
                 "stake_addresses": {
                     "type": "array",
                 }
             }
         },
         "multi_asset": {
             "type": "object",
             "properties": {
                 "enable": {
                     type: "boolean"
                 },
                 "policies": {
                     "type": "array",
                 }
             }
         },
         "metadata": {
             "type": "object",
             "properties": {
                 "enable": {
                     type: "boolean"
                 },
                 "keys": {
                     "type": "array",
                 }
             }
         },
         "plutus": {
             "type": "object"
             "properties": {
                 "enable": {
                     type: "boolean"
                 },
                 "script_hashes": {
                     "type": "array",
                 }
             }
         },
         "governance": {
             "enum": ["enable", "disable"]
         },
         "offchain_pool_data": ["enable", "disable"],
         "json_type": ["text", "jsonb", "disable"]
     }
 }

[sgillespie]

Here's an example using the schema above:

{
  "tx_out": "prune",
  "ledger": "dont-use",
  "shelley": {
    "enable": true,
    "stake_addresses": ["seans-stake-address"]
  },
  "multi_asset": {
    "enable": true,
    "policies": ["seans-policy"]
  },
  "metadata": {
    "enable": true
    "keys": ["seans-key"]
  },
  "plutus": {
    "enable": true,
    "script_hash": ["seans-script-hash"]
  },
  "governance": "enable"
  "offchain_pool_data": "enable",
  "json_type": "text"
}

Thoughts?

[Cmdv]

@kderme was the idea that we always have to have every value in the schema present? looking at bellow

"metadata": {
    "enable": true
    "keys": ["seans-key"]
  },

for example the act of having "metadata" present, would make it enabled but if not then disabled. The working "metadata" type would turn from:

data MetadataConfig
  = MetadataEnable
  | MetadataDisable
  | MetadataKeys [ByteString]
  deriving (Eq, Show)

into:

data MetadataConfig
  = MetadataDisable
  | MetadataKeys [ByteString]
  deriving (Eq, Show)

secondly I'm slightly confused with ledger: (on) / off / dont-use, are dont-use and off the same here? our type representation right now is:

-- A representation of if we are using a ledger or not given CLI options
data LedgerEnv where
  HasLedger :: HasLedgerEnv -> LedgerEnv
  NoLedger :: NoLedgerEnv -> LedgerEnv

[sgillespie]

for example the act of having "metadata" present, would make it enabled but if not then disabled. The working "metadata" type would turn from

My thinking was "enable": true along with keys: [ ... ], would make it whitelisted. If keys are null or empty, we would fill in all metadata.

There's another option, we could have enable = true | false | whitelist-keys and in that case keys would only be interpreted if "enable": "whiitelist-keys"

[sgillespie]

secondly I'm slightly confused with ledger: (on) / off / dont-use, are dont-use and off the same here?

This corresponds to --dont-use-ledger: https://github.com/IntersectMBO/cardano-db-sync/blob/master/doc/configuration.md#--dont-use-ledger--experimental