Enhance settings.xml to better support remote and clustered MongoDB

At present, settings.xml files are used, during build, to supply values that are then injected into files such as configuration.yaml and 'storage-configuration.yaml.

When configuring MongoDB, there are several places where settings need to be defined:

in configuration.yaml in order to define the database settings for the catalog database
in storage-configuration.yaml in order to define the database settings for alignment database
in storage-configuration.yaml in order to define the database settings for the variant database

Across these three locations, where we need to inject details about MongoDB, there is mixed support for injecting values into these locations from settings.xml:

For the catalog, we can define the following place holders...

  database:
    hosts:
    - ${OPENCGA.CATALOG.DB.HOSTS}
    user: ${OPENCGA.CATALOG.DB.USER}
    password: ${OPENCGA.CATALOG.DB.PASSWORD}
    options:
      authenticationDatabase: ${OPENCGA.CATALOG.DB.AUTHENTICATION_DATABASE}
      connectionsPerHost: ${OPENCGA.CATALOG.DB.CONNECTIONS_PER_HOST}

... but note that sslEnabled can't be set (understandable, as this is a new option).

For the alignment database, fewer MongoDB settings can be injected from settings.xml (no options):

database:
      hosts:
      - "${OPENCGA.STORAGE.ALIGNMENT.DB.HOSTS}"
      user: "${OPENCGA.STORAGE.ALIGNMENT.DB.USER}"
      password: "${OPENCGA.STORAGE.ALIGNMENT.DB.PASSWORD}"
      options: {}

for the variant database, a similar set to the catalog are settable from settings.xml:

      hosts:
      - "${OPENCGA.STORAGE.VARIANT.DB.HOSTS}"
      user: "${OPENCGA.STORAGE.VARIANT.DB.USER}"
      password: "${OPENCGA.STORAGE.VARIANT.DB.PASSWORD}"
      options:    ## This is intended for database specific options such as --authenticationDatabase in MongoDB
        authenticationDatabase: ${OPENCGA.STORAGE.MONGODB.VARIANT.DB.AUTHENTICATION_DATABASE}
        connectionsPerHost: ${OPENCGA.STORAGE.MONGODB.VARIANT.DB.CONNECTIONS_PER_HOST}
        readPreference: "secondaryPreferred"

There should be a consistent set of values, across these three MongoDB databases, that can be defined in settings.xml and injected into the YAML config files. This set should include sslEnabled.

opencb / opencga

Enhance settings.xml to better support remote and clustered MongoDB #977