As a system admin, I want simpler wres system configuration

[epag]

---

Author Name: James (James) Original Redmine Issue: 105153, https://vlab.noaa.gov/redmine/issues/105153 Original Date: 2022-05-26

---

Given a need to configure wres options, such as database options When I configure those options Then the configuration

• should be as simple/flat as possible
• involve as few files as possible
• and the options should be determined at runtime, not compile time

[epag]

---

Original Redmine Comment Author Name: James (James) Original Date: 2022-05-26T12:00:29Z

---

Current situation is (for the core module):

• wresconfig.xml
• eventsbroker.properties
• logging.properties
• logback.xml
• wres_jfr.jfc

Most of these are fine, but there is an opportunity to simplify @wresconfig.xml@ using a flat/properties file format (like the @eventsbroker.properties@) whose key-value pairs can be easily read into a map with no special xml processing or option-specific switch/ifelse statements, as witnessed in @SystemSettings.java@ and @DatabaseSettings.java@.

Crucially, all of these options should be determined at runtime, not compile time. Many of the database-specific options in @DatabaseSettings.java@ are determined at compile time. Hikari CP can consume a properties file or object on construction, so we can pass these properties through to the connection pool provider at runtime or deployment runtime. Likewise with the @DriverManager@ approach we are using for unpooled connections.

There is perhaps a further opportunity to combine all of the options in property-style files into one @wres.properties@, eliminating @wresconfig.xml@, @logging.properties@ and @eventsbroker.properties@.

[epag]

---

Original Redmine Comment Author Name: James (James) Original Date: 2022-05-27T10:50:38Z

---

Working in this area w/r to #103770, so I can see some things that are worth progressing now and others that can be left until later. Priority is to remove all the compile-time database configuration junk from source code to configuration code.

[epag]

---

Original Redmine Comment Author Name: James (James) Original Date: 2022-05-27T10:51:21Z

---

Probably 16 overall, but I am not going to expend that amount of effort now.

[epag]

---

Original Redmine Comment Author Name: James (James) Original Date: 2022-05-27T15:57:14Z

---

( Reflecting reality, I have been distracted by #104715 and many other things today. )

[epag]

---

Original Redmine Comment Author Name: James (James) Original Date: 2022-05-27T19:22:48Z

---

Well, it was a nice idea, but as I look more closely, this unravels quickly and it would require very extensive refactoring of the @SystemSettings@ and @DatabaseSettings@.

Another awkwardness is that any properties that are unique to a given db implementation need to be separated out from other wres properties because Hikari CP will not ignore settings that do not apply to a given db implementation, it will produce an error. In other words, it is not possible to achieve a single wres properties file that can also be passed directly through to Hikari CP for the relevant db properties to be set for the chosen db implementation.

Yuck. On hold indefinitely. But really, @SystemSettings@ and @DatabaseSettings@ do need some significant work.

Another thing to add to the list is that java system property overrides should be transparent. It is fine if they are qualified with @wres@, like @wres.foo=bar@, which should result in wres setting @foo@ having value @bar@, but it should not otherwise be necessary for the software to individually unpack these settings and hence to enumerate every single override, only the subset where we are offering some kind of api to a more general setting, such as @wres.databaseHost@, which could map to any number of db-implementation-specific host name settings, like @host@ or @serverName@ or whatever. Instead any java system property overrides should be used to mutate a general map of settings read from a configuration file.

In other words, something like this should be achievable for overriding properties read from a file that are not part of a more general api:

for ( String propertyName : properties.stringPropertyNames() ) {
    String systemPropertyValue = System.getProperty( propertyName );

    if ( systemPropertyValue != null ) {
        properties.setProperty( propertyName, systemPropertyValue );
    }
}
</code>

edit: as things stand, look at the horror of system property overrides in @DatabaseSettings@.

[epag]

---

Original Redmine Comment Author Name: James (James) Original Date: 2022-05-27T19:28:00Z

---

Oh, and for posterity, this is the set of database properties I'd put together before deciding this would unravel too quickly.

################################################################################ 
1.                         General database settings
################################################################################

user=wres_user
password=postgres
maxPoolSize=8
maxHighPriorityPoolSize=5
maxIdleTime=80
attemptToMigrate=true
connectionTimeoutMs=18000000

1. Options for databaseType as of v6.5: POSTGRESQL, H2, MYSQL, MARIADB, SQLITE
1. Postgres is supported and recommended.
1. H2 has experimental support for in-memory mode and is used in tests.
1. Code changes and testing are needed to support MySQL.
1. Code changes and testing are needed to support MariaDB.
1. Code changes and testing are needed to support SQLite.

databaseType=POSTGRESQL

################################################################################ 
1.                        PostgreSQL database settings
################################################################################

databaseName=wres
serverName=localhost
portNumber=5432
options=-c statement_timeout=18000s
prepareThreshold=2
ssl=true
sslfactory=wres.system.PgSSLSocketFactory
sslfactoryarg=classpath:dod_sw_ca-54_expires_2022-11.pem

################################################################################ 
1.                            H2 database settings
################################################################################

#jdbcurl=jdbc:h2:mem:test;MODE=PostgreSQL;TRACE_LEVEL_FILE=4;DB_CLOSE_DELAY=-1;INIT=create schema if not exists wres\;MAX_QUERY_TIMEOUT=18000000

################################################################################ 
1.                         MariaDB database settings
################################################################################

#port=5432
#host=localhost
#useServerPrepStmts=true
#useSSL=false
#verifyServerCertificate=true
#trustServerCertificate=false
#serverSslCert=classpath:dod_sw_ca-54_expires_2022-11.pem
#max_statement_time=18000

################################################################################ 
1.                              MySQL settings
################################################################################

#port=5432
#host=localhost
#useServerPrepStmts=true
#useSSL=false
#verifyServerCertificate=true
#trustServerCertificate=false
#serverSslCert=classpath:dod_sw_ca-54_expires_2022-11.pem
#max_statement_time=18000000

[epag]

---

Original Redmine Comment Author Name: James (James) Original Date: 2024-07-26T12:01:36Z

---

May be more achievable now. Also, should consider yaml as the language.