Models with an invalid config do not show up as errors

Follow-up and generalization of https://github.com/neuronsimulator/nrn-modeldb-ci/pull/132#discussion_r1775373769.

If using a model with an explicitly specified, but invalid (non-existing) model_dir, the exit code of the runmodels command is 0, but there is an error in the logs:

Traceback (most recent call last):
  File modeldb/modelrun.py, line 300, in run_model
    raise Exception(
Exception: Explicitly specified model_dir %model_dir%/<dir> does not exist

This is a rather insidious bug because the CI can quietly pass, and without looking at the full logs it's not at all obvious whether there are errors anywhere. Note that the bug is not specific to model_dir, other parts of the config are affected as well (see below).

Expected behavior

The runmodels command should return a non-zero code when an invalid config is specified for a given model, and, to save time, this should be mentioned explicitly in the CI logs, not just the artifacts of the job.

Affected models

The following models are affected:

33975

YAML config:

33975:
    # The "interesting" feature of this model is that it produces an error message, roughly:
    #   We were unable to associate a PlayRecord item with a RANGE variable
    # whose format changed after 8.2.2, and which went away in the data structure PR #2027.
    # The error going away caused an (empty) gout file to be written, which also caused a diff.
    # To avoid false positives, we regex away the error + traceback, and never produce a gout.
    curate_patterns:
    - pattern: 'hoc_run1: caught exception: hoc_execerror: We were unable to associate a PlayRecord item with a RANGE variable'
      repl: ''
    - pattern: '%neuron-executable%: We were unable to associate a PlayRecord item with a RANGE variable'
      repl: ''
    - pattern: ' in %model_dir%/driver.hoc near line \d+'
      repl: ''
    - pattern: '^\s+(finitialize|init|run|stdinit)\((|-65)\)'
      repl: ''
    - pattern: '^\s+\^'
      repl: ''
    # this should be removed once the latest release and development branch include nrn#2027
    ignore_exit_code: true
    run:
    - ringperf()
    - run()

Error log:

    "33975": {
        "gout": [],
        "ignore_exit_code": true,
        "logs": [
            "Traceback (most recent call last):",
            "  File \"/home/runner/work/nrn-modeldb-ci/nrn-modeldb-ci/modeldb/modelrun.py\", line 318, in run_model",
            "    raise Exception(",
            "Exception: Found multiple possible .mod file directores, please configure the correct subset: ['.', 'rings']",
            ""
        ],
        "nrn_run": [
            "RUNNING -> ./x86_64/special -nobanner %model_dir%/mosinit.hoc %model_dir%/driver.hoc",
            "Traceback (most recent call last):",
            "  File \"/home/runner/work/nrn-modeldb-ci/nrn-modeldb-ci/modeldb/modelrun.py\", line 366, in run_model",
            "    run_neuron_cmds(model, model_run_cmds)",
            "  File \"/home/runner/work/nrn-modeldb-ci/nrn-modeldb-ci/modeldb/modelrun.py\", line 124, in run_neuron_cmds",
            "    sp = subprocess.Popen(",
            "  File \"/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/subprocess.py\", line 951, in __init__",
            "    self._execute_child(args, executable, preexec_fn, close_fds,",
            "  File \"/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/subprocess.py\", line 1837, in _execute_child",
            "    raise child_exception_type(errno_num, err_msg, err_filename)",
            "FileNotFoundError: [Errno 2] No such file or directory: './x86_64/special'",
            ""
        ]

144538

YAML config:

144538:
    curate_patterns:
    - pattern: 't=([0-9\.]+);(\d+)\([0-9.\-e]+\)'
      repl: 't=\1;\2(%some_kind_of_clock%)'
    - pattern: '%neuron-executable%: Assignment to modern physical constant \w+'
      repl: ''
    - pattern: ' in .*\.hoc near line \d+'
      repl: ''
    - pattern: ' FARADAY.*'
      repl: ''
    - pattern: '\s*\^\s*'
      repl: ''
    run: null
    script:
    - sed -i'.bak' -r 's/(tstop=mytstop=htmax= \(LearnDur \+ ZipDur \+ BaseDur\*2\) \*) 1e3/\1 2e1/g' params.hoc

Error log:

    "144538": {
        "gout": [],
        "logs": [
            "",
            "Traceback (most recent call last):",
            "  File \"/home/runner/work/nrn-modeldb-ci/nrn-modeldb-ci/modeldb/modelrun.py\", line 285, in run_model",
            "    model[\"comment\"]",
            "KeyError: 'comment'",
            ""
        ],

neuronsimulator / nrn-modeldb-ci