configfile does not completely overwrite previous config files

[fgvieira]

Snakemake version 5.27.4

Describe the bug When there is a config file loaded and another one is loaded through --configfile, the config dict is updated and not overwritten.

Minimal example With a snakefile:

import json
configfile: "config.yaml"
print(json.dumps(config, indent="\t"))

and a config file:

ref:
  n_chromosomes: 17
  species: saccharomyces_cerevisiae
  release: 100
  build: R64-1-1
  url: ensembl.org

you get (with some errors that are not relevant for here):

$ snakemake -j 1
{
        "ref": {
                "n_chromosomes": 17,
                "species": "saccharomyces_cerevisiae",
                "release": 100,
                "build": "R64-1-1",
                "url": "ensembl.org"
        }
}

but, if you also load this config through the command line:

ref:
  n_chromosomes: 17
  species: saccharomyces_cerevisiae
  release: 101
  build: R64-1-1
  name: NEW

you get:

$ snakemake -j 1 --configfile config_NEW.yaml
{
        "ref": {
                "n_chromosomes": 17,
                "species": "saccharomyces_cerevisiae",
                "release": 101,
                "build": "R64-1-1",
                "name": "NEW",
                "url": "ensembl.org"
        }
}

Shouldn't we expect the config to be fully overwritten (and the key url should not be there)? If not and snakemake actually merges config files (instead of overwriting them), is there a way to enforce the old one to be overwritten?

thanks,

[sroener]

Shouldn't we expect the config to be fully overwritten (and the key url should not be there)?

No, as the documentation states:

Finally, you can use the –configfile command line argument to overwrite values from the configfile statement. Note that any values parsed into the config dictionary with any of above mechanisms are merged, i.e., all keys defined via a configfile statement, or the –configfile and –config command line arguments will end up in the final config dictionary, but if two methods define the same key, command line overwrites the configfile statement.

But I think that an option to overwrite a config file completely could be useful in some cases.

[johanneskoester]

Exactly. PRs for an additional flag to modify the behavior in this regard are welcome.