move most of the configuration from secrets to code

[garbas]

needs #977 and #992 first

currently we (ab)use secrets to also configure our projects. it would be nice to have layered solution where most of the configuration will actually be in code while still have the possibility to adjust it after deployment via taskcluster secrets.

How this layer configuration will work?

• a configuration is a JSON file of the following structure

  {
    "common": {
       "option-A": "value-A"
    },
    "project-A": {
      "option-A": "value-B"
    },
    "project-B": {},
    "project-C": {
      "option-B": "value-A"
    }
  }

  it allows us to have common configuration shared by all projects, but still have per project specification. in above example we define configuration for 3 projects and it gets merged as following:

  • project-A

    {
      "option-A": "value-B"
    }

  • project-B

    {
      "option-A": "value-A"
    }

  • project-C

    {
      "option-A": "value-A",
      "option-B": "value-A"
    }

• apart of having common section in a configuration file this PR would like to introduce a concept of layers. initially we would have 2 layers (in-repository and in-takcluster-secrets layer), but we might add more in the future if needed). Each layer specifies an above defined configuration file.

  • in-repository layer

  This layer leaved alongside a project and is defined in Nix. before running ./please command this configuration is exported by builtins.toJSON. Details of this will be done as part of #977

  Example:

  {
    "option-A": "value-A",
    "option-C": "value-A"
  }

  • in-taskcluser-secrets layer

  We are already using this layer. We are actually only using this layer. Mostly this layer will be used to specify secrets, but I can also see it when we need to change - temporary - some configuration options in production, without doing a release.

  Example:

  {
    "option-A": "value-B",
    "option-B": "very-secret"
  }

How would be configuration in layers merged?

In each layer configuration for a project-A will be evaluated. Merging would happen in order which is in-repository, in-tasklcuster-secrets. Giving above example an end configuration would be:

{
  "option-A": "value-B",
  "option-B": "very-secret",
  "option-C": "value-A"
}

[La0]

Looks good, but we should use YAML as Taskcluster is now using it for secrets.