Proposal for new rules file

[samrocketman]

To make your rules file less verbose you can abstract ruby specific things into more generic concepts and then translate them into Ruby at runtime. For instance, defining your types:

• Multitype vs a single type. A multitype could be e.g. multiple types like a list or a map or a string for a single key. The multitype could have some additional args with it.

When defining the other types; think more along the lines of the YAML spec (as opposed to ruby built-ins). Other types include:

• Object (i.e. a Hash)
• List (Array)
• String (...)
• Number (int or double, etc)
• Boolean (TrueClass or FalseClass)

Other things to think about with a redesign of the spec: key dependencies. Some keys may be dependent on other keys. You should be able to specify when a key depends on another key (and when validating the other key use the other key's rules normally).

[samrocketman]

Types of required existence:

1. One key or a different key (either or are okay but at least one of them must exist).
2. Other key must key exist (only a single type)
3. Optional keys may only require keys if the optional key exists. In that case, the other key may or may not need to exist; but only if the optional key does exist.

Case 1

Let's say you have the following yaml file.

foo:
  default_baz: hurdurr
  bar:
    - baz: durrhurr
      pizza:
        - large

Let's say pizza key has a dependency on the baz key. However, the baz key can have another place where defaults are defined (i.e. default_baz). We need to handle this case. Here's some validation examples.

Valid (baz exists for pizza):

foo:
  bar:
    - baz: durrhurr
      pizza:
        - large

Valid (although baz doesn't exist, its default_baz does):

foo:
  default_baz: hurrdurr
  bar:
    - pizza:
        - large

Not Valid (pizza missing baz and there's no default defined 😢 ):

foo:
  bar:
    - pizza:
        - large

Case 2

Let's say you have a key that depends on another key. Let's say foo key is required but has a dependency on another key bar. foo key exists then bar must exist.

Valid:

foo: baz
bar: baz

Not Valid:

foo: baz

Case 3

Let's say secrets_id (which is a String) and secrets (which is a List of Hashes) are both keys that are optional in a YAML spec. However, if secrets exists then secrets_id must exist because secrets depends on it. With that in mind:

Valid (no content because both keys are optional):

Valid (secrets key is optional):

secrets_id: foo

Valid (both optional keys are present):

secrets_id: foo
secrets:
  - ...

Not Valid (when the optional key secrets exists; secrets_id is a requirement):

secrets:
  - ...

Therefore, the rule to determine whether or not secrets_id is a requirement should solely live with the rules associated with secrets key.

[samrocketman]

If you decide to adopt what I call "addressing" for a YAML or JSON object, then you could do something like this for Case 1. The concept of "addressing" for YAML was invented by me for the Jervis project and Sandscape project (see getObjectValue() and setObjectValue() methods).

foo:
  default_baz: hurdurr
  bar:
    - baz: durrhurr
      pizza:
        - large

For key foo.bar[*].pizza, relative to the pizza key: baz or ../../default_baz are required.

[skippyPeanutButter]

yaml

language: go
jdk: oraclejdk7
install:
  - "sudo yum clean all"
  - "echo 'y' | android update sdk --no-ui --all --filter build-tools-23.0.3"
  - "sudo yum -y install glibc.i686 libz.so.1 libgcc libgcc_s.so.1"

script:
  - export tpsReleaseSigningStoreFile=${WORKSPACE}/keystore/tpsrelease.keystore.jks
  - export debugSigningStoreFile=${WORKSPACE}/keystore/test-release.keystore.jks
  - export prodReleaseSigningStoreFile=${WORKSPACE}/keystore/prodrelease.keystore.jks
  - export debugKeyPassword=xfhome
  - export debugStorePassword=xfhome
  - export debugKeyAlias=xfhrel
  - ./gradlew antHillBuild --info

jenkins:
  use_gerrit: false
  sudo: true
  collect:
    artifacts:
      - "**/App/build/**/*.apk"
      - "**/reports/tests/**/index.html"

Rules

root_keys:
  optional:
    - language:
        accepted_types:
          - String
        values:
          - 'android'
          - 'go'
          - 'python'
          - 'ruby'
    - before_install:
        accepted_types:
          - String
          - Array
    - install:
        accepted_types:
          - String
          - Array
    - before_script:
        accepted_types:
          - String
          - Array
    - script:
        accepted_types:
          - String
          - Array
    - branches:
        subkeys:
          optional:
            - only:
                accepted_types:
                  - Array
            - except:
                accepted_types:
                  - Array
    - jenkins:
        subkeys:
          optional:
            - stage:
                accepted_types:
                  - Array
            - sudo:
                accepted_types:
                  - TrueClass
                  - FalseClass
            - use_gerrit:
                accepted_types:
                  - TrueClass
                  - FalseClass
            - collect:
                subkeys:
                  optional:
                    - artifacts:
                        accepted_types:
                          - String
                          - Array

[samrocketman]

Generic rules definition.

defaults:
  <default values for rules>
rules:
  - key: 'yaml address'
    required_key: true|false
    and_requires: ['list of yaml addresses']
    or_requires: ['list of yaml addresses']
    value_whitelist: ['list of values']
    value_blacklist: ['list of values']
    types: ['list of types']

By defining defaults for all rules, you can greatly reduce the amount of information you have to provide for each rule. It helps reduce duplication. By using YAML addressing, you can reduce the rules into a flat list. This greatly reduces the complexity and provides an order in which rules are evaluated.

defaults:
  required_key: false
  types: ['String', 'List']
rules:
  - key: 'language'
    required_key: true
    types: ['String']
    value_whitelist: ['android', 'go', 'python', 'ruby']
  - key: 'env'
    types: ['String', 'List', 'Object']
  - key: 'env.global'
  - key: 'env.matrix'
  - key: 'before_install'
  - key: 'install'
  - key: 'before_script'
  - key: 'script'
  - key: 'branches.only'
    types: ['List']
  - key: 'branches.except'
    types: ['List']
  - key: 'jenkins.sudo'
    types: ['Boolean', 'String']
  - key: 'jenkins.use_gerrit'
    types: ['Boolean', 'String']
  - key: 'jenkins.collect.artifacts'

[skippyPeanutButter]

Implemented in PR https://github.com/skippyPeanutButter/yaml_bot/pull/25