Structured output improvement

[eyarz]

Is your feature request related to a problem? Please describe. We support JSON and YAML output but the structure of the output is not "developers/machine/pipe-line friendly" enough

Describe the solution you'd like The output structure should be documented, clear, and consistent.

Additional context YAML output: datree test manifest.yaml -o yaml JSON output: datree test manifest.yaml -o json Related to https://github.com/datreeio/helm-datree/issues/5#issuecomment-892128126

[eyarz]

here is a draft of how the structured output should look like:

yamlValidationResults:
  - fileName: /mnt/d/Dev/datree/internal/fixtures/kube/bad_yaml.yaml
    yamlErrors: "yaml: line 2: mapping values are not allowed in this context"
k8sValidationResults:
  - fileName: /mnt/d/Dev/datree/internal/fixtures/kube/Chart.yaml
    schemaErrors: "error while parsing: missing 'kind' key"
  - fileName: /mnt/d/Dev/datree/internal/fixtures/kube/invalidK8sSchema.yaml
    schemaErrors: "For field (root): Additional property apiversion is not allowed"
policyValidationResults:
  - fileName: /mnt/d/Dev/datree/internal/fixtures/kube/fail-30.yaml
    ruleRresults:
      - identifier: CONTAINERS_MISSING_IMAGE_VALUE_VERSION
        name: Ensure each container image has a pinned (tag) version
        messageOnFailure: Incorrect value for key `image` - specify an image version to avoid unpleasant "version surprises" in the future
        occurrencesDetails:
          - metadataName: static-web
            kind: Pod
          - metadataName: hello
            kind: CronJob
      - identifier: CONTAINERS_MISSING_MEMORY_REQUEST_KEY
        name: Ensure each container has a configured memory request
        messageOnFailure: Missing property object `requests.memory` - value should be within the accepted boundaries recommended by the organization
        occurrencesDetails:
          - metadataName: hello
            kind: CronJob
  - fileName: /mnt/d/Dev/datree/internal/fixtures/kube/nginxDeployment.yaml
    ruleRresults:
      - identifier: CONTAINERS_MISSING_MEMORY_REQUEST_KEY
        name: Ensure each container has a configured memory request
        messageOnFailure: Missing property object `requests.memory` - value should be within the accepted boundaries recommended by the organization
        occurrencesDetails:
          - metadataName: nginx-deployment
            kind: Deployment
policySummary:
  policyName: Default
  totalRulesInPolicy: 18
  totalRulesFailed: 31
  totalPassedCount: 2
evaluationsummary:
  filesCount: 7
  configsCount: 15
  passedYamlvaliVationCount: 7
  passedK8sValidationCount: 5
  passedPolicyValidationCount: 2

[eyarz]

and I also prepared a JSON Schema for the new output:

{
    "definitions": {},
    "$schema": "http://json-schema.org/draft-07/schema#", 
    "$id": "https://example.com/object1635619920.json", 
    "title": "Root", 
    "type": "object",
    "required": [
        "yamlValidationResults",
        "k8sValidationResults",
        "policyValidationResults",
        "policySummary",
        "evaluationsummary"
    ],
    "properties": {
        "yamlValidationResults": {
            "$id": "#root/yamlValidationResults", 
            "title": "Yamlvalidationresults", 
            "type": "array",
            "default": [],
            "items":{
                "$id": "#root/yamlValidationResults/items", 
                "title": "Items", 
                "type": "object",
                "required": [
                    "fileName",
                    "yamlErrors"
                ],
                "properties": {
                    "fileName": {
                        "$id": "#root/yamlValidationResults/items/fileName", 
                        "title": "Filename", 
                        "type": "string",
                        "default": "",
                        "examples": [
                            "/mnt/d/Dev/datree/internal/fixtures/kube/bad_yaml.yaml"
                        ],
                        "pattern": "^.*$"
                    },
                    "yamlErrors": {
                        "$id": "#root/yamlValidationResults/items/yamlErrors", 
                        "title": "Yamlerrors", 
                        "type": "string",
                        "default": "",
                        "examples": [
                            "yaml: line 2: mapping values are not allowed in this context"
                        ],
                        "pattern": "^.*$"
                    }
                }
            }

        },
        "k8sValidationResults": {
            "$id": "#root/k8sValidationResults", 
            "title": "K8svalidationresults", 
            "type": "array",
            "default": [],
            "items":{
                "$id": "#root/k8sValidationResults/items", 
                "title": "Items", 
                "type": "object",
                "required": [
                    "fileName",
                    "schemaErrors"
                ],
                "properties": {
                    "fileName": {
                        "$id": "#root/k8sValidationResults/items/fileName", 
                        "title": "Filename", 
                        "type": "string",
                        "default": "",
                        "examples": [
                            "/mnt/d/Dev/datree/internal/fixtures/kube/invalidK8sSchema.yaml"
                        ],
                        "pattern": "^.*$"
                    },
                    "schemaErrors": {
                        "$id": "#root/k8sValidationResults/items/schemaErrors", 
                        "title": "Schemaerrors", 
                        "type": "string",
                        "default": "",
                        "examples": [
                            "For field (root): Additional property apiversion is not allowed"
                        ],
                        "pattern": "^.*$"
                    }
                }
            }

        },
        "policyValidationResults": {
            "$id": "#root/policyValidationResults", 
            "title": "Policyvalidationresults", 
            "type": "array",
            "default": [],
            "items":{
                "$id": "#root/policyValidationResults/items", 
                "title": "Items", 
                "type": "object",
                "required": [
                    "fileName",
                    "ruleRresults"
                ],
                "properties": {
                    "fileName": {
                        "$id": "#root/policyValidationResults/items/fileName", 
                        "title": "Filename", 
                        "type": "string",
                        "default": "",
                        "examples": [
                            "/mnt/d/Dev/datree/internal/fixtures/kube/fail-30.yaml"
                        ],
                        "pattern": "^.*$"
                    },
                    "ruleRresults": {
                        "$id": "#root/policyValidationResults/items/ruleRresults", 
                        "title": "Rulerresults", 
                        "type": "array",
                        "default": [],
                        "items":{
                            "$id": "#root/policyValidationResults/items/ruleRresults/items", 
                            "title": "Items", 
                            "type": "object",
                            "required": [
                                "identifier",
                                "name",
                                "messageOnFailure",
                                "occurrencesDetails"
                            ],
                            "properties": {
                                "identifier": {
                                    "$id": "#root/policyValidationResults/items/ruleRresults/items/identifier", 
                                    "title": "Identifier", 
                                    "type": "string",
                                    "default": "",
                                    "examples": [
                                        "CONTAINERS_MISSING_IMAGE_VALUE_VERSION"
                                    ],
                                    "pattern": "^.*$"
                                },
                                "name": {
                                    "$id": "#root/policyValidationResults/items/ruleRresults/items/name", 
                                    "title": "Name", 
                                    "type": "string",
                                    "default": "",
                                    "examples": [
                                        "Ensure each container image has a pinned (tag) version"
                                    ],
                                    "pattern": "^.*$"
                                },
                                "messageOnFailure": {
                                    "$id": "#root/policyValidationResults/items/ruleRresults/items/messageOnFailure", 
                                    "title": "Messageonfailure", 
                                    "type": "string",
                                    "default": "",
                                    "examples": [
                                        "Incorrect value for key `image` - specify an image version to avoid unpleasant \"version surprises\" in the future"
                                    ],
                                    "pattern": "^.*$"
                                },
                                "occurrencesDetails": {
                                    "$id": "#root/policyValidationResults/items/ruleRresults/items/occurrencesDetails", 
                                    "title": "Occurrencesdetails", 
                                    "type": "array",
                                    "default": [],
                                    "items":{
                                        "$id": "#root/policyValidationResults/items/ruleRresults/items/occurrencesDetails/items", 
                                        "title": "Items", 
                                        "type": "object",
                                        "required": [
                                            "metadataName",
                                            "kind"
                                        ],
                                        "properties": {
                                            "metadataName": {
                                                "$id": "#root/policyValidationResults/items/ruleRresults/items/occurrencesDetails/items/metadataName", 
                                                "title": "Metadataname", 
                                                "type": "string",
                                                "default": "",
                                                "examples": [
                                                    "hello"
                                                ],
                                                "pattern": "^.*$"
                                            },
                                            "kind": {
                                                "$id": "#root/policyValidationResults/items/ruleRresults/items/occurrencesDetails/items/kind", 
                                                "title": "Kind", 
                                                "type": "string",
                                                "default": "",
                                                "examples": [
                                                    "CronJob"
                                                ],
                                                "pattern": "^.*$"
                                            }
                                        }
                                    }

                                }
                            }
                        }

                    }
                }
            }

        },
        "policySummary": {
            "$id": "#root/policySummary", 
            "title": "Policysummary", 
            "type": "object",
            "required": [
                "policyName",
                "totalRulesInPolicy",
                "totalRulesFailed",
                "totalPassedCount"
            ],
            "properties": {
                "policyName": {
                    "$id": "#root/policySummary/policyName", 
                    "title": "Policyname", 
                    "type": "string",
                    "default": "",
                    "examples": [
                        "Default"
                    ],
                    "pattern": "^.*$"
                },
                "totalRulesInPolicy": {
                    "$id": "#root/policySummary/totalRulesInPolicy", 
                    "title": "Totalrulesinpolicy", 
                    "type": "integer",
                    "examples": [
                        18
                    ],
                    "default": 0
                },
                "totalRulesFailed": {
                    "$id": "#root/policySummary/totalRulesFailed", 
                    "title": "Totalrulesfailed", 
                    "type": "integer",
                    "examples": [
                        31
                    ],
                    "default": 0
                },
                "totalPassedCount": {
                    "$id": "#root/policySummary/totalPassedCount", 
                    "title": "Totalpassedcount", 
                    "type": "integer",
                    "examples": [
                        2
                    ],
                    "default": 0
                }
            }
        }
,
        "evaluationsummary": {
            "$id": "#root/evaluationsummary", 
            "title": "Evaluationsummary", 
            "type": "object",
            "required": [
                "filesCount",
                "configsCount",
                "passedYamlvaliVationCount",
                "passedK8sValidationCount",
                "passedPolicyValidationCount"
            ],
            "properties": {
                "filesCount": {
                    "$id": "#root/evaluationsummary/filesCount", 
                    "title": "Filescount", 
                    "type": "integer",
                    "examples": [
                        7
                    ],
                    "default": 0
                },
                "configsCount": {
                    "$id": "#root/evaluationsummary/configsCount", 
                    "title": "Configscount", 
                    "type": "integer",
                    "examples": [
                        15
                    ],
                    "default": 0
                },
                "passedYamlvaliVationCount": {
                    "$id": "#root/evaluationsummary/passedYamlvaliVationCount", 
                    "title": "Passedyamlvalivationcount", 
                    "type": "integer",
                    "examples": [
                        7
                    ],
                    "default": 0
                },
                "passedK8sValidationCount": {
                    "$id": "#root/evaluationsummary/passedK8sValidationCount", 
                    "title": "Passedk8svalidationcount", 
                    "type": "integer",
                    "examples": [
                        5
                    ],
                    "default": 0
                },
                "passedPolicyValidationCount": {
                    "$id": "#root/evaluationsummary/passedPolicyValidationCount", 
                    "title": "Passedpolicyvalidationcount", 
                    "type": "integer",
                    "examples": [
                        2
                    ],
                    "default": 0
                }
            }
        }

    }
}

[Abhra303]

@eyarz you defined the yamlErrors property of yamlValidationResults as string type. Should not this be an array instead? There may be multiple yaml errors in a file. Same for schemaErrors in k8sValidationResults.

[eyarz]

That's a good point! I checked it by introducing two yaml syntax errors in a single file:

-- # <- 1
apiVersion; apps/v1 # <- 2
kind: Deployment
metadata:
  name: rss-site
  namespace: test
  labels:
    owner: --
    env: prod
    app: web

Datree's output includes only one error message:

So you're right, it's supposed to be a list, but right now, it's a string. Let's proceed as it is (a string), and I will open a new issue and ask to update the output structure when resolving this issue.

I did the same thing with two K8s schema syntax errors:

apiVersion: apps/v1
kind: deployment # <- 1
metaData: # <- 2
  name: rss-site
  namespace: test
  labels:
    owner: --
    env: prod
    app: web

In this case, it looks like the output includes two errors, but as one string: So yes, it should be a list and not a string. Do you think it should be fixed as part of this issue, or should we open a new issue also for this?

[Abhra303]

Do you think this should be fixed as part of this issue, or should we open a new issue also for this?

I think we should open a new issue. Because, this issue is only about improving the output structure for non-interactive options (i.e. yaml, json and xml). So, the fix of this issue should only be limited to that.

[royhadad]

As @Abhra303 mentioned, the rule identifier was missing from the ruleResults API, I have added it in the following PR https://github.com/datreeio/datree/pull/352. the rule identifier can be accessed through EvaluationResult.Rule.Identifier 😊

[eyarz]

@Abhra303 FYI, #350 was already resolved and merged to main