Evaluation failed because of missing dynatrace.conf makes WaitForKeptnEvent step pass

[pchila]

Summary Triggering an evaluation using dynatrace monitoring without dynatrace.conf and waiting for the keptn event results in a passed step.

Pipeline step definition:

- task: WaitForKeptnEventTask@2
  inputs:
    keptnApiEndpoint: 'Cloud Automation Keco DEV'
    waitForEventType: 'evaluation'
    timeout: '3'
    keptnContextVar: 'keptnContext'
    markBuildOnError: 'FAILED'

Event received from Cloud Automation:

************* Result from Keptn ****************
{
  "data": {
    "evaluation": {
      "gitCommit": "",
      "indicatorResults": null,
      "result": "pass",
      "score": 0,
      "sloFileContent": "",
      "timeEnd": "2022-02-03T09:54:08.467Z",
      "timeStart": "2022-02-03T09:39:08.467Z"
    },
    "labels": {
      "buildId": "20220203.4",
      "runby": "Paolo Chila"
    },
    "message": "no evaluation performed by lighthouse because SLI failed with message cannot handle event: could not get configuration: could not find resource: 'dynatrace/dynatrace.conf.yaml' of project 'pchilafoobar2'",
    "project": "pchilafoobar2",
    "result": "fail",
    "service": "fooservice",
    "stage": "dev",
    "status": "succeeded"
  },
  "id": "53ae4f1a-272e-487a-b435-1a6e11c2484a",
  "source": "lighthouse-service",
  "specversion": "1.0",
  "time": "2022-02-03T09:54:08.945Z",
  "type": "sh.keptn.event.evaluation.finished",
  "shkeptncontext": "9141ad0f-8cdc-485f-a7cd-91f6341e8a0e",
  "shkeptnspecversion": "0.2.3",
  "triggeredid": "d01268e9-3415-4732-b9a7-0d14feea0110"
}

WaitForKeptnEvent step is green in this case while the whole sequence is red on keptn UI

Environment

• Keptn Version (keptn version): 0.12.0
• Kubernetes Cloud Provider (e.g., GKE, AKS): N/A
• Kubernetes version (kubectl version): N/A
• Client OS (e.g., Linux, macOS, Windows): N/A
• Client Browser: N/A

Affected Component

• [ ] Web UI / Bridge
• [ ] CLI
• [ ] REST API or webhooks
• [ ] User experience
• [ ] Developer experience
• [ ] Deployment/manegement with Helm

To Reproduce

Steps to reproduce the behavior:

1. Trigger an evaluation from azure devops pipeline for a project configured with dynatrace monitoring without providing a dynatrace.conf file
2. Send a Keptn event to trigger evaluation
3. Wait for a Keptn event with setting markBuildOnError: 'FAILED'

Expected behavior Step fails because sequence is marked as failed in Cloud Automation UI

Current behavior Step succeeds

Screenshots

Additional context