[Heartbeat] Monitor - Add ability for check an array of objects from a JSON payload response

[gchiartosini]

Describe the enhancement: Add the ability for check an array of objects from a JSON payload response.

Describe a specific use case for the enhancement or feature: I received a sample JSON payload containing an array of objects in my Heartbeat HTTP monitor: { "result": { "status": "OK" }, "something": [ { "item1": "value1", "item2": "value2" }, { "item1": "value3", "item2": "value4" } ] }

where the HTTP monitor option would be able to check response using something like: `check.response: status: [200] json:

• description: equals condition: equals: something[0].item1: "value1"`

or `check.response: status: [200] json:

• description: equals and exists condition: and:
  • equals: result.status: "OK"
  • has_fields: ['something[0].item1']`

Furthermore, the ability to verify the size of the array would also be useful.

[elasticmachine]

Pinging @elastic/uptime (Team:Uptime)

[jsoriano]

@gchiartosini would synthetics cover your use case? https://www.elastic.co/guide/en/observability/7.14/synthetic-monitoring.html

[gbschenkel]

I would like to heartbeat check objects on JSON array. I saw this subject keep been abandoned. #11884 #12490 #13982 And I can't see how synthetic-monitoring can help with that. If synthetic-monitoring is able to read a JSON array, why can't heartbeat do the same?

I am trying monitoring Brazilian Open Banking API Status, but heartbeats can't do it on a simple way. Brazilian Open Banking are using this type of structure:

{
  "data": {
    "status": [
      {
        "code": "SCHEDULED_OUTAGE",
        "explanation": "Manutenção Planejada",
        "detectionTime": "2020-01-01T01:00:00Z",
        "expectedResolutionTime": "2020-01-01T01:00:00Z",
        "updateTime": "2020-01-02T01:00:00Z",
        "unavailableEndpoints": [
          "https://api.banco.com.br/open-banking/channels/v1/branches"
        ]
      },
      {
        "code": "PARTIAL_FAILURE",
        "explanation": "Falha na execução do serviço",
        "detectionTime": "2020-01-01T01:00:00Z",
        "expectedResolutionTime": "2020-01-01T01:00:00Z",
        "updateTime": "2020-01-02T01:00:00Z",
        "unavailableEndpoints": [
          "https://api.banco.com.br/open-banking/channels/v1/electronic-channels"
        ]
      }
    ],
    "links": {
      "self": "https://api.banco.com.br/open-banking/discovery/v1/status"
    },
    "meta": {
      "totalRecords": 1,
      "totalPages": 1
    }
  }
}

accessible by using the specific url: https://api.[bank.com.br]/open-banking/discovery/v1/status or https://openbanking.[bank.com.br]/open-banking/discovery/v1/status

I would like to use just data.status.[0].code: OK or data.status.0.code: OK

[gchiartosini]

@gchiartosini would synthetics cover your use case? https://www.elastic.co/guide/en/observability/7.14/synthetic-monitoring.html

I agree with what @gbschenkel has written. Also I am already in production environment with a hearbeat on premise installation and I cannot foresee an architectural change to install the synthetics solution currently only available on docker.

[riccup]

I agree with @gbschenkel. It feels a bit weird that Heartbeat would lack such a seemingly commonplace functionality as checking the size of a JSON array and reading the objects within, at least at the first level. As a developer, I find this limitation rather difficult to explain to my customer. Installing Synthetics would add a significant overhead to the infrastructure, and the information could not be retrieved in a straightforward way for further use (e.g. watchers or a custom dashboard). Please consider implementation. Thank you.

[andrewvc]

Thanks for all the feedback, I've opened https://github.com/elastic/beats/pull/28073 to improve the validation of JSON responses by validating with gval under the hood. Let me know in the PR if there are any concerns or other thoughts about the approach there.

[justinkambic]

Post FF Testing LGTM

JSON path object query

I used this JSON blob to test this feature

{"summaries":[{"monitor_id":"my-monitor","state":{"timestamp":"2021-10-26T18:51:08.411Z","monitor":{"name":"My Monitor","type":"http"},"url":{"full":"http://localhost:9200","scheme":"http","domain":"localhost","port":9200},"summary":{"up":0,"down":1,"status":"down"},"summaryPings":[{"docId":"naTxvXwBAoVeS_D6ItDo","timestamp":"2021-10-26T18:51:08.411Z","@timestamp":"2021-10-26T18:51:08.411Z","url":{"full":"http://localhost:9200","scheme":"http","domain":"localhost","port":9200},"error":{"type":"validate","message":"401 Unauthorized"},"summary":{"up":0,"down":1},"event":{"dataset":"http"},"agent":{"name":"Justins-MBP.fios-router.home","type":"heartbeat","version":"7.16.0","hostname":"Justins-MBP.fios-router.home","ephemeral_id":"89e56021-d313-46b5-afdc-989667be5070","id":"f50021c0-649a-42e3-832f-833a6170c76f"},"http":{"response":{"status_code":401,"body":{"hash":"961eeb737ca61c461169576805b3e85340e4222f167b09e0f438f0e95704a084","bytes":381,"content":"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}}],\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}},\"status\":401}"},"mime_type":"application/json","headers":{"Content-Type":"application/json; charset=UTF-8","Content-Length":"381","Www-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"rtt":{"total":{"us":1514},"write_request":{"us":51},"response_header":{"us":999},"validate":{"us":1116},"content":{"us":117}}},"monitor":{"status":"down","duration":{"us":2340},"check_group":"a855a7e4-368d-11ec-8f01-acde48001122","ip":"127.0.0.1","id":"my-monitor","name":"My Monitor","type":"http","timespan":{"lt":"2021-10-26T18:51:18.413Z","gte":"2021-10-26T18:51:08.413Z"}},"ecs":{"version":"1.12.0"},"observer":{"hostname":"Justins-MBP.fios-router.home","ip":["fe80::aede:48ff:fe00:1122","fe80::1027:7a0c:a5f2:eeba","192.168.1.241","fe80::5415:86ff:fe6d:77a3","fe80::5415:86ff:fe6d:77a3","fe80::97b2:399f:84ed:fffd","fe80::8ca2:9b8b:1553:7d0e"],"mac":["ac:de:48:00:11:22","82:76:f7:29:4c:05","82:76:f7:29:4c:04","3e:22:fb:19:bf:97","3c:22:fb:19:bf:97","82:76:f7:29:4c:01","82:76:f7:29:4c:00","56:15:86:6d:77:a3","56:15:86:6d:77:a3","82:76:f7:29:4c:01"]},"tcp":{"rtt":{"connect":{"us":311}}},"resolve":{"ip":"127.0.0.1","rtt":{"us":761}}}],"observer":{"geo":{"name":[]}}}}],"nextPagePagination":null,"prevPagePagination":null}

I configured the monitor like:

Failing:

- type: http
  id: json-test
  name: JSON test
  urls: ['http://localhost:12349/complex-json']
  schedule: '@every 5s'
  check.response:
    status: [200]
    json:
      - description: check monitors
        # added some garbage to the end of the selector
        expression: '$.summaries[0].monitor_id == "my-monitor-fail-plz"'

Passing:

- type: http
  id: json-test
  name: JSON test
  urls: ['http://localhost:12349/complex-json']
  schedule: '@every 5s'
  check.response:
    status: [200]
    json:
      - description: check monitors
        expression: '$.summaries[0].monitor_id == "my-monitor"'

List JSON

I tested the list-shaped JSON validation using an endpoint that returned an object like:

[{"a":"b"},{"c":"d"},{"a":"b"}]

I configured the monitor like:

Failing:

- type: http
  id: json-list
  name: JSON list
  urls: ['http://localhost:12349/json-list']
  schedule: '@every 5s'
  check.response:
    status: [200]
    json:
      - description: check list
        # no 'a' key exists on this object
        expression: '$[1].a == "d"'

Passing:

- type: http
  id: json-list
  name: JSON list
  urls: ['http://localhost:12349/json-list']
  schedule: '@every 5s'
  check.response:
    status: [200]
    json:
      - description: check list
        # check for defined key in second list item
        expression: '$[1].c == "d"'