[SLOs] Can't create an SLO of custom KQL/metric SLI type

[ablnk]

Version: Serverless project v 8.15.0

Environments: Observability test environment: keepserverless-qa-oblt

Description:

SLO creation fails, 502 Bad Gateway returns:

{ok: false, message: "backend closed connection"}
message
: 
"backend closed connection"
ok
: 
false

Error shown in Kibana:

Error: backend closed connection
    at Object.onError (https://kibana.estccdn.com/5381dd714d2a/bundles/plugin/slo/1.0.0/slo.chunk.3.js:3:33424)
    at s.execute (https://kibana.estccdn.com/5381dd714d2a/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:361:977022)
    at async https://kibana.estccdn.com/5381dd714d2a/bundles/plugin/slo/1.0.0/slo.chunk.3.js:3:35549

Update: It seems this error causes ES to be unresponsive to any queries.

Steps to reproduce:

1. Go to SLOs - Create new SLO.
2. Try to create a new SLO with the following parameters:

• SLI type: Custom Query
• Index: APM
• Timestamp field: @timestamp
• Good query: transaction.duration.us <= 100000 or error.log.level : "warning"
• Objectives: by default

Expected behavior: SLO is created.

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[elasticmachine]

Pinging @elastic/obs-ux-management-team (Team:obs-ux-management)