search

elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.62k stars 8.22k forks source link

Failing ES Promotion: FTR Configs #48 / Alerts APIs - Trial License/Complete Tier @ess Alerts Compatibility Query should generate a signal-on-legacy-signal with AAD index pattern #176105

Closed mistic closed 9 months ago

mistic commented 9 months ago

Detection Engine - Alerts Integration Tests - ESS Env - Trial License x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/alerts/trial_license_complete_tier/alerts_compatibility.ts

Alerts APIs - Trial License/Complete Tier @ess Alerts Compatibility Query should generate a signal-on-legacy-signal with AAD index pattern

This failure is preventing the promotion of the current Elasticsearch snapshot.

For more information on the Elasticsearch snapshot process including how to reproduce using the unverified ES build please read the failed promotion annotation. Other important information can be found at:

Error: Timeout of 360000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/var/lib/buildkite-agent/builds/kb-n2-4-spot-3e04e97b4d80091d/elastic/kibana-elasticsearch-snapshot-verify/kibana/x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/alerts/trial_license_complete_tier/alerts_compatibility.ts)
    at listOnTimeout (node:internal/timers:573:17)
    at processTimers (node:internal/timers:514:7) {
  code: 'ERR_MOCHA_TIMEOUT',
  timeout: 360000,
  file: '/var/lib/buildkite-agent/builds/kb-n2-4-spot-3e04e97b4d80091d/elastic/kibana-elasticsearch-snapshot-verify/kibana/x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/alerts/trial_license_complete_tier/alerts_compatibility.ts'
}
elasticmachine commented 9 months ago

Pinging @elastic/security-detection-engine (Team:Detection Engine)

mistic commented 9 months ago

Skipped.

main: aabf7b7

elasticmachine commented 9 months ago

Pinging @elastic/security-solution (Team: SecuritySolution)

elasticmachine commented 9 months ago

Pinging @elastic/security-detections-response (Team:Detections and Resp)

vitaliidm commented 9 months ago

It appears, aliases not working properly anymore Here is successfully passing tests on https://buildkite.com/elastic/kibana-elasticsearch-snapshot-verify/builds/3610 The last successful snapshot: 

ES_SNAPSHOT_MANIFEST="https://storage.googleapis.com/kibana-ci-es-snapshots-daily/8.13.0/archives/20240131-141727_1fd2756f/manifest.json" node scripts/functional_tests_server.js

Here is alias configuration for .alerts-security.alerts-default and .siem-signals-default, with count results of indexed documents. Results match, so tests are passing.

GET .alerts-security.alerts-default

{
    ".internal.alerts-security.alerts-default-000001": {
      "aliases": {
        ".alerts-security.alerts-default": {
          "is_write_index": true
        },
        ".siem-signals-default": {
          "is_write_index": false
        }
      },
    ....
  },
    ".siem-signals-default-000001-7.16.0": {
      "aliases": {
        ".alerts-security.alerts-default": {
          "is_write_index": false
        },
        ".siem-signals-default": {
          "is_write_index": true
        }
      },
  ...
  },
}

Search results for documents in .alerts-security.alerts-default

GET .alerts-security.alerts-default/_count

{
  "count": 23,
  "_shards": {
    "total": 2,
    "successful": 2,
    "skipped": 0,
    "failed": 0
  }
}

Search results for documents in .siem-signals-default

GET .siem-signals-default/_count

{
  "count": 23,
  "_shards": {
    "total": 2,
    "successful": 2,
    "skipped": 0,
    "failed": 0
  }
}

But next promotion build https://buildkite.com/elastic/kibana-elasticsearch-snapshot-verify/builds/3614, with snapshot:

ES_SNAPSHOT_MANIFEST="https://storage.googleapis.com/kibana-ci-es-snapshots-daily/8.13.0/archives/20240201-141609_d82821f3/manifest.json" node scripts/functional_tests_server.js

not working anymore

when looking for aliases of .alerts-security.alerts-default, .siem-signals-default is not anymore present

GET .alerts-security.alerts-default

{
  ".internal.alerts-security.alerts-default-000001": {
    "aliases": {
      ".alerts-security.alerts-default": {
        "is_write_index": true
      },
      ".siem-signals-default": {
        "is_write_index": false
      }
    },
...
}
}

Local Kibana ran with these 2 snapshots is the same. Looks like, there some changes on ES side. Continue looking into it

vitaliidm commented 9 months ago

pinged ES team in internal slack

yctercero commented 9 months ago

@mistic based on @vitaliidm 's findings and it being a legitimate failure that should be blocking promotion. Should we unskip it so we know when a valid fix is in?

mattc58 commented 9 months ago

Hi, I'm trying to determine if elasticsearch#104145 would have caused this. It seems likely given the timelines here.

I am having a hard time understanding the exact data streams or indices in scope for this test. It seems like this is entirely focused on indices and aliases and not data streams, and the changes in the linked ES ticket should have only fixed a bug in how data stream aliases were returned. To help me, could you describe the specific indices and aliases that are used here throughout the lifetime of the failing test? And then maybe also describe which specific query is failing? It's looking like a timeout but I'm not sure which query is causing that to happen.

Also, is this only running on main right now?

vitaliidm commented 9 months ago

Summary on tests:

7.16 here is the version, indexed documents are originated from. We index them in .siem-signals-default-000001-7.16.0 and expected 2 tests to pass

  1. rule should find them in .siem-signals-*
  2. rule should find them in .alerts-security.alerts-default

So, second test fails, as .alerts-security does not have alias

I don't see any alias related failures, here is log from alias creation Here logs from build that does not fail tests


info [o.e.x.i.IndexLifecycleTransition] [ftr] moving index [.siem-signals-default-000001-7.16.0] from [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] to [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] in policy [.siem-signals-default]
 proc [kibana] [2024-02-07T16:21:11.752+00:00][DEBUG][elasticsearch.query.data] 404 - 241.0B
 proc [kibana] DELETE /_template/.siem-signals-default [index_template_missing_exception]: index_template [.siem-signals-default] missing {"http":{"request":{"id":"21693efa-34e2-4e45-a599-7edd6111f837","method":"DELETE","headers":{"user-agent":"Kibana/8.13.0","x-elastic-product-origin":"kibana","authorization":"[REDACTED]","x-opaque-id":"21693efa-34e2-4e45-a599-7edd6111f837","x-elastic-client-meta":"es=8.9.1p,js=20.10.0,t=8.3.3,hc=20.10.0","accept":"application/vnd.elasticsearch+json; compatible-with=8,text/plain"}},"response":{"body":{"bytes":241},"status_code":404,"headers":{"x-opaque-id":"21693efa-34e2-4e45-a599-7edd6111f837","x-elastic-product":"Elasticsearch","content-type":"application/json","content-length":"241"}}},"url":{"path":"/_template/.siem-signals-default","query":""},"service":{"node":{"roles":["background_tasks","ui"]}}}
 proc [kibana] [2024-02-07T16:21:11.757+00:00][DEBUG][elasticsearch.query.data] 200
 proc [kibana] GET /.siem-signals-default {"http":{"request":{"id":"21693efa-34e2-4e45-a599-7edd6111f837","method":"GET","headers":{"user-agent":"Kibana/8.13.0","x-elastic-product-origin":"kibana","authorization":"[REDACTED]","x-opaque-id":"21693efa-34e2-4e45-a599-7edd6111f837","x-elastic-client-meta":"es=8.9.1p,js=20.10.0,t=8.3.3,hc=20.10.0","accept":"application/vnd.elasticsearch+json; compatible-with=8,text/plain"}},"response":{"body":{},"status_code":200,"headers":{"x-opaque-id":"21693efa-34e2-4e45-a599-7edd6111f837","x-elastic-product":"Elasticsearch","content-type":"application/json","transfer-encoding":"chunked"}}},"url":{"path":"/.siem-signals-default","query":""},"service":{"node":{"roles":["background_tasks","ui"]}}}
 info [o.e.c.m.MetadataMappingService] [ftr] [.siem-signals-default-000001-7.16.0/21igjrlOQO-k9-o-bexfSQ] update_mapping [_doc]
 proc [kibana] [2024-02-07T16:21:11.920+00:00][DEBUG][elasticsearch.query.data] 200 - 21.0B
 proc [kibana] PUT /.siem-signals-default-000001-7.16.0/_mapping?allow_no_indices=true
 proc [kibana] {"properties":{"signal":{"type":"object","properties":{"_meta":{"type":"object","properties":{"version":{"type":"long"}}},"ancestors":{"properties":{"rule":{"type":"keyword"},"index":{"type":"keyword"},"id":{"type":"keyword"},"type":{"type":"keyword"},"depth":{"type":"long"}}},"depth":{"type":"integer"},"group":{"type":"object","properties":{"id":{"type":"keyword"},"index":{"type":"integer"}}},"original_event":{"type":"object","properties":{"reason":{"type":"keyword"}}},"reason":{"type":"keyword"},"rule":{"type":"object","properties":{"author":{"type":"keyword"},"building_block_type":{"type":"keyword"},"license":{"type":"keyword"},"note":{"type":"text"},"risk_score_mapping":{"type":"object","properties":{"field":{"type":"keyword"},"operator":{"type":"keyword"},"value":{"type":"keyword"}}},"rule_name_override":{"type":"keyword"},"severity_mapping":{"type":"object","properties":{"field":{"type":"keyword"},"operator":{"type":"keyword"},"value":{"type":"keyword"},"severity":{"type":"keyword"}}},"threat":{"type":"object","properties":{"technique":{"type":"object","properties":{"subtechnique":{"type":"object","properties":{"id":{"type":"keyword"},"name":{"type":"keyword"},"reference":{"type":"keyword"}}}}}}},"threat_index":{"type":"keyword"},"threat_indicator_path":{"type":"keyword"},"threat_language":{"type":"keyword"},"threat_mapping":{"type":"object","properties":{"entries":{"type":"object","properties":{"field":{"type":"keyword"},"value":{"type":"keyword"},"type":{"type":"keyword"}}}}},"threat_query":{"type":"keyword"},"threshold":{"type":"object","properties":{"field":{"type":"keyword"},"value":{"type":"float"}}},"timestamp_override":{"type":"keyword"}}},"threshold_result":{"properties":{"from":{"type":"date"},"terms":{"properties":{"field":{"type":"keyword"},"value":{"type":"keyword"}}},"cardinality":{"properties":{"field":{"type":"keyword"},"value":{"type":"long"}}},"count":{"type":"long"}}}}},"kibana.alert.ancestors.depth":{"type":"alias","path":"signal.ancestors.depth"},"kibana.alert.ancestors.id":{"type":"alias","path":"signal.ancestors.id"},"kibana.alert.ancestors.index":{"type":"alias","path":"signal.ancestors.index"},"kibana.alert.ancestors.type":{"type":"alias","path":"signal.ancestors.type"},"kibana.alert.depth":{"type":"alias","path":"signal.depth"},"kibana.alert.group.id":{"type":"alias","path":"signal.group.id"},"kibana.alert.group.index":{"type":"alias","path":"signal.group.index"},"kibana.alert.original_event.action":{"type":"alias","path":"signal.original_event.action"},"kibana.alert.original_event.category":{"type":"alias","path":"signal.original_event.category"},"kibana.alert.original_event.code":{"type":"alias","path":"signal.original_event.code"},"kibana.alert.original_event.created":{"type":"alias","path":"signal.original_event.created"},"kibana.alert.original_event.dataset":{"type":"alias","path":"signal.original_event.dataset"},"kibana.alert.original_event.duration":{"type":"alias","path":"signal.original_event.duration"},"kibana.alert.original_event.end":{"type":"alias","path":"signal.original_event.end"},"kibana.alert.original_event.hash":{"type":"alias","path":"signal.original_event.hash"},"kibana.alert.original_event.id":{"type":"alias","path":"signal.original_event.id"},"kibana.alert.original_event.kind":{"type":"alias","path":"signal.original_event.kind"},"kibana.alert.original_event.module":{"type":"alias","path":"signal.original_event.module"},"kibana.alert.original_event.outcome":{"type":"alias","path":"signal.original_event.outcome"},"kibana.alert.original_event.provider":{"type":"alias","path":"signal.original_event.provider"},"kibana.alert.original_event.reason":{"type":"alias","path":"signal.original_event.reason"},"kibana.alert.original_event.risk_score":{"type":"alias","path":"signal.original_event.risk_score"},"kibana.alert.original_event.risk_score_norm":{"type":"alias","path":"signal.original_event.risk_score_norm"},"kibana.alert.original_event.sequence":{"type":"alias","path":"signal.original_event.sequence"},"kibana.alert.original_event.severity":{"type":"alias","path":"signal.original_event.severity"},"kibana.alert.original_event.start":{"type":"alias","path":"signal.original_event.start"},"kibana.alert.original_event.timezone":{"type":"alias","path":"signal.original_event.timezone"},"kibana.alert.original_event.type":{"type":"alias","path":"signal.original_event.type"},"kibana.alert.original_time":{"type":"alias","path":"signal.original_time"},"kibana.alert.reason":{"type":"alias","path":"signal.reason"},"kibana.alert.rule.author":{"type":"alias","path":"signal.rule.author"},"kibana.alert.building_block_type":{"type":"alias","path":"signal.rule.building_block_type"},"kibana.alert.rule.created_at":{"type":"alias","path":"signal.rule.created_at"},"kibana.alert.rule.created_by":{"type":"alias","path":"signal.rule.created_by"},"kibana.alert.rule.description":{"type":"alias","path":"signal.rule.description"},"kibana.alert.rule.enabled":{"type":"alias","path":"signal.rule.enabled"},"kibana.alert.rule.false_positives":{"type":"alias","path":"signal.rule.false_positives"},"kibana.alert.rule.from":{"type":"alias","path":"signal.rule.from"},"kibana.alert.rule.uuid":{"type":"alias","path":"signal.rule.id"},"kibana.alert.rule.immutable":{"type":"alias","path":"signal.rule.immutable"},"kibana.alert.rule.interval":{"type":"alias","path":"signal.rule.interval"},"kibana.alert.rule.license":{"type":"alias","path":"signal.rule.license"},"kibana.alert.rule.max_signals":{"type":"alias","path":"signal.rule.max_signals"},"kibana.alert.rule.name":{"type":"alias","path":"signal.rule.name"},"kibana.alert.rule.note":{"type":"alias","path":"signal.rule.note"},"kibana.alert.rule.references":{"type":"alias","path":"signal.rule.references"},"kibana.alert.risk_score":{"type":"alias","path":"signal.rule.risk_score"},"kibana.alert.rule.rule_id":{"type":"alias","path":"signal.rule.rule_id"},"kibana.alert.rule.rule_name_override":{"type":"alias","path":"signal.rule.rule_name_override"},"kibana.alert.severity":{"type":"alias","path":"signal.rule.severity"},"kibana.alert.rule.tags":{"type":"alias","path":"signal.rule.tags"},"kibana.alert.rule.threat.framework":{"type":"alias","path":"signal.rule.threat.framework"},"kibana.alert.rule.threat.tactic.id":{"type":"alias","path":"signal.rule.threat.tactic.id"},"kibana.alert.rule.threat.tactic.name":{"type":"alias","path":"signal.rule.threat.tactic.name"},"kibana.alert.rule.threat.tactic.reference":{"type":"alias","path":"signal.rule.threat.tactic.reference"},"kibana.alert.rule.threat.technique.id":{"type":"alias","path":"signal.rule.threat.technique.id"},"kibana.alert.rule.threat.technique.name":{"type":"alias","path":"signal.rule.threat.technique.name"},"kibana.alert.rule.threat.technique.reference":{"type":"alias","path":"signal.rule.threat.technique.reference"},"kibana.alert.rule.threat.technique.subtechnique.id":{"type":"alias","path":"signal.rule.threat.technique.subtechnique.id"},"kibana.alert.rule.threat.technique.subtechnique.name":{"type":"alias","path":"signal.rule.threat.technique.subtechnique.name"},"kibana.alert.rule.threat.technique.subtechnique.reference":{"type":"alias","path":"signal.rule.threat.technique.subtechnique.reference"},"kibana.alert.rule.timeline_id":{"type":"alias","path":"signal.rule.timeline_id"},"kibana.alert.rule.timeline_title":{"type":"alias","path":"signal.rule.timeline_title"},"kibana.alert.rule.timestamp_override":{"type":"alias","path":"signal.rule.timestamp_override"},"kibana.alert.rule.to":{"type":"alias","path":"signal.rule.to"},"kibana.alert.rule.type":{"type":"alias","path":"signal.rule.type"},"kibana.alert.rule.updated_at":{"type":"alias","path":"signal.rule.updated_at"},"kibana.alert.rule.updated_by":{"type":"alias","path":"signal.rule.updated_by"},"kibana.alert.rule.version":{"type":"alias","path":"signal.rule.version"},"kibana.alert.workflow_status":{"type":"alias","path":"signal.status"},"kibana.alert.threshold_result.from":{"type":"alias","path":"signal.threshold_result.from"},"kibana.alert.threshold_result.terms.field":{"type":"alias","path":"signal.threshold_result.terms.field"},"kibana.alert.threshold_result.terms.value":{"type":"alias","path":"signal.threshold_result.terms.value"},"kibana.alert.threshold_result.cardinality.field":{"type":"alias","path":"signal.threshold_result.cardinality.field"},"kibana.alert.threshold_result.cardinality.value":{"type":"alias","path":"signal.threshold_result.cardinality.value"},"kibana.alert.threshold_result.count":{"type":"alias","path":"signal.threshold_result.count"},"kibana.space_ids":{"type":"constant_keyword","value":"default"}},"dynamic":false,"_meta":{"version":57,"aliases_version":4}} {"http":{"request":{"id":"21693efa-34e2-4e45-a599-7edd6111f837","method":"PUT","headers":{"user-agent":"Kibana/8.13.0","x-elastic-product-origin":"kibana","authorization":"[REDACTED]","x-opaque-id":"21693efa-34e2-4e45-a599-7edd6111f837","x-elastic-client-meta":"es=8.9.1p,js=20.10.0,t=8.3.3,hc=20.10.0","content-type":"application/vnd.elasticsearch+json; compatible-with=8","accept":"application/vnd.elasticsearch+json; compatible-with=8","content-length":"8697"}},"response":{"body":{"bytes":21},"status_code":200,"headers":{"x-opaque-id":"21693efa-34e2-4e45-a599-7edd6111f837","x-elastic-product":"Elasticsearch","content-type":"application/vnd.elasticsearch+json;compatible-with=8","content-length":"21"}}},"url":{"path":"/.siem-signals-default-000001-7.16.0/_mapping","query":"allow_no_indices=true"},"service":{"node":{"roles":["background_tasks","ui"]}}}
 proc [kibana] [2024-02-07T16:21:11.922+00:00][DEBUG][elasticsearch.query.data] 200 - 101.0B
 proc [kibana] GET /.siem-signals-default-*/_alias/.siem-signals-default {"http":{"request":{"id":"21693efa-34e2-4e45-a599-7edd6111f837","method":"GET","headers":{"user-agent":"Kibana/8.13.0","x-elastic-product-origin":"kibana","x-elastic-client-meta":"es=8.9.1p,js=20.10.0,t=8.3.3,hc=20.10.0","authorization":"[REDACTED]","x-opaque-id":"21693efa-34e2-4e45-a599-7edd6111f837","accept":"application/vnd.elasticsearch+json; compatible-with=8,text/plain"}},"response":{"body":{"bytes":101},"status_code":200,"headers":{"x-opaque-id":"21693efa-34e2-4e45-a599-7edd6111f837","x-elastic-product":"Elasticsearch","content-type":"application/json","content-length":"101"}}},"url":{"path":"/.siem-signals-default-*/_alias/.siem-signals-default","query":""},"service":{"node":{"roles":["background_tasks","ui"]}}}
 proc [kibana] [2024-02-07T16:21:11.957+00:00][DEBUG][elasticsearch.query.data] 200 - 21.0B
 proc [kibana] POST /_aliases
 proc [kibana] {"actions":[{"add":{"index":".siem-signals-default-000001-7.16.0","alias":".alerts-security.alerts-default","is_write_index":false}}]} {"http":{"request":{"id":"21693efa-34e2-4e45-a599-7edd6111f837","method":"POST","headers":{"user-agent":"Kibana/8.13.0","x-elastic-product-origin":"kibana","x-elastic-client-meta":"es=8.9.1p,js=20.10.0,t=8.3.3,hc=20.10.0","authorization":"[REDACTED]","x-opaque-id":"21693efa-34e2-4e45-a599-7edd6111f837","content-type":"application/vnd.elasticsearch+json; compatible-with=8","accept":"application/vnd.elasticsearch+json; compatible-with=8","content-length":"134"}},"response":{"body":{"bytes":21},"status_code":200,"headers":{"x-opaque-id":"21693efa-34e2-4e45-a599-7edd6111f837","x-elastic-product":"Elasticsearch","content-type":"application/vnd.elasticsearch+json;compatible-with=8","content-length":"21"}}},"url":{"path":"/_aliases","query":""},"service":{"node":{"roles":["background_tasks","ui"]}}}
 proc [kibana] [2024-02-07T16:21:11.959+00:00][DEBUG][elasticsearch.query.data] 200 - 331.0B
 proc [kibana] GET /.siem-signals-default/_alias {"http":{"request":{"id":"21693efa-34e2-4e45-a599-7edd6111f837","method":"GET","headers":{"user-agent":"Kibana/8.13.0","x-elastic-product-origin":"kibana","authorization":"[REDACTED]","x-opaque-id":"21693efa-34e2-4e45-a599-7edd6111f837","x-elastic-client-meta":"es=8.9.1p,js=20.10.0,t=8.3.3,hc=20.10.0","accept":"application/vnd.elasticsearch+json; compatible-with=8,text/plain"}},"response":{"body":{"bytes":331},"status_code":200,"headers":{"x-opaque-id":"21693efa-34e2-4e45-a599-7edd6111f837","x-elastic-product":"Elasticsearch","content-type":"application/json","content-length":"331"}}},"url":{"path":"/.siem-signals-default/_alias","query":""},"service":{"node":{"roles":["background_tasks","ui"]}}}
 proc [kibana] [2024-02-07T16:21:11.961+00:00][DEBUG][elasticsearch.query.data] 200
 proc [kibana] GET /.siem-signals-default-000001-7.16.0 {"http":{"request":{"id":"21693efa-34e2-4e45-a599-7edd6111f837","method":"GET","headers":{"user-agent":"Kibana/8.13.0","x-elastic-product-origin":"kibana","authorization":"[REDACTED]","x-opaque-id":"21693efa-34e2-4e45-a599-7edd6111f837","x-elastic-client-meta":"es=8.9.1p,js=20.10.0,t=8.3.3,hc=20.10.0","accept":"application/vnd.elasticsearch+json; compatible-with=8,text/plain"}},"response":{"body":{},"status_code":200,"headers":{"x-opaque-id":"21693efa-34e2-4e45-a599-7edd6111f837","x-elastic-product":"Elasticsearch","content-type":"application/json","transfer-encoding":"chunked"}}},"url":{"path":"/.siem-signals-default-000001-7.16.0","query":""},"service":{"node":{"roles":["background_tasks","ui"]}}}
 proc [kibana] [2024-02-07T16:21:11.963+00:00][DEBUG][elasticsearch.query.data] 400 - 303.0B
 proc [kibana] POST /.siem-signals-default/_rollover [illegal_argument_exception]: index name [.siem-signals-default-000001-7.16.0] does not match pattern '^.*-\d+$' {"http":{"request":{"id":"21693efa-34e2-4e45-a599-7edd6111f837","method":"POST","headers":{"user-agent":"Kibana/8.13.0","x-elastic-product-origin":"kibana","authorization":"[REDACTED]","x-opaque-id":"21693efa-34e2-4e45-a599-7edd6111f837","x-elastic-client-meta":"es=8.9.1p,js=20.10.0,t=8.3.3,hc=20.10.0","accept":"application/vnd.elasticsearch+json; compatible-with=8,text/plain"}},"response":{"body":{"bytes":303},"status_code":400,"headers":{"x-opaque-id":"21693efa-34e2-4e45-a599-7edd6111f837","x-elastic-product":"Elasticsearch","content-type":"application/json","content-length":"303"}}},"url":{"path":"/.siem-signals-default/_rollover","query":""},"service":{"node":{"roles":["background_tasks","ui"]}}}

To help me, could you describe the specific indices and aliases that are used here throughout the lifetime of the failing test?

POST alias

 proc [kibana] POST /_aliases
 proc [kibana] {"actions":[{"add":{"index":".siem-signals-default-000001-7.16.0","alias":".alerts-security.alerts-default","is_write_index":false}}]} {"http":{"request":{"id":"21693efa-34e2-4e45-a599-7edd6111f837","method":"POST","headers":{"user-agent":"Kibana/8.13.0","x-elastic-product-origin":"kibana","x-elastic-client-meta":"es=8.9.1p,js=20.10.0,t=8.3.3,hc=20.10.0","authorization":"[REDACTED]","x-opaque-id":"21693efa-34e2-4e45-a599-7edd6111f837","content-type":"application/vnd.elasticsearch+json; compatible-with=8","accept":"application/vnd.elasticsearch+json; compatible-with=8","content-length":"134"}},"response":{"body":{"bytes":21},"status_code":200,"headers":{"x-opaque-id":"21693efa-34e2-4e45-a599-7edd6111f837","x-elastic-product":"Elasticsearch","content-type":"application/vnd.elasticsearch+json;compatible-with=8","content-length":"21"}}},"url":{"path":"/_aliases","query":""},"service":{"node":{"roles":["background_tasks","ui"]}}}

And as mentioned from comments above, aliases are created 

GET .alerts-security.alerts-default/_alias

{
  ".internal.alerts-security.alerts-default-000001": {
    "aliases": {
      ".alerts-security.alerts-default": {
        "is_write_index": true
      },
      ".siem-signals-default": {
        "is_write_index": false
      }
    }
  },
  ".siem-signals-default-000001-7.16.0": {
    "aliases": {
      ".alerts-security.alerts-default": {
        "is_write_index": false
      },
      ".siem-signals-default": {
        "is_write_index": true
      }
    }
  }
}

And then maybe also describe which specific query is failing? It's looking like a timeout but I'm not sure which query is causing that to happen.

No specific query is failing, and timeout happens because we looking for search results that should be returned from .alerts-security.alerts-default alias

Also, is this only running on main right now?

I think so

Here logs from build that DOES FAIL tests

proc [kibana] GET /.siem-signals-default {"http":{"request":{"id":"c1bd32b8-6835-4fb1-bfba-4895b1e21ba7","method":"GET","headers":{"user-agent":"Kibana/8.13.0","x-elastic-product-origin":"kibana","authorization":"[REDACTED]","x-opaque-id":"c1bd32b8-6835-4fb1-bfba-4895b1e21ba7","x-elastic-client-meta":"es=8.9.1p,js=20.10.0,t=8.3.3,hc=20.10.0","accept":"application/vnd.elasticsearch+json; compatible-with=8,text/plain"}},"response":{"body":{},"status_code":200,"headers":{"x-opaque-id":"c1bd32b8-6835-4fb1-bfba-4895b1e21ba7","x-elastic-product":"Elasticsearch","content-type":"application/json","transfer-encoding":"chunked"}}},"url":{"path":"/.siem-signals-default","query":""},"service":{"node":{"roles":["background_tasks","ui"]}}}
 proc [kibana] [2024-02-07T16:29:16.557+00:00][DEBUG][elasticsearch.query.data] 400 - 289.0B
 proc [kibana] PUT /.siem-signals-default-000001-7.16.0/_mapping?allow_no_indices=true
 proc [kibana] {"properties":{"signal":{"type":"object","properties":{"_meta":{"type":"object","properties":{"version":{"type":"long"}}},"ancestors":{"properties":{"rule":{"type":"keyword"},"index":{"type":"keyword"},"id":{"type":"keyword"},"type":{"type":"keyword"},"depth":{"type":"long"}}},"depth":{"type":"integer"},"group":{"type":"object","properties":{"id":{"type":"keyword"},"index":{"type":"integer"}}},"original_event":{"type":"object","properties":{"reason":{"type":"keyword"}}},"reason":{"type":"keyword"},"rule":{"type":"object","properties":{"author":{"type":"keyword"},"building_block_type":{"type":"keyword"},"license":{"type":"keyword"},"note":{"type":"text"},"risk_score_mapping":{"type":"object","properties":{"field":{"type":"keyword"},"operator":{"type":"keyword"},"value":{"type":"keyword"}}},"rule_name_override":{"type":"keyword"},"severity_mapping":{"type":"object","properties":{"field":{"type":"keyword"},"operator":{"type":"keyword"},"value":{"type":"keyword"},"severity":{"type":"keyword"}}},"threat":{"type":"object","properties":{"technique":{"type":"object","properties":{"subtechnique":{"type":"object","properties":{"id":{"type":"keyword"},"name":{"type":"keyword"},"reference":{"type":"keyword"}}}}}}},"threat_index":{"type":"keyword"},"threat_indicator_path":{"type":"keyword"},"threat_language":{"type":"keyword"},"threat_mapping":{"type":"object","properties":{"entries":{"type":"object","properties":{"field":{"type":"keyword"},"value":{"type":"keyword"},"type":{"type":"keyword"}}}}},"threat_query":{"type":"keyword"},"threshold":{"type":"object","properties":{"field":{"type":"keyword"},"value":{"type":"float"}}},"timestamp_override":{"type":"keyword"}}},"threshold_result":{"properties":{"from":{"type":"date"},"terms":{"properties":{"field":{"type":"keyword"},"value":{"type":"keyword"}}},"cardinality":{"properties":{"field":{"type":"keyword"},"value":{"type":"long"}}},"count":{"type":"long"}}}}},"kibana.alert.ancestors.depth":{"type":"alias","path":"signal.ancestors.depth"},"kibana.alert.ancestors.id":{"type":"alias","path":"signal.ancestors.id"},"kibana.alert.ancestors.index":{"type":"alias","path":"signal.ancestors.index"},"kibana.alert.ancestors.type":{"type":"alias","path":"signal.ancestors.type"},"kibana.alert.depth":{"type":"alias","path":"signal.depth"},"kibana.alert.group.id":{"type":"alias","path":"signal.group.id"},"kibana.alert.group.index":{"type":"alias","path":"signal.group.index"},"kibana.alert.original_event.action":{"type":"alias","path":"signal.original_event.action"},"kibana.alert.original_event.category":{"type":"alias","path":"signal.original_event.category"},"kibana.alert.original_event.code":{"type":"alias","path":"signal.original_event.code"},"kibana.alert.original_event.created":{"type":"alias","path":"signal.original_event.created"},"kibana.alert.original_event.dataset":{"type":"alias","path":"signal.original_event.dataset"},"kibana.alert.original_event.duration":{"type":"alias","path":"signal.original_event.duration"},"kibana.alert.original_event.end":{"type":"alias","path":"signal.original_event.end"},"kibana.alert.original_event.hash":{"type":"alias","path":"signal.original_event.hash"},"kibana.alert.original_event.id":{"type":"alias","path":"signal.original_event.id"},"kibana.alert.original_event.kind":{"type":"alias","path":"signal.original_event.kind"},"kibana.alert.original_event.module":{"type":"alias","path":"signal.original_event.module"},"kibana.alert.original_event.outcome":{"type":"alias","path":"signal.original_event.outcome"},"kibana.alert.original_event.provider":{"type":"alias","path":"signal.original_event.provider"},"kibana.alert.original_event.reason":{"type":"alias","path":"signal.original_event.reason"},"kibana.alert.original_event.risk_score":{"type":"alias","path":"signal.original_event.risk_score"},"kibana.alert.original_event.risk_score_norm":{"type":"alias","path":"signal.original_event.risk_score_norm"},"kibana.alert.original_event.sequence":{"type":"alias","path":"signal.original_event.sequence"},"kibana.alert.original_event.severity":{"type":"alias","path":"signal.original_event.severity"},"kibana.alert.original_event.start":{"type":"alias","path":"signal.original_event.start"},"kibana.alert.original_event.timezone":{"type":"alias","path":"signal.original_event.timezone"},"kibana.alert.original_event.type":{"type":"alias","path":"signal.original_event.type"},"kibana.alert.original_time":{"type":"alias","path":"signal.original_time"},"kibana.alert.reason":{"type":"alias","path":"signal.reason"},"kibana.alert.rule.author":{"type":"alias","path":"signal.rule.author"},"kibana.alert.building_block_type":{"type":"alias","path":"signal.rule.building_block_type"},"kibana.alert.rule.created_at":{"type":"alias","path":"signal.rule.created_at"},"kibana.alert.rule.created_by":{"type":"alias","path":"signal.rule.created_by"},"kibana.alert.rule.description":{"type":"alias","path":"signal.rule.description"},"kibana.alert.rule.enabled":{"type":"alias","path":"signal.rule.enabled"},"kibana.alert.rule.false_positives":{"type":"alias","path":"signal.rule.false_positives"},"kibana.alert.rule.from":{"type":"alias","path":"signal.rule.from"},"kibana.alert.rule.uuid":{"type":"alias","path":"signal.rule.id"},"kibana.alert.rule.immutable":{"type":"alias","path":"signal.rule.immutable"},"kibana.alert.rule.interval":{"type":"alias","path":"signal.rule.interval"},"kibana.alert.rule.license":{"type":"alias","path":"signal.rule.license"},"kibana.alert.rule.max_signals":{"type":"alias","path":"signal.rule.max_signals"},"kibana.alert.rule.name":{"type":"alias","path":"signal.rule.name"},"kibana.alert.rule.note":{"type":"alias","path":"signal.rule.note"},"kibana.alert.rule.references":{"type":"alias","path":"signal.rule.references"},"kibana.alert.risk_score":{"type":"alias","path":"signal.rule.risk_score"},"kibana.alert.rule.rule_id":{"type":"alias","path":"signal.rule.rule_id"},"kibana.alert.rule.rule_name_override":{"type":"alias","path":"signal.rule.rule_name_override"},"kibana.alert.severity":{"type":"alias","path":"signal.rule.severity"},"kibana.alert.rule.tags":{"type":"alias","path":"signal.rule.tags"},"kibana.alert.rule.threat.framework":{"type":"alias","path":"signal.rule.threat.framework"},"kibana.alert.rule.threat.tactic.id":{"type":"alias","path":"signal.rule.threat.tactic.id"},"kibana.alert.rule.threat.tactic.name":{"type":"alias","path":"signal.rule.threat.tactic.name"},"kibana.alert.rule.threat.tactic.reference":{"type":"alias","path":"signal.rule.threat.tactic.reference"},"kibana.alert.rule.threat.technique.id":{"type":"alias","path":"signal.rule.threat.technique.id"},"kibana.alert.rule.threat.technique.name":{"type":"alias","path":"signal.rule.threat.technique.name"},"kibana.alert.rule.threat.technique.reference":{"type":"alias","path":"signal.rule.threat.technique.reference"},"kibana.alert.rule.threat.technique.subtechnique.id":{"type":"alias","path":"signal.rule.threat.technique.subtechnique.id"},"kibana.alert.rule.threat.technique.subtechnique.name":{"type":"alias","path":"signal.rule.threat.technique.subtechnique.name"},"kibana.alert.rule.threat.technique.subtechnique.reference":{"type":"alias","path":"signal.rule.threat.technique.subtechnique.reference"},"kibana.alert.rule.timeline_id":{"type":"alias","path":"signal.rule.timeline_id"},"kibana.alert.rule.timeline_title":{"type":"alias","path":"signal.rule.timeline_title"},"kibana.alert.rule.timestamp_override":{"type":"alias","path":"signal.rule.timestamp_override"},"kibana.alert.rule.to":{"type":"alias","path":"signal.rule.to"},"kibana.alert.rule.type":{"type":"alias","path":"signal.rule.type"},"kibana.alert.rule.updated_at":{"type":"alias","path":"signal.rule.updated_at"},"kibana.alert.rule.updated_by":{"type":"alias","path":"signal.rule.updated_by"},"kibana.alert.rule.version":{"type":"alias","path":"signal.rule.version"},"kibana.alert.workflow_status":{"type":"alias","path":"signal.status"},"kibana.alert.threshold_result.from":{"type":"alias","path":"signal.threshold_result.from"},"kibana.alert.threshold_result.terms.field":{"type":"alias","path":"signal.threshold_result.terms.field"},"kibana.alert.threshold_result.terms.value":{"type":"alias","path":"signal.threshold_result.terms.value"},"kibana.alert.threshold_result.cardinality.field":{"type":"alias","path":"signal.threshold_result.cardinality.field"},"kibana.alert.threshold_result.cardinality.value":{"type":"alias","path":"signal.threshold_result.cardinality.value"},"kibana.alert.threshold_result.count":{"type":"alias","path":"signal.threshold_result.count"},"kibana.space_ids":{"type":"constant_keyword","value":"default"}},"dynamic":false,"_meta":{"version":57,"aliases_version":4}} [mapper_parsing_exception]: Alias [kibana.alert.severity] is defined both as an alias and a concrete field {"http":{"request":{"id":"c1bd32b8-6835-4fb1-bfba-4895b1e21ba7","method":"PUT","headers":{"user-agent":"Kibana/8.13.0","x-elastic-product-origin":"kibana","authorization":"[REDACTED]","x-opaque-id":"c1bd32b8-6835-4fb1-bfba-4895b1e21ba7","x-elastic-client-meta":"es=8.9.1p,js=20.10.0,t=8.3.3,hc=20.10.0","content-type":"application/vnd.elasticsearch+json; compatible-with=8","accept":"application/vnd.elasticsearch+json; compatible-with=8","content-length":"8697"}},"response":{"body":{"bytes":289},"status_code":400,"headers":{"x-opaque-id":"c1bd32b8-6835-4fb1-bfba-4895b1e21ba7","x-elastic-product":"Elasticsearch","content-type":"application/vnd.elasticsearch+json;compatible-with=8","content-length":"289"}}},"url":{"path":"/.siem-signals-default-000001-7.16.0/_mapping","query":"allow_no_indices=true"},"service":{"node":{"roles":["background_tasks","ui"]}}}
 proc [kibana] [2024-02-07T16:29:16.559+00:00][DEBUG][elasticsearch.query.data] 200 - 1.4KB

No request to create alias for this test though. Something went wrong, probably some GET request did not return results

vitaliidm commented 9 months ago

Looking deeper into logs, I can see put mapping is failing

proc [kibana] PUT /.siem-signals-default-000001-7.16.0/_mapping?allow_no_indices=true
 proc [kibana] {"properties":{"signal":{"type":"object","properties":{"_meta":{"type":"object","properties":{"version":{"type":"long"}}},"ancestors":{"properties":{"rule":{"type":"keyword"},"index":{"type":"keyword"},"id":{"type":"keyword"},"type":{"type":"keyword"},"depth":{"type":"long"}}},"depth":{"type":"integer"},"group":{"type":"object","properties":{"id":{"type":"keyword"},"index":{"type":"integer"}}},"original_event":{"type":"object","properties":{"reason":{"type":"keyword"}}},"reason":{"type":"keyword"},"rule":{"type":"object","properties":{"author":{"type":"keyword"},"building_block_type":{"type":"keyword"},"license":{"type":"keyword"},"note":{"type":"text"},"risk_score_mapping":{"type":"object","properties":{"field":{"type":"keyword"},"operator":{"type":"keyword"},"value":{"type":"keyword"}}},"rule_name_override":{"type":"keyword"},"severity_mapping":{"type":"object","properties":{"field":{"type":"keyword"},"operator":{"type":"keyword"},"value":{"type":"keyword"},"severity":{"type":"keyword"}}},"threat":{"type":"object","properties":{"technique":{"type":"object","properties":{"subtechnique":{"type":"object","properties":{"id":{"type":"keyword"},"name":{"type":"keyword"},"reference":{"type":"keyword"}}}}}}},"threat_index":{"type":"keyword"},"threat_indicator_path":{"type":"keyword"},"threat_language":{"type":"keyword"},"threat_mapping":{"type":"object","properties":{"entries":{"type":"object","properties":{"field":{"type":"keyword"},"value":{"type":"keyword"},"type":{"type":"keyword"}}}}},"threat_query":{"type":"keyword"},"threshold":{"type":"object","properties":{"field":{"type":"keyword"},"value":{"type":"float"}}},"timestamp_override":{"type":"keyword"}}},"threshold_result":{"properties":{"from":{"type":"date"},"terms":{"properties":{"field":{"type":"keyword"},"value":{"type":"keyword"}}},"cardinality":{"properties":{"field":{"type":"keyword"},"value":{"type":"long"}}},"count":{"type":"long"}}}}},"kibana.alert.ancestors.depth":{"type":"alias","path":"signal.ancestors.depth"},"kibana.alert.ancestors.id":{"type":"alias","path":"signal.ancestors.id"},"kibana.alert.ancestors.index":{"type":"alias","path":"signal.ancestors.index"},"kibana.alert.ancestors.type":{"type":"alias","path":"signal.ancestors.type"},"kibana.alert.depth":{"type":"alias","path":"signal.depth"},"kibana.alert.group.id":{"type":"alias","path":"signal.group.id"},"kibana.alert.group.index":{"type":"alias","path":"signal.group.index"},"kibana.alert.original_event.action":{"type":"alias","path":"signal.original_event.action"},"kibana.alert.original_event.category":{"type":"alias","path":"signal.original_event.category"},"kibana.alert.original_event.code":{"type":"alias","path":"signal.original_event.code"},"kibana.alert.original_event.created":{"type":"alias","path":"signal.original_event.created"},"kibana.alert.original_event.dataset":{"type":"alias","path":"signal.original_event.dataset"},"kibana.alert.original_event.duration":{"type":"alias","path":"signal.original_event.duration"},"kibana.alert.original_event.end":{"type":"alias","path":"signal.original_event.end"},"kibana.alert.original_event.hash":{"type":"alias","path":"signal.original_event.hash"},"kibana.alert.original_event.id":{"type":"alias","path":"signal.original_event.id"},"kibana.alert.original_event.kind":{"type":"alias","path":"signal.original_event.kind"},"kibana.alert.original_event.module":{"type":"alias","path":"signal.original_event.module"},"kibana.alert.original_event.outcome":{"type":"alias","path":"signal.original_event.outcome"},"kibana.alert.original_event.provider":{"type":"alias","path":"signal.original_event.provider"},"kibana.alert.original_event.reason":{"type":"alias","path":"signal.original_event.reason"},"kibana.alert.original_event.risk_score":{"type":"alias","path":"signal.original_event.risk_score"},"kibana.alert.original_event.risk_score_norm":{"type":"alias","path":"signal.original_event.risk_score_norm"},"kibana.alert.original_event.sequence":{"type":"alias","path":"signal.original_event.sequence"},"kibana.alert.original_event.severity":{"type":"alias","path":"signal.original_event.severity"},"kibana.alert.original_event.start":{"type":"alias","path":"signal.original_event.start"},"kibana.alert.original_event.timezone":{"type":"alias","path":"signal.original_event.timezone"},"kibana.alert.original_event.type":{"type":"alias","path":"signal.original_event.type"},"kibana.alert.original_time":{"type":"alias","path":"signal.original_time"},"kibana.alert.reason":{"type":"alias","path":"signal.reason"},"kibana.alert.rule.author":{"type":"alias","path":"signal.rule.author"},"kibana.alert.building_block_type":{"type":"alias","path":"signal.rule.building_block_type"},"kibana.alert.rule.created_at":{"type":"alias","path":"signal.rule.created_at"},"kibana.alert.rule.created_by":{"type":"alias","path":"signal.rule.created_by"},"kibana.alert.rule.description":{"type":"alias","path":"signal.rule.description"},"kibana.alert.rule.enabled":{"type":"alias","path":"signal.rule.enabled"},"kibana.alert.rule.false_positives":{"type":"alias","path":"signal.rule.false_positives"},"kibana.alert.rule.from":{"type":"alias","path":"signal.rule.from"},"kibana.alert.rule.uuid":{"type":"alias","path":"signal.rule.id"},"kibana.alert.rule.immutable":{"type":"alias","path":"signal.rule.immutable"},"kibana.alert.rule.interval":{"type":"alias","path":"signal.rule.interval"},"kibana.alert.rule.license":{"type":"alias","path":"signal.rule.license"},"kibana.alert.rule.max_signals":{"type":"alias","path":"signal.rule.max_signals"},"kibana.alert.rule.name":{"type":"alias","path":"signal.rule.name"},"kibana.alert.rule.note":{"type":"alias","path":"signal.rule.note"},"kibana.alert.rule.references":{"type":"alias","path":"signal.rule.references"},"kibana.alert.risk_score":{"type":"alias","path":"signal.rule.risk_score"},"kibana.alert.rule.rule_id":{"type":"alias","path":"signal.rule.rule_id"},"kibana.alert.rule.rule_name_override":{"type":"alias","path":"signal.rule.rule_name_override"},"kibana.alert.severity":{"type":"alias","path":"signal.rule.severity"},"kibana.alert.rule.tags":{"type":"alias","path":"signal.rule.tags"},"kibana.alert.rule.threat.framework":{"type":"alias","path":"signal.rule.threat.framework"},"kibana.alert.rule.threat.tactic.id":{"type":"alias","path":"signal.rule.threat.tactic.id"},"kibana.alert.rule.threat.tactic.name":{"type":"alias","path":"signal.rule.threat.tactic.name"},"kibana.alert.rule.threat.tactic.reference":{"type":"alias","path":"signal.rule.threat.tactic.reference"},"kibana.alert.rule.threat.technique.id":{"type":"alias","path":"signal.rule.threat.technique.id"},"kibana.alert.rule.threat.technique.name":{"type":"alias","path":"signal.rule.threat.technique.name"},"kibana.alert.rule.threat.technique.reference":{"type":"alias","path":"signal.rule.threat.technique.reference"},"kibana.alert.rule.threat.technique.subtechnique.id":{"type":"alias","path":"signal.rule.threat.technique.subtechnique.id"},"kibana.alert.rule.threat.technique.subtechnique.name":{"type":"alias","path":"signal.rule.threat.technique.subtechnique.name"},"kibana.alert.rule.threat.technique.subtechnique.reference":{"type":"alias","path":"signal.rule.threat.technique.subtechnique.reference"},"kibana.alert.rule.timeline_id":{"type":"alias","path":"signal.rule.timeline_id"},"kibana.alert.rule.timeline_title":{"type":"alias","path":"signal.rule.timeline_title"},"kibana.alert.rule.timestamp_override":{"type":"alias","path":"signal.rule.timestamp_override"},"kibana.alert.rule.to":{"type":"alias","path":"signal.rule.to"},"kibana.alert.rule.type":{"type":"alias","path":"signal.rule.type"},"kibana.alert.rule.updated_at":{"type":"alias","path":"signal.rule.updated_at"},"kibana.alert.rule.updated_by":{"type":"alias","path":"signal.rule.updated_by"},"kibana.alert.rule.version":{"type":"alias","path":"signal.rule.version"},"kibana.alert.workflow_status":{"type":"alias","path":"signal.status"},"kibana.alert.threshold_result.from":{"type":"alias","path":"signal.threshold_result.from"},"kibana.alert.threshold_result.terms.field":{"type":"alias","path":"signal.threshold_result.terms.field"},"kibana.alert.threshold_result.terms.value":{"type":"alias","path":"signal.threshold_result.terms.value"},"kibana.alert.threshold_result.cardinality.field":{"type":"alias","path":"signal.threshold_result.cardinality.field"},"kibana.alert.threshold_result.cardinality.value":{"type":"alias","path":"signal.threshold_result.cardinality.value"},"kibana.alert.threshold_result.count":{"type":"alias","path":"signal.threshold_result.count"},"kibana.space_ids":{"type":"constant_keyword","value":"default"}},"dynamic":false,"_meta":{"version":57,"aliases_version":4}} [mapper_parsing_exception]: Alias [kibana.alert.severity] is defined both as an alias and a concrete field {"http":{"request":{"id":"c1bd32b8-6835-4fb1-bfba-4895b1e21ba7","method":"PUT","headers":{"user-agent":"Kibana/8.13.0","x-elastic-product-origin":"kibana","authorization":"[REDACTED]","x-opaque-id":"c1bd32b8-6835-4fb1-bfba-4895b1e21ba7","x-elastic-client-meta":"es=8.9.1p,js=20.10.0,t=8.3.3,hc=20.10.0","content-type":"application/vnd.elasticsearch+json; compatible-with=8","accept":"application/vnd.elasticsearch+json; compatible-with=8","content-length":"8697"}},"response":{"body":{"bytes":289},"status_code":400,"headers":{"x-opaque-id":"c1bd32b8-6835-4fb1-bfba-4895b1e21ba7","x-elastic-product":"Elasticsearch","content-type":"application/vnd.elasticsearch+json;compatible-with=8","content-length":"289"}}},"url":{"path":"/.siem-signals-default-000001-7.16.0/_mapping","query":"allow_no_indices=true"},"service":{"node":{"roles":["background_tasks","ui"]}}}

[mapper_parsing_exception]: Alias [kibana.alert.severity] is defined both as an alias and a concrete field {"http":{"request":{"id":"c1bd32b8-6835-4fb1-bfba-4895b1e21ba7","method":"PUT","headers":{"user-agent":"Kibana/8.13.0","x-elastic-product-origin":"kibana","authorization":"[REDACTED]","x-opaque-id":"c1bd32b8-6835-4fb1-bfba-4895b1e21ba7","x-elastic-client-meta":"es=8.9.1p,js=20.10.0,t=8.3.3,hc=20.10.0","content-type":"application/vnd.elasticsearch+json; compatible-with=8","accept":"application/vnd.elasticsearch+json; compatible-with=8","content-length":"8697"}},"response":{"body":{"bytes":289},"status_code":400,"headers":{"x-opaque-id":"c1bd32b8-6835-4fb1-bfba-4895b1e21ba7","x-elastic-product":"Elasticsearch","content-type":"application/vnd.elasticsearch+json;compatible-with=8","content-length":"289"}}},"url":{"path":"/.siem-signals-default-000001-7.16.0/_mapping","query":"allow_no_indices=true"},"service":{"node":{"roles":["background_tasks","ui"]}}}

That prevents, further add alias request to be performed

Exactly the same query works fine for the earlier build

vitaliidm commented 9 months ago

Failing PUT mapping query from above comment

request ``` PUT /.siem-signals-default-000001-7.16.0/_mapping?allow_no_indices=true { "properties": { "signal": { "type": "object", "properties": { "_meta": { "type": "object", "properties": { "version": { "type": "long" } } }, "ancestors": { "properties": { "rule": { "type": "keyword" }, "index": { "type": "keyword" }, "id": { "type": "keyword" }, "type": { "type": "keyword" }, "depth": { "type": "long" } } }, "depth": { "type": "integer" }, "group": { "type": "object", "properties": { "id": { "type": "keyword" }, "index": { "type": "integer" } } }, "original_event": { "type": "object", "properties": { "reason": { "type": "keyword" } } }, "reason": { "type": "keyword" }, "rule": { "type": "object", "properties": { "author": { "type": "keyword" }, "building_block_type": { "type": "keyword" }, "license": { "type": "keyword" }, "note": { "type": "text" }, "risk_score_mapping": { "type": "object", "properties": { "field": { "type": "keyword" }, "operator": { "type": "keyword" }, "value": { "type": "keyword" } } }, "rule_name_override": { "type": "keyword" }, "severity_mapping": { "type": "object", "properties": { "field": { "type": "keyword" }, "operator": { "type": "keyword" }, "value": { "type": "keyword" }, "severity": { "type": "keyword" } } }, "threat": { "type": "object", "properties": { "technique": { "type": "object", "properties": { "subtechnique": { "type": "object", "properties": { "id": { "type": "keyword" }, "name": { "type": "keyword" }, "reference": { "type": "keyword" } } } } } } }, "threat_index": { "type": "keyword" }, "threat_indicator_path": { "type": "keyword" }, "threat_language": { "type": "keyword" }, "threat_mapping": { "type": "object", "properties": { "entries": { "type": "object", "properties": { "field": { "type": "keyword" }, "value": { "type": "keyword" }, "type": { "type": "keyword" } } } } }, "threat_query": { "type": "keyword" }, "threshold": { "type": "object", "properties": { "field": { "type": "keyword" }, "value": { "type": "float" } } }, "timestamp_override": { "type": "keyword" } } }, "threshold_result": { "properties": { "from": { "type": "date" }, "terms": { "properties": { "field": { "type": "keyword" }, "value": { "type": "keyword" } } }, "cardinality": { "properties": { "field": { "type": "keyword" }, "value": { "type": "long" } } }, "count": { "type": "long" } } } } }, "kibana.alert.ancestors.depth": { "type": "alias", "path": "signal.ancestors.depth" }, "kibana.alert.ancestors.id": { "type": "alias", "path": "signal.ancestors.id" }, "kibana.alert.ancestors.index": { "type": "alias", "path": "signal.ancestors.index" }, "kibana.alert.ancestors.type": { "type": "alias", "path": "signal.ancestors.type" }, "kibana.alert.depth": { "type": "alias", "path": "signal.depth" }, "kibana.alert.group.id": { "type": "alias", "path": "signal.group.id" }, "kibana.alert.group.index": { "type": "alias", "path": "signal.group.index" }, "kibana.alert.original_event.action": { "type": "alias", "path": "signal.original_event.action" }, "kibana.alert.original_event.category": { "type": "alias", "path": "signal.original_event.category" }, "kibana.alert.original_event.code": { "type": "alias", "path": "signal.original_event.code" }, "kibana.alert.original_event.created": { "type": "alias", "path": "signal.original_event.created" }, "kibana.alert.original_event.dataset": { "type": "alias", "path": "signal.original_event.dataset" }, "kibana.alert.original_event.duration": { "type": "alias", "path": "signal.original_event.duration" }, "kibana.alert.original_event.end": { "type": "alias", "path": "signal.original_event.end" }, "kibana.alert.original_event.hash": { "type": "alias", "path": "signal.original_event.hash" }, "kibana.alert.original_event.id": { "type": "alias", "path": "signal.original_event.id" }, "kibana.alert.original_event.kind": { "type": "alias", "path": "signal.original_event.kind" }, "kibana.alert.original_event.module": { "type": "alias", "path": "signal.original_event.module" }, "kibana.alert.original_event.outcome": { "type": "alias", "path": "signal.original_event.outcome" }, "kibana.alert.original_event.provider": { "type": "alias", "path": "signal.original_event.provider" }, "kibana.alert.original_event.reason": { "type": "alias", "path": "signal.original_event.reason" }, "kibana.alert.original_event.risk_score": { "type": "alias", "path": "signal.original_event.risk_score" }, "kibana.alert.original_event.risk_score_norm": { "type": "alias", "path": "signal.original_event.risk_score_norm" }, "kibana.alert.original_event.sequence": { "type": "alias", "path": "signal.original_event.sequence" }, "kibana.alert.original_event.severity": { "type": "alias", "path": "signal.original_event.severity" }, "kibana.alert.original_event.start": { "type": "alias", "path": "signal.original_event.start" }, "kibana.alert.original_event.timezone": { "type": "alias", "path": "signal.original_event.timezone" }, "kibana.alert.original_event.type": { "type": "alias", "path": "signal.original_event.type" }, "kibana.alert.original_time": { "type": "alias", "path": "signal.original_time" }, "kibana.alert.reason": { "type": "alias", "path": "signal.reason" }, "kibana.alert.rule.author": { "type": "alias", "path": "signal.rule.author" }, "kibana.alert.building_block_type": { "type": "alias", "path": "signal.rule.building_block_type" }, "kibana.alert.rule.created_at": { "type": "alias", "path": "signal.rule.created_at" }, "kibana.alert.rule.created_by": { "type": "alias", "path": "signal.rule.created_by" }, "kibana.alert.rule.description": { "type": "alias", "path": "signal.rule.description" }, "kibana.alert.rule.enabled": { "type": "alias", "path": "signal.rule.enabled" }, "kibana.alert.rule.false_positives": { "type": "alias", "path": "signal.rule.false_positives" }, "kibana.alert.rule.from": { "type": "alias", "path": "signal.rule.from" }, "kibana.alert.rule.uuid": { "type": "alias", "path": "signal.rule.id" }, "kibana.alert.rule.immutable": { "type": "alias", "path": "signal.rule.immutable" }, "kibana.alert.rule.interval": { "type": "alias", "path": "signal.rule.interval" }, "kibana.alert.rule.license": { "type": "alias", "path": "signal.rule.license" }, "kibana.alert.rule.max_signals": { "type": "alias", "path": "signal.rule.max_signals" }, "kibana.alert.rule.name": { "type": "alias", "path": "signal.rule.name" }, "kibana.alert.rule.note": { "type": "alias", "path": "signal.rule.note" }, "kibana.alert.rule.references": { "type": "alias", "path": "signal.rule.references" }, "kibana.alert.risk_score": { "type": "alias", "path": "signal.rule.risk_score" }, "kibana.alert.rule.rule_id": { "type": "alias", "path": "signal.rule.rule_id" }, "kibana.alert.rule.rule_name_override": { "type": "alias", "path": "signal.rule.rule_name_override" }, "kibana.alert.severity": { "type": "alias", "path": "signal.rule.severity" }, "kibana.alert.rule.tags": { "type": "alias", "path": "signal.rule.tags" }, "kibana.alert.rule.threat.framework": { "type": "alias", "path": "signal.rule.threat.framework" }, "kibana.alert.rule.threat.tactic.id": { "type": "alias", "path": "signal.rule.threat.tactic.id" }, "kibana.alert.rule.threat.tactic.name": { "type": "alias", "path": "signal.rule.threat.tactic.name" }, "kibana.alert.rule.threat.tactic.reference": { "type": "alias", "path": "signal.rule.threat.tactic.reference" }, "kibana.alert.rule.threat.technique.id": { "type": "alias", "path": "signal.rule.threat.technique.id" }, "kibana.alert.rule.threat.technique.name": { "type": "alias", "path": "signal.rule.threat.technique.name" }, "kibana.alert.rule.threat.technique.reference": { "type": "alias", "path": "signal.rule.threat.technique.reference" }, "kibana.alert.rule.threat.technique.subtechnique.id": { "type": "alias", "path": "signal.rule.threat.technique.subtechnique.id" }, "kibana.alert.rule.threat.technique.subtechnique.name": { "type": "alias", "path": "signal.rule.threat.technique.subtechnique.name" }, "kibana.alert.rule.threat.technique.subtechnique.reference": { "type": "alias", "path": "signal.rule.threat.technique.subtechnique.reference" }, "kibana.alert.rule.timeline_id": { "type": "alias", "path": "signal.rule.timeline_id" }, "kibana.alert.rule.timeline_title": { "type": "alias", "path": "signal.rule.timeline_title" }, "kibana.alert.rule.timestamp_override": { "type": "alias", "path": "signal.rule.timestamp_override" }, "kibana.alert.rule.to": { "type": "alias", "path": "signal.rule.to" }, "kibana.alert.rule.type": { "type": "alias", "path": "signal.rule.type" }, "kibana.alert.rule.updated_at": { "type": "alias", "path": "signal.rule.updated_at" }, "kibana.alert.rule.updated_by": { "type": "alias", "path": "signal.rule.updated_by" }, "kibana.alert.rule.version": { "type": "alias", "path": "signal.rule.version" }, "kibana.alert.workflow_status": { "type": "alias", "path": "signal.status" }, "kibana.alert.threshold_result.from": { "type": "alias", "path": "signal.threshold_result.from" }, "kibana.alert.threshold_result.terms.field": { "type": "alias", "path": "signal.threshold_result.terms.field" }, "kibana.alert.threshold_result.terms.value": { "type": "alias", "path": "signal.threshold_result.terms.value" }, "kibana.alert.threshold_result.cardinality.field": { "type": "alias", "path": "signal.threshold_result.cardinality.field" }, "kibana.alert.threshold_result.cardinality.value": { "type": "alias", "path": "signal.threshold_result.cardinality.value" }, "kibana.alert.threshold_result.count": { "type": "alias", "path": "signal.threshold_result.count" }, "kibana.space_ids": { "type": "constant_keyword", "value": "default" } }, "dynamic": false, "_meta": { "version": 57, "aliases_version": 4 } } ```

response

{
  "error": {
    "root_cause": [
      {
        "type": "mapper_parsing_exception",
        "reason": "Alias [kibana.alert.rule.created_by] is defined both as an alias and a concrete field"
      }
    ],
    "type": "mapper_parsing_exception",
    "reason": "Alias [kibana.alert.rule.created_by] is defined both as an alias and a concrete field"
  },
  "status": 400
}

Same query works without errors for ES snapshots dated earlier than 1st of February

mattc58 commented 9 months ago

We're ruling out elasticsearch#104145. The actual error here is happening before the index alias is retrieved, and this test is working on 8.12 which also has that fix in it.

elasticsearch#103648 touched code that could be in this area and we'll investigate that.

yctercero commented 9 months ago

Confirmed fix upstream https://github.com/elastic/kibana/pull/176661