search

elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.69k stars 8.24k forks source link

Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/rule_execution_logic/threat_match·ts - detection engine api security and spaces enabled - rule execution logic Threat match type rules terms and match should have the same alerts with pagination #155304

Closed kibanamachine closed 1 year ago

kibanamachine commented 1 year ago

A test failed on a tracked branch

Error: expected [ { 'kibana.version': '8.8.0-SNAPSHOT',
    'kibana.alert.rule.category': 'Indicator Match Rule',
    'kibana.alert.rule.consumer': 'siem',
    'kibana.alert.rule.producer': 'siem',
    'kibana.alert.rule.revision': 0,
    'kibana.alert.rule.rule_type_id': 'siem.indicatorRule',
    'kibana.space_ids': [ 'default' ],
    'kibana.alert.rule.tags': [],
    agent: 
     { ephemeral_id: '1b4978a0-48be-49b1-ac96-323425b389ab',
       hostname: 'zeek-sensor-amsterdam',
       id: 'e52588e6-7aa3-4c89-a2c4-d6bc5c286db1',
       type: 'auditbeat',
       version: '8.0.0' },
    auditd: 
     { data: [Object],
       message_type: 'user_err',
       result: 'fail',
       sequence: 2267,
       session: 'unset',
       summary: [Object] },
    cloud: 
     { instance: [Object],
       provider: 'digitalocean',
       region: 'ams3' },
    ecs: { version: '1.0.0-beta2' },
    host: 
     { architecture: 'x86_64',
       containerized: false,
       hostname: 'zeek-sensor-amsterdam',
       id: '2ce8b1e7d69e4a1d9c6bcddc473da9d9',
       name: 'zeek-sensor-amsterdam',
       os: [Object] },
    network: { direction: 'incoming' },
    process: { executable: '/usr/sbin/sshd', pid: 32739 },
    service: { type: 'auditd' },
    source: { ip: '46.101.47.213' },
    user: { audit: [Object], id: '0', name: 'root' },
    threat: { enrichments: [Object] },
    'event.action': 'error',
    'event.category': 'user-login',
    'event.module': 'auditd',
    'event.kind': 'signal',
    'kibana.alert.original_time': '2019-02-19T20:42:05.202Z',
    'kibana.alert.ancestors': [ [Object] ],
    'kibana.alert.status': 'active',
    'kibana.alert.workflow_status': 'open',
    'kibana.alert.depth': 1,
    'kibana.alert.severity': 'high',
    'kibana.alert.risk_score': 55,
    'kibana.alert.rule.actions': [],
    'kibana.alert.rule.author': [],
    'kibana.alert.rule.created_by': 'elastic',
    'kibana.alert.rule.description': 'Detecting root and admin users',
    'kibana.alert.rule.enabled': true,
    'kibana.alert.rule.exceptions_list': [],
    'kibana.alert.rule.false_positives': [],
    'kibana.alert.rule.from': '1900-01-01T00:00:00.000Z',
    'kibana.alert.rule.immutable': false,
    'kibana.alert.rule.interval': '5m',
    'kibana.alert.rule.indices': [ 'auditbeat-*' ],
    'kibana.alert.rule.max_signals': 100,
    'kibana.alert.rule.references': [],
    'kibana.alert.rule.risk_score_mapping': [],
    'kibana.alert.rule.severity_mapping': [],
    'kibana.alert.rule.threat': [],
    'kibana.alert.rule.to': 'now',
    'kibana.alert.rule.type': 'threat_match',
    'kibana.alert.rule.updated_by': 'elastic',
    'kibana.alert.rule.version': 1,
    'kibana.alert.rule.risk_score': 55,
    'kibana.alert.rule.severity': 'high',
    'kibana.alert.original_event.action': 'error',
    'kibana.alert.original_event.category': 'user-login',
    'kibana.alert.original_event.module': 'auditd' },
  { 'kibana.version': '8.8.0-SNAPSHOT',
    'kibana.alert.rule.category': 'Indicator Match Rule',
    'kibana.alert.rule.consumer': 'siem',
    'kibana.alert.rule.producer': 'siem',
    'kibana.alert.rule.revision': 0,
    'kibana.alert.rule.rule_type_id': 'siem.indicatorRule',
    'kibana.space_ids': [ 'default' ],
    'kibana.alert.rule.tags': [],
    agent: 
     { ephemeral_id: '1b4978a0-48be-49b1-ac96-323425b389ab',
       hostname: 'zeek-sensor-amsterdam',
       id: 'e52588e6-7aa3-4c89-a2c4-d6bc5c286db1',
       type: 'auditbeat',
       version: '8.0.0' },
    auditd: 
     { data: [Object],
       message_type: 'user_login',
       result: 'fail',
       sequence: 2266,
       session: 'unset',
       summary: [Object] },
    cloud: 
     { instance: [Object],
       provider: 'digitalocean',
       region: 'ams3' },
    ecs: { version: '1.0.0-beta2' },
    host: 
     { architecture: 'x86_64',
       containerized: false,
       hostname: 'zeek-sensor-amsterdam',
       id: '2ce8b1e7d69e4a1d9c6bcddc473da9d9',
       name: 'zeek-sensor-amsterdam',
       os: [Object] },
    network: { direction: 'incoming' },
    process: { executable: '/usr/sbin/sshd', pid: 32739 },
    service: { type: 'auditd' },
    source: { ip: '46.101.47.213' },
    user: { audit: [Object], id: '0', name: 'root' },
    threat: { enrichments: [Object] },
    'event.action': 'logged-in',
    'event.category': 'user-login',
    'event.module': 'auditd',
    'event.kind': 'signal',
    'kibana.alert.original_time': '2019-02-19T20:42:05.194Z',
    'kibana.alert.ancestors': [ [Object] ],
    'kibana.alert.status': 'active',
    'kibana.alert.workflow_status': 'open',
    'kibana.alert.depth': 1,
    'kibana.alert.severity': 'high',
    'kibana.alert.risk_score': 55,
    'kibana.alert.rule.actions': [],
    'kibana.alert.rule.author': [],
    'kibana.alert.rule.created_by': 'elastic',
    'kibana.alert.rule.description': 'Detecting root and admin users',
    'kibana.alert.rule.enabled': true,
    'kibana.alert.rule.exceptions_list': [],
    'kibana.alert.rule.false_positives': [],
    'kibana.alert.rule.from': '1900-01-01T00:00:00.000Z',
    'kibana.alert.rule.immutable': false,
    'kibana.alert.rule.interval': '5m',
    'kibana.alert.rule.indices': [ 'auditbeat-*' ],
    'kibana.alert.rule.max_signals': 100,
    'kibana.alert.rule.references': [],
    'kibana.alert.rule.risk_score_mapping': [],
    'kibana.alert.rule.severity_mapping': [],
    'kibana.alert.rule.threat': [],
    'kibana.alert.rule.to': 'now',
    'kibana.alert.rule.type': 'threat_match',
    'kibana.alert.rule.updated_by': 'elastic',
    'kibana.alert.rule.version': 1,
    'kibana.alert.rule.risk_score': 55,
    'kibana.alert.rule.severity': 'high',
    'kibana.alert.original_event.action': 'logged-in',
    'kibana.alert.original_event.category': 'user-login',
    'kibana.alert.original_event.module': 'auditd' },
  { 'kibana.version': '8.8.0-SNAPSHOT',
    'kibana.alert.rule.category': 'Indicator Match Rule',
    'kibana.alert.rule.consumer': 'siem',
    'kibana.alert.rule.producer': 'siem',
    'kibana.alert.rule.revision': 0,
    'kibana.alert.rule.rule_type_id': 'siem.indicatorRule',
    'kibana.space_ids': [ 'default' ],
    'kibana.alert.rule.tags': [],
    agent: 
     { ephemeral_id: '1b4978a0-48be-49b1-ac96-323425b389ab',
       hostname: 'zeek-sensor-amsterdam',
       id: 'e52588e6-7aa3-4c89-a2c4-d6bc5c286db1',
       type: 'auditbeat',
       version: '8.0.0' },
    auditd: 
     { data: [Object],
       message_type: 'user_login',
       result: 'fail',
       sequence: 2265,
       session: 'unset',
       summary: [Object] },
    cloud: 
     { instance: [Object],
       provider: 'digitalocean',
       region: 'ams3' },
    ecs: { version: '1.0.0-beta2' },
    host: 
     { architecture: 'x86_64',
       containerized: false,
       hostname: 'zeek-sensor-amsterdam',
       id: '2ce8b1e7d69e4a1d9c6bcddc473da9d9',
       name: 'zeek-sensor-amsterdam',
       os: [Object] },
    network: { direction: 'incoming' },
    process: { executable: '/usr/sbin/sshd', pid: 32739 },
    service: { type: 'auditd' },
    source: { ip: '46.101.47.213' },
    user: { audit: [Object], id: '0', name: 'root' },
    threat: { enrichments: [Object] },
    'event.action': 'logged-in',
    'event.category': 'user-login',
    'event.module': 'auditd',
    'event.kind': 'signal',
    'kibana.alert.original_time': '2019-02-19T20:42:05.190Z',
    'kibana.alert.ancestors': [ [Object] ],
    'kibana.alert.status': 'active',
    'kibana.alert.workflow_status': 'open',
    'kibana.alert.depth': 1,
    'kibana.alert.severity': 'high',
    'kibana.alert.risk_score': 55,
    'kibana.alert.rule.actions': [],
    'kibana.alert.rule.author': [],
    'kibana.alert.rule.created_by': 'elastic',
    'kibana.alert.rule.description': 'Detecting root and admin users',
    'kibana.alert.rule.enabled': true,
    'kibana.alert.rule.exceptions_list': [],
    'kibana.alert.rule.false_positives': [],
    'kibana.alert.rule.from': '1900-01-01T00:00:00.000Z',
    'kibana.alert.rule.immutable': false,
    'kibana.alert.rule.interval': '5m',
    'kibana.alert.rule.indices': [ 'auditbeat-*' ],
    'kibana.alert.rule.max_signals': 100,
    'kibana.alert.rule.references': [],
[report_failure] output truncated to 8192 characters

First failure: CI Build - main

elasticmachine commented 1 year ago

Pinging @elastic/security-solution (Team: SecuritySolution)

elasticmachine commented 1 year ago

Pinging @elastic/security-detections-response (Team:Detections and Resp)

kibanamachine commented 1 year ago

New failure: CI Build - main

jbudz commented 1 year ago

/skip

kibanamachine commented 1 year ago

New failure: CI Build - 8.8

kibanamachine commented 1 year ago

New failure: CI Build - main

jbudz commented 1 year ago

Skipped

main: https://github.com/elastic/kibana/commit/24927cb061d264708e62ac69fbf84c1010bdf1ec 8.8: https://github.com/elastic/kibana/commit/d050720490c57ca53d4c66a0233beb4cd3b3a2e7

rylnd commented 1 year ago

For posterity: this test was introduced to cover the changes in https://github.com/elastic/kibana/pull/144511.

As an aside: the failure message for this test is not helpful as the diff is too large to output. Perhaps it would make sense to compare these alerts in series, so that the failure shows two single objects? Just a thought.

yctercero commented 1 year ago

Merged in the PR unskipping these as running them multiple times locally and on the flakey test runner did not reveal any failures.

Lets keep in mind @rylnd 's suggestion if this pops up again.

PR: https://github.com/elastic/kibana/pull/156489

rylnd commented 1 year ago

If anyone happens to have reproduced this locally and was able to actually observe full failure message, please share here! Do we think this was simply a false negative?

kibanamachine commented 1 year ago

New failure: CI Build - main

kibanamachine commented 1 year ago

New failure: CI Build - main

mistic commented 1 year ago

Skipped.

main: 39dad65

yctercero commented 1 year ago

Unskipped here https://github.com/elastic/kibana/pull/160094

kibanamachine commented 1 year ago

New failure: CI Build - main

MadameSheema commented 1 year ago

Hey @yctercero may you please take a look at this new failure when you have the chance? Thanks!

yctercero commented 1 year ago

@MadameSheema the detections team is working 8.10 to address all open flakey and skipped tests. Will add this one to the list.

nkhristinin commented 1 year ago

There another PR which should address it

kibanamachine commented 1 year ago

New failure: CI Build - main

WafaaNasr commented 1 year ago

Running it through flaky runner https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2625

WafaaNasr commented 1 year ago

After successfully running this test through the Flaky Test Runner for 100 iterations without any failures, it has been determined that the test is not truly flaky or failing. As a result, the ticket can be closed.