[Security Solution]Unable to attach Data Quality Dashboard to Cases.

sukhwindersingh-qasource commented 8 months ago

Describe the bug:

Unable to attach Data Quality Dashboard to Cases.

Kibana/Elasticsearch Stack version

VERSION: 8.13.0
BUILD: 71738
COMMIT: b036a9705a55f6c81d065011ad8c991cbc3101d9

Browser and Browser OS Version: Firefox for windows OS Version: 122.0 (64-bit)

Elastic Endpoint Version:

N/A

Original install method:

None

Functional Area:

Data Quality dashboard

Initial Setup:

None

precondition

Kibana should be running
Have large data on Data quality dashboard.
Check all should be completed.

Steps to reproduce

Navigates to Data Quality Dashboard
Now click on add to new case
Fill in the details and Click on create case.
Observe error for fetching data is thrown

Additional Information

We are using the build which is upgraded from 7.17.18 > 8.13.0

Current Result

Unable to attach Data Quality Dashboard to Cases.

Expected Result

User should be able to attach Data Quality Dashboard to Cases.

Screen-cast

https://github.com/elastic/kibana/assets/108654988/4e890992-076a-4971-90f1-0e01179addc6

Error massage

elasticmachine commented 8 months ago

Pinging @elastic/security-solution (Team: SecuritySolution)

sukhwindersingh-qasource commented 8 months ago

@arvindersingh-qasource kindly review this Thanks!

elasticmachine commented 8 months ago

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

elasticmachine commented 8 months ago

Pinging @elastic/security-threat-hunting-explore (Team:Threat Hunting:Explore)

semd commented 8 months ago

Hi @sukhwindersingh-qasource, thanks for reporting this.

Could you paste here the content from the Copy to clipboard when this error happens, please? This way we have the markup we are trying to insert into the case.

sukhwindersingh-qasource commented 8 months ago

Hi @semd Please find the copy to clipboard text for the same

showErrorToast/<@https://1d524b0b5c8240689912ba07d1ee9ac5.europe-west1.gcp.cloud.es.io:9243/b036a9705a55/bundles/plugin/cases/1.0.0/cases.plugin.js:2:87759
showErrorToast@https://1d524b0b5c8240689912ba07d1ee9ac5.europe-west1.gcp.cloud.es.io:9243/b036a9705a55/bundles/plugin/cases/1.0.0/cases.plugin.js:2:87820
onError@https://1d524b0b5c8240689912ba07d1ee9ac5.europe-west1.gcp.cloud.es.io:9243/b036a9705a55/bundles/plugin/cases/1.0.0/cases.plugin.js:2:128918
execute@https://1d524b0b5c8240689912ba07d1ee9ac5.europe-west1.gcp.cloud.es.io:9243/b036a9705a55/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:361:986807

Also pasting the Text mentioned in Red color (if required)

The length of the comment is too long. The maximum length is 30000.,Invalid value "user" supplied to "type",Invalid value "undefined" supplied to "alertId",Invalid value "undefined" supplied to "index",Invalid value "undefined" supplied to "rule",Invalid value "undefined" supplied to "actions",Invalid value "undefined" supplied to "externalReferenceAttachmentTypeId",Invalid value "undefined" supplied to "externalReferenceMetadata",Invalid value "undefined" supplied to "externalReferenceId",Invalid value "undefined" supplied to "externalReferenceStorage",Invalid value "undefined" supplied to "persistableStateAttachmentTypeId",Invalid value "undefined" supplied to "persistableStateAttachmentState"

Please let us know if anything else is required from our end. Thanks!

semd commented 8 months ago

@sukhwindersingh-qasource just to confirm, is this the text that gets copied to your clipboard when this button is clicked?

copy button

sukhwindersingh-qasource commented 8 months ago

Hi @semd Just a misunderstanding, No the text we pasted in the above comment was from the error prompt

Please find the text of copy to clipboard as mentioned in the screenshot :

# Data quality

| Incompatible fields | Indices checked | Indices | Size | Docs |
|---------------------|-----------------|---------|------|------|
| 9 | 49 | 49 | 31.6MB | 55,721 |

## .alerts-security.alerts-default
`hot(3)`

| Incompatible fields | Indices checked | Indices | Size | Docs |
|---------------------|-----------------|---------|------|------|
| 2 | 3 | 3 | 1.2MB | 28 |

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ❌ | .siem-signals-default-000001 | 17 (60.7%) | 1 | `hot` | 867.4KB |
| ✅ | .internal.alerts-security.alerts-default-000001 | 11 (39.3%) | 0 | `hot` | 385KB |
| ❌ | .siem-signals-default-000002 | 0 (0.0%) | 1 | `hot` | 249B |
### .siem-signals-default-000001

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ❌ | .siem-signals-default-000001 | 17 (60.7%) | 1 | `hot` | 867.4KB |

### **Incompatible fields** `1` **Same family** `18` **Custom fields** `332` **ECS compliant fields** `919` **All fields** `1270`

#### 1 incompatible field

Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.11.0.

❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported

#### Incompatible field mappings - .siem-signals-default-000001

| Field | ECS mapping type (expected) | Index mapping type (actual) | 
|-------|-----------------------------|-----------------------------|
| log.origin.file.line | `long` | `integer`  |

### .internal.alerts-security.alerts-default-000001

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ✅ | .internal.alerts-security.alerts-default-000001 | 11 (39.3%) | 0 | `hot` | 385KB |

### **Incompatible fields** `0` **Same family** `0` **Custom fields** `288` **ECS compliant fields** `1641` **All fields** `1929`

### .siem-signals-default-000002

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ❌ | .siem-signals-default-000002 | 0 (0.0%) | 1 | `hot` | 249B |

### **Incompatible fields** `1` **Same family** `18` **Custom fields** `312` **ECS compliant fields** `919` **All fields** `1250`

#### 1 incompatible field

Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.11.0.

❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported

#### Incompatible field mappings - .siem-signals-default-000002

| Field | ECS mapping type (expected) | Index mapping type (actual) | 
|-------|-----------------------------|-----------------------------|
| log.origin.file.line | `long` | `integer`  |

## apm-*-transaction*
`hot(1)`

| Incompatible fields | Indices checked | Indices | Size | Docs |
|---------------------|-----------------|---------|------|------|
| 0 | 1 | 1 | 249B | 0 |

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ✅ | apm-7.17.18-transaction-000001 | 0 () | 0 | `hot` | 249B |
### apm-7.17.18-transaction-000001

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ✅ | apm-7.17.18-transaction-000001 | 0 () | 0 | `hot` | 249B |

### **Incompatible fields** `0` **Same family** `3` **Custom fields** `311` **ECS compliant fields** `1119` **All fields** `1433`

## logs-*
`hot(45)`

| Incompatible fields | Indices checked | Indices | Size | Docs |
|---------------------|-----------------|---------|------|------|
| 7 | 45 | 45 | 30.3MB | 55,693 |

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ✅ | .ds-logs-elastic_agent.endpoint_security-default-2024.02.23-000002 | 22,040 (39.6%) | 0 | `hot` | 3.8MB |
| ✅ | .ds-logs-system.syslog-default-2024.02.23-000001 | 9,145 (16.4%) | 0 | `hot` | 1.3MB |
| ✅ | .ds-logs-elastic_agent.endpoint_security-default-2024.02.23-000001 | 3,378 (6.1%) | 0 | `hot` | 866.4KB |
| ✅ | .ds-logs-elastic_agent.filebeat-default-2024.02.23-000002 | 3,322 (6.0%) | 0 | `hot` | 2.3MB |
| ✅ | .ds-logs-elastic_agent.metricbeat-default-2024.02.23-000002 | 3,084 (5.5%) | 0 | `hot` | 2.1MB |
| ✅ | .ds-logs-system.security-default-2024.02.23-000001 | 2,854 (5.1%) | 0 | `hot` | 1.4MB |
| ✅ | .ds-logs-elastic_agent-default-2024.02.23-000002 | 1,308 (2.3%) | 0 | `hot` | 628.8KB |
| ✅ | .ds-logs-system.system-default-2024.02.23-000001 | 1,246 (2.2%) | 0 | `hot` | 583.8KB |
| ✅ | .ds-logs-endpoint.events.process-default-2024.02.23-000003 | 1,214 (2.2%) | 0 | `hot` | 2.2MB |
| ✅ | .ds-logs-endpoint.events.registry-default-2024.02.23-000002 | 1,096 (2.0%) | 0 | `hot` | 732.9KB |
| ✅ | .ds-logs-endpoint.events.network-default-2024.02.23-000004 | 1,049 (1.9%) | 0 | `hot` | 1.4MB |
| ✅ | .ds-logs-endpoint.events.file-default-2024.02.23-000004 | 879 (1.6%) | 0 | `hot` | 1.9MB |
| ✅ | .ds-logs-endpoint.events.process-default-2024.02.23-000001 | 769 (1.4%) | 0 | `hot` | 800.9KB |
| ✅ | .ds-logs-endpoint.events.registry-default-2024.02.23-000001 | 631 (1.1%) | 0 | `hot` | 527.1KB |
| ✅ | .ds-logs-system.auth-default-2024.02.23-000001 | 620 (1.1%) | 0 | `hot` | 363.1KB |
| ✅ | .ds-logs-endpoint.events.library-default-2024.02.23-000003 | 424 (0.8%) | 0 | `hot` | 876.2KB |
| ✅ | .ds-logs-elastic_agent.osquerybeat-default-2024.02.23-000001 | 379 (0.7%) | 0 | `hot` | 1.4MB |
| ✅ | .ds-logs-endpoint.events.library-default-2024.02.23-000001 | 341 (0.6%) | 0 | `hot` | 432.7KB |
| ✅ | .ds-logs-endpoint.events.file-default-2024.02.23-000001 | 313 (0.6%) | 0 | `hot` | 606.1KB |
| ✅ | .ds-logs-elastic_agent.filebeat-default-2024.02.23-000001 | 230 (0.4%) | 0 | `hot` | 632.2KB |
| ✅ | .ds-logs-system.syslog-default-2024.02.23-000002 | 218 (0.4%) | 0 | `hot` | 215.2KB |
| ✅ | .ds-logs-elastic_agent.metricbeat-default-2024.02.23-000001 | 206 (0.4%) | 0 | `hot` | 650.8KB |
| ✅ | .ds-logs-system.application-default-2024.02.23-000001 | 159 (0.3%) | 0 | `hot` | 228.6KB |
| ✅ | .ds-logs-system.auth-default-2024.02.23-000002 | 115 (0.2%) | 0 | `hot` | 569.9KB |
| ✅ | .ds-logs-elastic_agent-default-2024.02.23-000001 | 112 (0.2%) | 0 | `hot` | 100.1KB |
| ✅ | .ds-logs-endpoint.events.network-default-2024.02.23-000001 | 96 (0.2%) | 0 | `hot` | 337.7KB |
| ✅ | .ds-logs-endpoint.events.network-default-2024.02.23-000003 | 93 (0.2%) | 0 | `hot` | 146.4KB |
| ✅ | .ds-logs-endpoint.events.process-default-2024.02.23-000002 | 86 (0.2%) | 0 | `hot` | 308.5KB |
| ✅ | .ds-logs-endpoint.events.security-default-2024.02.23-000002 | 55 (0.1%) | 0 | `hot` | 488.1KB |
| ✅ | .ds-logs-system.security-default-2024.02.23-000002 | 55 (0.1%) | 0 | `hot` | 1MB |
| ✅ | .ds-logs-endpoint.events.library-default-2024.02.23-000002 | 52 (0.1%) | 0 | `hot` | 137KB |
| ✅ | .ds-logs-endpoint.events.security-default-2024.02.23-000001 | 37 (0.1%) | 0 | `hot` | 87.5KB |
| ✅ | .ds-logs-system.system-default-2024.02.23-000002 | 36 (0.1%) | 0 | `hot` | 873.7KB |
| ✅ | .ds-logs-endpoint.alerts-default-2024.02.23-000001 | 22 (0.0%) | 0 | `hot` | 274.7KB |
| ✅ | .ds-logs-endpoint.events.file-default-2024.02.23-000003 | 20 (0.0%) | 0 | `hot` | 149KB |
| ❌ | .ent-search-api-ecs-ilm-logs-production-2024.02.23-000001 | 5 (0.0%) | 1 | `hot` | 76.9KB |
| ❌ | .ds-logs-enterprise_search.audit-default-2024.02.23-000001 | 2 (0.0%) | 1 | `hot` | 16KB |
| ✅ | .ds-logs-system.application-default-2024.02.23-000002 | 2 (0.0%) | 0 | `hot` | 58.5KB |
| ❌ | .ent-search-workplace-search-analytics-ecs-ilm-logs-production-2024.02.23-000001 | 0 (0.0%) | 1 | `hot` | 249B |
| ❌ | .ent-search-app-search-analytics-ecs-ilm-logs-production-2024.02.23-000001 | 0 (0.0%) | 1 | `hot` | 249B |
| ❌ | .ent-search-search-relevance-suggestions-ecs-ilm-logs-production-2024.02.23-000001 | 0 (0.0%) | 1 | `hot` | 249B |
| ✅ | .ds-logs-endpoint.events.file-default-2024.02.23-000002 | 0 (0.0%) | 0 | `hot` | 249B |
| ✅ | .ds-logs-endpoint.events.network-default-2024.02.23-000002 | 0 (0.0%) | 0 | `hot` | 249B |
| ❌ | .ent-search-workplace-search-content-events-ecs-ilm-logs-production-2024.02.23-000001 | 0 (0.0%) | 1 | `hot` | 249B |
| ❌ | .ent-search-crawler-ecs-ilm-logs-production-2024.02.23-000001 | 0 (0.0%) | 1 | `hot` | 249B |
### .ds-logs-elastic_agent.endpoint_security-default-2024.02.23-000002

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ✅ | .ds-logs-elastic_agent.endpoint_security-default-2024.02.23-000002 | 22,040 (39.6%) | 0 | `hot` | 3.8MB |

### **Incompatible fields** `0` **Same family** `2` **Custom fields** `9` **ECS compliant fields** `33` **All fields** `44`

### .ds-logs-system.syslog-default-2024.02.23-000001

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ✅ | .ds-logs-system.syslog-default-2024.02.23-000001 | 9,145 (16.4%) | 0 | `hot` | 1.3MB |

### **Incompatible fields** `0` **Same family** `2` **Custom fields** `12` **ECS compliant fields** `54` **All fields** `68`

### .ds-logs-elastic_agent.endpoint_security-default-2024.02.23-000001

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ✅ | .ds-logs-elastic_agent.endpoint_security-default-2024.02.23-000001 | 3,378 (6.1%) | 0 | `hot` | 866.4KB |

### **Incompatible fields** `0` **Same family** `2` **Custom fields** `9` **ECS compliant fields** `33` **All fields** `44`

### .ds-logs-elastic_agent.filebeat-default-2024.02.23-000002

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ✅ | .ds-logs-elastic_agent.filebeat-default-2024.02.23-000002 | 3,322 (6.0%) | 0 | `hot` | 2.3MB |

### **Incompatible fields** `0` **Same family** `2` **Custom fields** `9` **ECS compliant fields** `33` **All fields** `44`

### .ds-logs-elastic_agent.metricbeat-default-2024.02.23-000002

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ✅ | .ds-logs-elastic_agent.metricbeat-default-2024.02.23-000002 | 3,084 (5.5%) | 0 | `hot` | 2.1MB |

### **Incompatible fields** `0` **Same family** `2` **Custom fields** `9` **ECS compliant fields** `33` **All fields** `44`

### .ds-logs-system.security-default-2024.02.23-000001

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ✅ | .ds-logs-system.security-default-2024.02.23-000001 | 2,854 (5.1%) | 0 | `hot` | 1.4MB |

### **Incompatible fields** `0` **Same family** `2` **Custom fields** `296` **ECS compliant fields** `87` **All fields** `385`

### .ds-logs-elastic_agent-default-2024.02.23-000002

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ✅ | .ds-logs-elastic_agent-default-2024.02.23-000002 | 1,308 (2.3%) | 0 | `hot` | 628.8KB |

### **Incompatible fields** `0` **Same family** `2` **Custom fields** `18` **ECS compliant fields** `39` **All fields** `59`

### .ds-logs-system.system-default-2024.02.23-000001

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ✅ | .ds-logs-system.system-default-2024.02.23-000001 | 1,246 (2.2%) | 0 | `hot` | 583.8KB |

### **Incompatible fields** `0` **Same family** `2` **Custom fields** `193` **ECS compliant fields** `52` **All fields** `247`

### .ds-logs-endpoint.events.process-default-2024.02.23-000003

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ✅ | .ds-logs-endpoint.events.process-default-2024.02.23-000003 | 1,214 (2.2%) | 0 | `hot` | 2.2MB |

### **Incompatible fields** `0` **Same family** `0` **Custom fields** `139` **ECS compliant fields** `307` **All fields** `446`

### .ds-logs-endpoint.events.registry-default-2024.02.23-000002

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ✅ | .ds-logs-endpoint.events.registry-default-2024.02.23-000002 | 1,096 (2.0%) | 0 | `hot` | 732.9KB |

### **Incompatible fields** `0` **Same family** `0` **Custom fields** `34` **ECS compliant fields** `94` **All fields** `128`

### .ds-logs-endpoint.events.network-default-2024.02.23-000004

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ✅ | .ds-logs-endpoint.events.network-default-2024.02.23-000004 | 1,049 (1.9%) | 0 | `hot` | 1.4MB |

### **Incompatible fields** `0` **Same family** `0` **Custom fields** `31` **ECS compliant fields** `134` **All fields** `165`

### .ds-logs-endpoint.events.file-default-2024.02.23-000004

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ✅ | .ds-logs-endpoint.events.file-default-2024.02.23-000004 | 879 (1.6%) | 0 | `hot` | 1.9MB |

### **Incompatible fields** `0` **Same family** `0` **Custom fields** `79` **ECS compliant fields** `126` **All fields** `205`

### .ds-logs-endpoint.events.process-default-2024.02.23-000001

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ✅ | .ds-logs-endpoint.events.process-default-2024.02.23-000001 | 769 (1.4%) | 0 | `hot` | 800.9KB |

### **Incompatible fields** `0` **Same family** `3` **Custom fields** `64` **ECS compliant fields** `134` **All fields** `201`

### .ds-logs-endpoint.events.registry-default-2024.02.23-000001

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ✅ | .ds-logs-endpoint.events.registry-default-2024.02.23-000001 | 631 (1.1%) | 0 | `hot` | 527.1KB |

### **Incompatible fields** `0` **Same family** `2` **Custom fields** `18` **ECS compliant fields** `84` **All fields** `104`

### .ds-logs-system.auth-default-2024.02.23-000001

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ✅ | .ds-logs-system.auth-default-2024.02.23-000001 | 620 (1.1%) | 0 | `hot` | 363.1KB |

### **Incompatible fields** `0` **Same family** `2` **Custom fields** `25` **ECS compliant fields** `74` **All fields** `101`

### .ds-logs-endpoint.events.library-default-2024.02.23-000003

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ✅ | .ds-logs-endpoint.events.library-default-2024.02.23-000003 | 424 (0.8%) | 0 | `hot` | 876.2KB |

### **Incompatible fields** `0` **Same family** `0` **Custom fields** `61` **ECS compliant fields** `125` **All fields** `186`

### .ds-logs-elastic_agent.osquerybeat-default-2024.02.23-000001

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ✅ | .ds-logs-elastic_agent.osquerybeat-default-2024.02.23-000001 | 379 (0.7%) | 0 | `hot` | 1.4MB |

### **Incompatible fields** `0` **Same family** `2` **Custom fields** `9` **ECS compliant fields** `33` **All fields** `44`

### .ds-logs-endpoint.events.library-default-2024.02.23-000001

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ✅ | .ds-logs-endpoint.events.library-default-2024.02.23-000001 | 341 (0.6%) | 0 | `hot` | 432.7KB |

### **Incompatible fields** `0` **Same family** `1` **Custom fields** `34` **ECS compliant fields** `109` **All fields** `144`

### .ds-logs-endpoint.events.file-default-2024.02.23-000001

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ✅ | .ds-logs-endpoint.events.file-default-2024.02.23-000001 | 313 (0.6%) | 0 | `hot` | 606.1KB |

### **Incompatible fields** `0` **Same family** `1` **Custom fields** `48` **ECS compliant fields** `110` **All fields** `159`

### .ds-logs-elastic_agent.filebeat-default-2024.02.23-000001

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ✅ | .ds-logs-elastic_agent.filebeat-default-2024.02.23-000001 | 230 (0.4%) | 0 | `hot` | 632.2KB |

### **Incompatible fields** `0` **Same family** `2` **Custom fields** `9` **ECS compliant fields** `33` **All fields** `44`

### .ds-logs-system.syslog-default-2024.02.23-000002

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ✅ | .ds-logs-system.syslog-default-2024.02.23-000002 | 218 (0.4%) | 0 | `hot` | 215.2KB |

### **Incompatible fields** `0` **Same family** `2` **Custom fields** `15` **ECS compliant fields** `54` **All fields** `71`

### .ds-logs-elastic_agent.metricbeat-default-2024.02.23-000001

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ✅ | .ds-logs-elastic_agent.metricbeat-default-2024.02.23-000001 | 206 (0.4%) | 0 | `hot` | 650.8KB |

### **Incompatible fields** `0` **Same family** `2` **Custom fields** `9` **ECS compliant fields** `33` **All fields** `44`

### .ds-logs-system.application-default-2024.02.23-000001

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ✅ | .ds-logs-system.application-default-2024.02.23-000001 | 159 (0.3%) | 0 | `hot` | 228.6KB |

### **Incompatible fields** `0` **Same family** `2` **Custom fields** `149` **ECS compliant fields** `48` **All fields** `199`

### .ds-logs-system.auth-default-2024.02.23-000002

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ✅ | .ds-logs-system.auth-default-2024.02.23-000002 | 115 (0.2%) | 0 | `hot` | 569.9KB |

### **Incompatible fields** `0` **Same family** `2` **Custom fields** `26` **ECS compliant fields** `74` **All fields** `102`

### .ds-logs-elastic_agent-default-2024.02.23-000001

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ✅ | .ds-logs-elastic_agent-default-2024.02.23-000001 | 112 (0.2%) | 0 | `hot` | 100.1KB |

### **Incompatible fields** `0` **Same family** `2` **Custom fields** `18` **ECS compliant fields** `39` **All fields** `59`

### .ds-logs-endpoint.events.network-default-2024.02.23-000001

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ✅ | .ds-logs-endpoint.events.network-default-2024.02.23-000001 | 96 (0.2%) | 0 | `hot` | 337.7KB |

### **Incompatible fields** `0` **Same family** `3` **Custom fields** `25` **ECS compliant fields** `117` **All fields** `145`

### .ds-logs-endpoint.events.network-default-2024.02.23-000003

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ✅ | .ds-logs-endpoint.events.network-default-2024.02.23-000003 | 93 (0.2%) | 0 | `hot` | 146.4KB |

### **Incompatible fields** `0` **Same family** `3` **Custom fields** `25` **ECS compliant fields** `117` **All fields** `145`

### .ds-logs-endpoint.events.process-default-2024.02.23-000002

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ✅ | .ds-logs-endpoint.events.process-default-2024.02.23-000002 | 86 (0.2%) | 0 | `hot` | 308.5KB |

### **Incompatible fields** `0` **Same family** `3` **Custom fields** `64` **ECS compliant fields** `134` **All fields** `201`

### .ds-logs-system.security-default-2024.02.23-000002

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ✅ | .ds-logs-system.security-default-2024.02.23-000002 | 55 (0.1%) | 0 | `hot` | 1MB |

### **Incompatible fields** `0` **Same family** `2` **Custom fields** `284` **ECS compliant fields** `87` **All fields** `373`

### .ds-logs-endpoint.events.security-default-2024.02.23-000002

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ✅ | .ds-logs-endpoint.events.security-default-2024.02.23-000002 | 55 (0.1%) | 0 | `hot` | 488.1KB |

### **Incompatible fields** `0` **Same family** `0` **Custom fields** `24` **ECS compliant fields** `87` **All fields** `111`

### .ds-logs-endpoint.events.library-default-2024.02.23-000002

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ✅ | .ds-logs-endpoint.events.library-default-2024.02.23-000002 | 52 (0.1%) | 0 | `hot` | 137KB |

### **Incompatible fields** `0` **Same family** `1` **Custom fields** `34` **ECS compliant fields** `109` **All fields** `144`

### .ds-logs-endpoint.events.security-default-2024.02.23-000001

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ✅ | .ds-logs-endpoint.events.security-default-2024.02.23-000001 | 37 (0.1%) | 0 | `hot` | 87.5KB |

### **Incompatible fields** `0` **Same family** `1` **Custom fields** `18` **ECS compliant fields** `78` **All fields** `97`

### .ds-logs-system.system-default-2024.02.23-000002

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ✅ | .ds-logs-system.system-default-2024.02.23-000002 | 36 (0.1%) | 0 | `hot` | 873.7KB |

### **Incompatible fields** `0` **Same family** `2` **Custom fields** `143` **ECS compliant fields** `52` **All fields** `197`

### .ds-logs-endpoint.alerts-default-2024.02.23-000001

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ✅ | .ds-logs-endpoint.alerts-default-2024.02.23-000001 | 22 (0.0%) | 0 | `hot` | 274.7KB |

### **Incompatible fields** `0` **Same family** `12` **Custom fields** `765` **ECS compliant fields** `514` **All fields** `1291`

### .ds-logs-endpoint.events.file-default-2024.02.23-000003

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ✅ | .ds-logs-endpoint.events.file-default-2024.02.23-000003 | 20 (0.0%) | 0 | `hot` | 149KB |

### **Incompatible fields** `0` **Same family** `1` **Custom fields** `48` **ECS compliant fields** `110` **All fields** `159`

### .ent-search-api-ecs-ilm-logs-production-2024.02.23-000001

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ❌ | .ent-search-api-ecs-ilm-logs-production-2024.02.23-000001 | 5 (0.0%) | 1 | `hot` | 76.9KB |

### **Incompatible fields** `1` **Same family** `14` **Custom fields** `63` **ECS compliant fields** `609` **All fields** `687`

#### 1 incompatible field

Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.11.0.

❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported

#### Incompatible field mappings - .ent-search-api-ecs-ilm-logs-production-2024.02.23-000001

| Field | ECS mapping type (expected) | Index mapping type (actual) | 
|-------|-----------------------------|-----------------------------|
| log.origin.file.line | `long` | `integer`  |

### .ds-logs-system.application-default-2024.02.23-000002

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ✅ | .ds-logs-system.application-default-2024.02.23-000002 | 2 (0.0%) | 0 | `hot` | 58.5KB |

### **Incompatible fields** `0` **Same family** `2` **Custom fields** `143` **ECS compliant fields** `48` **All fields** `193`

### .ds-logs-enterprise_search.audit-default-2024.02.23-000001

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ❌ | .ds-logs-enterprise_search.audit-default-2024.02.23-000001 | 2 (0.0%) | 1 | `hot` | 16KB |

### **Incompatible fields** `1` **Same family** `11` **Custom fields** `60` **ECS compliant fields** `613` **All fields** `685`

#### 1 incompatible field

Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.11.0.

❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported

#### Incompatible field mappings - .ds-logs-enterprise_search.audit-default-2024.02.23-000001

| Field | ECS mapping type (expected) | Index mapping type (actual) | 
|-------|-----------------------------|-----------------------------|
| log.origin.file.line | `long` | `integer`  |

### .ent-search-workplace-search-content-events-ecs-ilm-logs-production-2024.02.23-000001

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ❌ | .ent-search-workplace-search-content-events-ecs-ilm-logs-production-2024.02.23-000001 | 0 (0.0%) | 1 | `hot` | 249B |

### **Incompatible fields** `1` **Same family** `11` **Custom fields** `57` **ECS compliant fields** `610` **All fields** `679`

#### 1 incompatible field

Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.11.0.

❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported

#### Incompatible field mappings - .ent-search-workplace-search-content-events-ecs-ilm-logs-production-2024.02.23-000001

| Field | ECS mapping type (expected) | Index mapping type (actual) | 
|-------|-----------------------------|-----------------------------|
| log.origin.file.line | `long` | `integer`  |

### .ent-search-workplace-search-analytics-ecs-ilm-logs-production-2024.02.23-000001

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ❌ | .ent-search-workplace-search-analytics-ecs-ilm-logs-production-2024.02.23-000001 | 0 (0.0%) | 1 | `hot` | 249B |

### **Incompatible fields** `1` **Same family** `11` **Custom fields** `57` **ECS compliant fields** `610` **All fields** `679`

#### 1 incompatible field

Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.11.0.

❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported

#### Incompatible field mappings - .ent-search-workplace-search-analytics-ecs-ilm-logs-production-2024.02.23-000001

| Field | ECS mapping type (expected) | Index mapping type (actual) | 
|-------|-----------------------------|-----------------------------|
| log.origin.file.line | `long` | `integer`  |

### .ent-search-search-relevance-suggestions-ecs-ilm-logs-production-2024.02.23-000001

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ❌ | .ent-search-search-relevance-suggestions-ecs-ilm-logs-production-2024.02.23-000001 | 0 (0.0%) | 1 | `hot` | 249B |

### **Incompatible fields** `1` **Same family** `11` **Custom fields** `57` **ECS compliant fields** `610` **All fields** `679`

#### 1 incompatible field

Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.11.0.

❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported

#### Incompatible field mappings - .ent-search-search-relevance-suggestions-ecs-ilm-logs-production-2024.02.23-000001

| Field | ECS mapping type (expected) | Index mapping type (actual) | 
|-------|-----------------------------|-----------------------------|
| log.origin.file.line | `long` | `integer`  |

### .ent-search-crawler-ecs-ilm-logs-production-2024.02.23-000001

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ❌ | .ent-search-crawler-ecs-ilm-logs-production-2024.02.23-000001 | 0 (0.0%) | 1 | `hot` | 249B |

### **Incompatible fields** `1` **Same family** `11` **Custom fields** `57` **ECS compliant fields** `610` **All fields** `679`

#### 1 incompatible field

Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.11.0.

❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported

#### Incompatible field mappings - .ent-search-crawler-ecs-ilm-logs-production-2024.02.23-000001

| Field | ECS mapping type (expected) | Index mapping type (actual) | 
|-------|-----------------------------|-----------------------------|
| log.origin.file.line | `long` | `integer`  |

### .ent-search-app-search-analytics-ecs-ilm-logs-production-2024.02.23-000001

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ❌ | .ent-search-app-search-analytics-ecs-ilm-logs-production-2024.02.23-000001 | 0 (0.0%) | 1 | `hot` | 249B |

### **Incompatible fields** `1` **Same family** `11` **Custom fields** `57` **ECS compliant fields** `610` **All fields** `679`

#### 1 incompatible field

Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.11.0.

❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported

#### Incompatible field mappings - .ent-search-app-search-analytics-ecs-ilm-logs-production-2024.02.23-000001

| Field | ECS mapping type (expected) | Index mapping type (actual) | 
|-------|-----------------------------|-----------------------------|
| log.origin.file.line | `long` | `integer`  |

### .ds-logs-endpoint.events.network-default-2024.02.23-000002

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ✅ | .ds-logs-endpoint.events.network-default-2024.02.23-000002 | 0 (0.0%) | 0 | `hot` | 249B |

### **Incompatible fields** `0` **Same family** `3` **Custom fields** `25` **ECS compliant fields** `117` **All fields** `145`

### .ds-logs-endpoint.events.file-default-2024.02.23-000002

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ✅ | .ds-logs-endpoint.events.file-default-2024.02.23-000002 | 0 (0.0%) | 0 | `hot` | 249B |

### **Incompatible fields** `0` **Same family** `1` **Custom fields** `48` **ECS compliant fields** `110` **All fields** `159`

Please let us know if anything else is required from our end. Thanks!

semd commented 8 months ago

This is a content length limitation on the Cases side. When trying to add the same markup content in a comment manually in a case, the same error happens.

cases_manual

We'll have to talk with the team to decide how to solve that. @dhru42

Meanwhile, there are different workarounds to this problem:

Create a new Data View with only the problematic index patterns ( .ent-search-* .ds-logs-enterprise_search.* in this case) and re-execute the DQ check using this new data view, to reduce the amount of information.
Use the copy to clipboard to get the entire content and paste the text to a new case manually, removing all the non-relevant information before submitting.
If the previous workarounds did not solve the problem (there may be a lot of indices with failures), we can also use the individual add to case button to create a new case for each index. Or, otherwise, we could also create a new case manually and add the individual copy to clipboard content of the different indices in different comments of this same case.

individual_add_to_case

dhru42 commented 8 months ago

This is a content length limitation on the Cases side. When trying to add the same markup content in a comment manually in a case, the same error happens.

We'll have to talk with the team to decide how to solve that. @dhru42

Meanwhile, there are different workarounds to this problem:

Create a new Data View with only the problematic index patterns ( .ent-search-* .ds-logs-enterprise_search.* in this case) and re-execute the DQ check using this new data view, to reduce the amount of information.

Use the copy to clipboard to get the entire content and paste the text to a new case manually, removing all the non-relevant information before submitting.

If the previous workarounds did not solve the problem (there may be a lot of indices with failures), we can also use the individual add to case button to create a new case for each index. Or, otherwise, we could also create a new case manually and add the individual copy to clipboard content of the different indices in different comments of this same case.

@paulewing @shanisagiv1 please advise on the above issue related to cases length limit - how can we overcome this?

shanisagiv1 commented 8 months ago

cc @cnasikas

cnasikas commented 8 months ago

Hey everyone! Cases have a lot of circuit breakers like the number of characters per comment to ensure the system's stability and performance. I suggest breaking the text into multiple comments (one comment per 30K characters).

dhru42 commented 8 months ago

Hey everyone! Cases have a lot of circuit breakers like the number of characters per comment to ensure the system's stability and performance. I suggest breaking the text into multiple comments (one comment per 30K characters).

@cnasikas hey Christos - in these edge cases, it would make more sense for cases to have workflow added which would break this text (IF this is the only solution) instead of all workflows that escalate to cases that make the change on their end.

cc: @andrew-goldstein @semd @angorayc @shanisagiv1 please let me know if you see any issues with that.

cnasikas commented 8 months ago

Usually, the responsibility lies with consumers of an API and not vice-versa. As our APIs are also used by users, moving this to cases will open a door for them to abuse the system, put it under stress, and lead to timeouts (sending an arbitrary number of characters will make cases create a lot of comments under the scenes). We need to protect the system from these scenarios. I can think of the following solutions to the issue:

Split the text into multiple comments.
Create a condensed summarization of the data with a link for users to navigate back to the full report.
Use cases attachment framework to attach the visualization shown on the top of the page.

elastic / kibana

[Security Solution]Unable to attach Data Quality Dashboard to Cases. #177687