search

opensearch-project / security-analytics

Security Analytics enables users for detecting security threats on their security event log data. It will also allow them to modify/tailor the pre-packaged solution.
Apache License 2.0
72 stars 74 forks source link

[BUG] Issues in threat intel monitor scans #1319

Open eirsep opened 1 month ago

eirsep commented 1 month ago

Sorting by _seq_no field fails on empty index as mapping isn't created yet. This failure is expected and needs to be handled to return empty list instead of throwing exception and calling it a monitor failure

Caused by: org.opensearch.index.query.QueryShardException: No mapping found for [_seq_no] in order to sort on
        at org.opensearch.search.sort.FieldSortBuilder.resolveUnmappedType(FieldSortBuilder.java:564) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.search.sort.FieldSortBuilder.build(FieldSortBuilder.java:411) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.search.sort.SortBuilder.buildSort(SortBuilder.java:168) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.search.SearchService.parseSource(SearchService.java:1268) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.search.SearchService.createContext(SearchService.java:998) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.search.SearchService.executeQueryPhase(SearchService.java:606) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:579) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:74) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:89) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:913) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.lang.Thread.run(Thread.java:1583) ~[?:?]

There is an edge case scenario when there are hits in data but no iocs are actually present to scan. That case is causing creation of a grouped listener of size 0 which is throwing error

[2024-09-15T18:03:08,110][ERROR][o.o.s.t.i.s.IoCScanService] [384634db970ddbb1dc6211644bcfe4fe] Threat intel monitor oulm9JEBop9BFvxFVds2: Unexpected failure in running scan for 246 docs
java.lang.IllegalArgumentException: groupSize must be greater than 0 but was 0
        at org.opensearch.action.support.GroupedActionListener.<init>(GroupedActionListener.java:66)
        at org.opensearch.securityanalytics.threatIntel.iocscan.service.SaIoCScanService.getGroupedListenerForIocScanFromAllIocTypes(SaIoCScanService.java:320)
        at org.opensearch.securityanalytics.threatIntel.iocscan.service.SaIoCScanService.matchAgainstThreatIntelAndReturnMaliciousIocs(SaIoCScanService.java:254)
        at org.opensearch.securityanalytics.threatIntel.iocscan.service.IoCScanService.scanIoCs(IoCScanService.java:85)
        at org.opensearch.securityanalytics.threatIntel.model.monitor.TransportThreatIntelMonitorFanOutAction.lambda$onGetIocTypeToIndices$7(TransportThreatIntelMonitorFanOutAction.java:186)
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82)
        at org.opensearch.securityanalytics.threatIntel.model.monitor.TransportThreatIntelMonitorFanOutAction.lambda$fetchDataFromShards$9(TransportThreatIntelMonitorFanOutAction.java:224)
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82)
        at org.opensearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:81)
        at org.opensearch.securityanalytics.threatIntel.model.monitor.TransportThreatIntelMonitorFanOutAction.fetchLatestDocsFromShard(TransportThreatIntelMonitorFanOutAction.java:261)
        at org.opensearch.securityanalytics.threatIntel.model.monitor.TransportThreatIntelMonitorFanOutAction.lambda$fetchLatestDocsFromShard$11(TransportThreatIntelMonitorFanOutAction.java:291)
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82)
        at org.opensearch.securityanalytics.threatIntel.model.monitor.TransportThreatIntelMonitorFanOutAction.lambda$searchShard$15(TransportThreatIntelMonitorFanOutAction.java:352)
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82)
        at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:115)
        at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:109)
        at org.opensearch.core.action.ActionListener$5.onResponse(ActionListener.java:268)
        at org.opensearch.action.search.AbstractSearchAsyncAction.sendSearchResponse(AbstractSearchAsyncAction.java:769)
        at org.opensearch.action.search.ExpandSearchPhase.run(ExpandSearchPhase.java:132)
        at org.opensearch.action.search.SearchPhase.recordAndRun(SearchPhase.java:61)
        at org.opensearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:491)
        at org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:458)
        at org.opensearch.action.search.FetchSearchPhase.moveToNextPhase(FetchSearchPhase.java:300)
        at org.opensearch.action.search.FetchSearchPhase.lambda$innerRun$1(FetchSearchPhase.java:138)
        at org.opensearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:150)
        at org.opensearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:122)
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
        at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78)
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
        at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59)
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:950)
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)

Index out of bounds exception as wrong listener is used.

[2024-09-15T06:50:29,074][ERROR][o.o.s.t.i.s.SaIoCScanService] [efdc3b06f28dcf50f11616883b4a106e] Threat intel monitor oulm9JEBop9BFvxFVds2 scan with 48720 user data indicators failed for ioc Type ipv4-addr
Failed to execute phase [expand],
        at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:780)
        at org.opensearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:501)
        at org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:458)
        at org.opensearch.action.search.FetchSearchPhase.moveToNextPhase(FetchSearchPhase.java:300)
        at org.opensearch.action.search.FetchSearchPhase.lambda$innerRun$1(FetchSearchPhase.java:138)
        at org.opensearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:158)
        at org.opensearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:122)
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
        at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78)
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
        at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59)
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:950)
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 2 out of bounds for length 1
        at java.base/jdk.internal.util.Preconditions$2.apply(Preconditions.java:63)
        at java.base/jdk.internal.util.Preconditions$2.apply(Preconditions.java:60)
        at java.base/jdk.internal.util.Preconditions$4.apply(Preconditions.java:213)
        at java.base/jdk.internal.util.Preconditions$4.apply(Preconditions.java:210)
        at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:98)
        at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:106)
        at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:302)
        at java.base/java.lang.invoke.VarHandleReferences$Array.compareAndSet(VarHandleReferences.java:655)
        at java.base/java.util.concurrent.atomic.AtomicReferenceArray.compareAndSet(AtomicReferenceArray.java:153)
        at org.opensearch.common.util.concurrent.AtomicArray.setOnce(AtomicArray.java:79)
        at org.opensearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:75)
        at org.opensearch.securityanalytics.threatIntel.iocscan.service.SaIoCScanService.lambda$performScanForMaliciousIocsPerIocType$24(SaIoCScanService.java:369)
        at org.opensearch.core.action.ActionListener$1.onFailure(ActionListener.java:90)
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:84)
        at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:115)
        at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:109)
        at org.opensearch.core.action.ActionListener$5.onResponse(ActionListener.java:268)
        at org.opensearch.action.search.AbstractSearchAsyncAction.sendSearchResponse(AbstractSearchAsyncAction.java:769)
        at org.opensearch.action.search.ExpandSearchPhase.run(ExpandSearchPhase.java:132)
        at org.opensearch.action.search.SearchPhase.recordAndRun(SearchPhase.java:61)
        at org.opensearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:491)
        ... 14 more
dblock commented 3 weeks ago

Was this fixed in #1317? Close?

[Catch All Triage - 1, 2, 3, 4]