mmguero
commented
1 week ago During initialization there are normally various warnings/error messages coming from opensearch while things are starting up and initializing that usually settle out once everything's initialized. Besides the error messages, what is actually not working? Does opensearch die? Do the other containers that use opensearch (dashboards, etc.) report that it is not available? These messages in and of themselves don't constitute a bug.
As far as the "deleting the opensearch pod" results in an error in your last paragraph... I mean, yeah, I would expect that deleting the opensearch pod would cause an error.
alleniverson33
Describe the bug After running opensearch for a period of time, an exception log appears
To Reproduce Steps to reproduce the behavior:
Expected behavior A clear and concise description of what you expected to happen.
Screenshots and/or Logs OpenSearch Security Plugin does not exist, disable by default OpenSearch Performance Analyzer Plugin does not exist, disable by default WARNING: A terminally deprecated method in java.lang.System has been called WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.8.0.jar) WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch WARNING: System::setSecurityManager will be removed in a future release WARNING: A terminally deprecated method in java.lang.System has been called WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.8.0.jar) WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security WARNING: System::setSecurityManager will be removed in a future release [2024-11-05T07:34:30,529][WARN ][o.o.g.DanglingIndicesState] [opensearch-deployment-58cfc9b467-g5f5z] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually [2024-11-05T07:34:32,261][WARN ][o.o.b.BootstrapChecks ] [opensearch-deployment-58cfc9b467-g5f5z] initial heap size [2147483648] not equal to maximum heap size [17179869184]; this can cause resize pauses and prevents memory locking from locking the entire heap [2024-11-05T07:42:11,437][WARN ][o.o.c.m.MetadataIndexTemplateService] [opensearch-deployment-58cfc9b467-g5f5z] index template [malcolm_template] has index patterns [arkime_sessions3-] matching patterns from existing older templates [arkime_sessions3_ecs_template,arkime_sessions3_template] with patterns (arkime_sessions3_ecs_template => [arkime_sessions3-],arkime_sessions3_template => [arkime_sessions3-*]); this template [malcolm_template] will take precedence during new index creation [2024-11-05T07:46:02,724][WARN ][o.o.a.t.RCFResultTransportAction] [opensearch-deployment-58cfc9b467-g5f5z] Anomaly Detector 7O9J-5IBqyhIJ63MfcWI org.opensearch.ad.common.exception.ResourceNotFoundException: No checkpoints found for model id 7O9J-5IBqyhIJ63MfcWI_model_rcf_0 [2024-11-05T07:46:02,725][ERROR][o.o.a.t.AnomalyResultTransportAction] [opensearch-deployment-58cfc9b467-g5f5z] Received an error from node MqBesElaSwyy9126QZoWEQ while doing model inference for 7O9J-5IBqyhIJ63MfcWI org.opensearch.transport.RemoteTransportException: [opensearch-deployment-58cfc9b467-g5f5z][10.32.0.51:9300][cluster:admin/opendistro/adinternal/rcf/result] Caused by: org.opensearch.ad.common.exception.ResourceNotFoundException: No checkpoints found for model id 7O9J-5IBqyhIJ63MfcWI_model_rcf_0 at org.opensearch.ad.ml.ModelManager.processRestoredTRcf(ModelManager.java:302) ~[?:?] at org.opensearch.ad.ml.ModelManager.lambda$getTRcfResult$1(ModelManager.java:185) ~[?:?] at org.opensearch.action.ActionListener$1.onResponse(ActionListener.java:80) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.ad.ml.CheckpointDao.lambda$getTRCFModel$15(CheckpointDao.java:688) [opensearch-anomaly-detection-2.8.0.0.jar:2.8.0.0] at org.opensearch.action.ActionListener$1.onFailure(ActionListener.java:88) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.ad.util.ClientUtil.lambda$asyncRequest$3(ClientUtil.java:128) [opensearch-anomaly-detection-2.8.0.0.jar:2.8.0.0] at org.opensearch.action.ActionListener$1.onFailure(ActionListener.java:88) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.support.TransportAction$1.onFailure(TransportAction.java:122) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:224) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) [opensearch-index-management-2.8.0.0.jar:2.8.0.0] at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.indexmanagement.controlcenter.notification.filter.IndexOperationActionFilter.apply(IndexOperationActionFilter.kt:39) [opensearch-index-management-2.8.0.0.jar:2.8.0.0] at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:476) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.client.support.AbstractClient.get(AbstractClient.java:572) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.ad.util.ClientUtil.asyncRequest(ClientUtil.java:126) [opensearch-anomaly-detection-2.8.0.0.jar:2.8.0.0] at org.opensearch.ad.ml.CheckpointDao.getTRCFModel(CheckpointDao.java:679) [opensearch-anomaly-detection-2.8.0.0.jar:2.8.0.0] at org.opensearch.ad.ml.ModelManager.getTRcfResult(ModelManager.java:181) [opensearch-anomaly-detection-2.8.0.0.jar:2.8.0.0] at org.opensearch.ad.transport.RCFResultTransportAction.doExecute(RCFResultTransportAction.java:77) [opensearch-anomaly-detection-2.8.0.0.jar:2.8.0.0] at org.opensearch.ad.transport.RCFResultTransportAction.doExecute(RCFResultTransportAction.java:36) [opensearch-anomaly-detection-2.8.0.0.jar:2.8.0.0] at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) [opensearch-index-management-2.8.0.0.jar:2.8.0.0] at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.indexmanagement.controlcenter.notification.filter.IndexOperationActionFilter.apply(IndexOperationActionFilter.kt:39) [opensearch-index-management-2.8.0.0.jar:2.8.0.0] at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:102) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:98) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:113) [opensearch-index-management-2.8.0.0.jar:2.8.0.0] at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:106) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.transport.TransportService.sendLocalRequest(TransportService.java:1058) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.transport.TransportService$3.sendRequest(TransportService.java:152) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:996) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.transport.TransportService.sendRequest(TransportService.java:883) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.transport.TransportService.sendRequest(TransportService.java:826) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.ad.transport.AnomalyResultTransportAction.lambda$onFeatureResponseForSingleEntityDetector$10(AnomalyResultTransportAction.java:604) [opensearch-anomaly-detection-2.8.0.0.jar:2.8.0.0] at org.opensearch.action.ActionListener$1.onResponse(ActionListener.java:80) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.ad.feature.FeatureManager.updateUnprocessedFeatures(FeatureManager.java:219) [opensearch-anomaly-detection-2.8.0.0.jar:2.8.0.0] at org.opensearch.ad.feature.FeatureManager.lambda$getCurrentFeatures$1(FeatureManager.java:165) [opensearch-anomaly-detection-2.8.0.0.jar:2.8.0.0] at org.opensearch.action.ActionListener$1.onResponse(ActionListener.java:80) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.ad.feature.SearchFeatureDao.lambda$getFeatureSamplesForPeriods$14(SearchFeatureDao.java:606) [opensearch-anomaly-detection-2.8.0.0.jar:2.8.0.0] at org.opensearch.action.ActionListener$1.onResponse(ActionListener.java:80) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.ActionListener$6.onResponse(ActionListener.java:299) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:113) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:107) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.search.TransportSearchAction.lambda$executeRequest$0(TransportSearchAction.java:399) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.ActionListener$1.onResponse(ActionListener.java:80) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.ActionListener$5.onResponse(ActionListener.java:266) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.search.AbstractSearchAsyncAction.sendSearchResponse(AbstractSearchAsyncAction.java:658) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.search.ExpandSearchPhase.run(ExpandSearchPhase.java:132) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:427) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:421) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.search.FetchSearchPhase.moveToNextPhase(FetchSearchPhase.java:299) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.search.FetchSearchPhase.lambda$innerRun$1(FetchSearchPhase.java:139) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:151) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:123) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.8.0.jar:2.8.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?] at java.lang.Thread.run(Thread.java:833) [?:?] [2024-11-05T07:47:02,808][ERROR][o.o.a.t.ADTaskManager ] [opensearch-deployment-58cfc9b467-g5f5z] Failed to update realtime task for detector 7O9J-5IBqyhIJ63MfcWI org.opensearch.ad.common.exception.ResourceNotFoundException: can't find latest task at org.opensearch.ad.task.ADTaskManager.lambda$updateLatestADTask$80(ADTaskManager.java:1976) [opensearch-anomaly-detection-2.8.0.0.jar:2.8.0.0] at org.opensearch.ad.task.ADTaskManager.lambda$getAndExecuteOnLatestADTask$21(ADTaskManager.java:943) [opensearch-anomaly-detection-2.8.0.0.jar:2.8.0.0] at org.opensearch.ad.task.ADTaskManager.lambda$getAndExecuteOnLatestADTasks$22(ADTaskManager.java:1016) [opensearch-anomaly-detection-2.8.0.0.jar:2.8.0.0] at org.opensearch.action.ActionListener$1.onResponse(ActionListener.java:80) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:113) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:107) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.search.TransportSearchAction.lambda$executeRequest$0(TransportSearchAction.java:399) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.ActionListener$1.onResponse(ActionListener.java:80) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.ActionListener$5.onResponse(ActionListener.java:266) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.search.AbstractSearchAsyncAction.sendSearchResponse(AbstractSearchAsyncAction.java:658) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.search.ExpandSearchPhase.run(ExpandSearchPhase.java:132) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:427) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:421) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.search.FetchSearchPhase.moveToNextPhase(FetchSearchPhase.java:299) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.search.FetchSearchPhase.lambda$innerRun$1(FetchSearchPhase.java:139) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:151) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:123) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.8.0.jar:2.8.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?] at java.lang.Thread.run(Thread.java:833) [?:?] [2024-11-05T07:47:02,810][ERROR][o.o.a.ExecuteADResultResponseRecorder] [opensearch-deployment-58cfc9b467-g5f5z] Can't find latest realtime task of detector 7O9J-5IBqyhIJ63MfcWI [2024-11-05T08:48:46,676][WARN ][o.o.m.f.FsHealthService ] [opensearch-deployment-58cfc9b467-g5f5z] health check of [/usr/share/opensearch/data/nodes/0] took [10403ms] which is above the warn threshold of [5s] [2024-11-05T19:34:38,287][ERROR][o.o.a.a.AlertIndices ] [opensearch-deployment-58cfc9b467-g5f5z] info deleteOldIndices [2024-11-05T19:34:38,287][ERROR][o.o.a.a.AlertIndices ] [opensearch-deployment-58cfc9b467-g5f5z] info deleteOldIndices [2024-11-05T19:34:38,314][ERROR][o.o.s.i.DetectorIndexManagementService] [opensearch-deployment-58cfc9b467-g5f5z] info deleteOldIndices [2024-11-05T19:34:38,314][ERROR][o.o.s.i.DetectorIndexManagementService] [opensearch-deployment-58cfc9b467-g5f5z] info deleteOldIndices
Malcolm Version:
How are you running Malcolm? k8s
Additional context
There is another issue in k8s where manually deleting the opensearch pod results in the opensearch startup report org.onsearch.action.search SearchPhaseExecutionException: all shards failed