search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
1.22k stars 24.85k forks source link

Elasticsearch service keep crashing #116748

Open moazam1 opened 2 days ago

moazam1 commented 2 days ago

Elasticsearch Version

8.15.4

Installed Plugins

No response

Java Version

1.8.0_432

OS Version

Linux 5.14.0-427.28.1.el9_4.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Aug 2 03:44:10 EDT 2024 x86_64 x86_64 x86_64 GNU/Linux

Problem Description

Almost every month Elasticsearch service is crashing and not restarting automatically. I think routine upgrade might be causing crash. Any helpe will be greatly appreciated.

Steps to Reproduce

I suspect routine upgrade might be causing this issue.

Logs (if relevant)

[2024-11-13T01:30:00,001][INFO ][o.e.x.m.MlDailyMaintenanceService] riggering scheduled [ML] maintenance tasks [2024-11-13T01:30:00,005][INFO ][o.e.x.m.a.TransportDeleteExpiredDataAction] Deleting expired data [2024-11-13T01:30:00,007][INFO ][o.e.x.m.j.r.UnusedStatsRemover] Successfully deleted [0] unused stats documents [2024-11-13T01:30:00,007][INFO ][o.e.x.m.a.TransportDeleteExpiredDataAction] Completed deletion of expired ML data [2024-11-13T01:30:00,008][INFO ][o.e.x.m.MlDailyMaintenanceService] Successfully completed [ML] maintenance task: triggerDeleteExpiredDataTask [2024-11-13T01:40:02,603][INFO ][o.e.c.m.MetadataMappingService] [sss_applied_late_lien_fees/F0MzKyNnS1arYCqRnbgBzw] update_mapping [_doc] [2024-11-13T03:20:30,001][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] fatal error in thread [elasticsearch [masterService#updateTask][T#6560]], exiting java.util.ServiceConfigurationError: Error loading SPI class list from URL: jar:file:///usr/share/elasticsearch/modules/x-pack-security/x-pack-security-8.15.3.jar!/META-INF/services/org.elasticsearch.index.SlowLogFieldProvider at org.elasticsearch.plugins.spi.SPIClassIterator.loadNextProfile(SPIClassIterator.java:136) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.plugins.spi.SPIClassIterator.hasNext(SPIClassIterator.java:148) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.plugins.PluginsService.createExtensions(PluginsService.java:379) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.plugins.PluginsService.loadServiceProviders(PluginsService.java:334) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.indices.IndicesService.loadSlowLogFieldProvider(IndicesService.java:1430) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.indices.IndicesService.createIndexService(IndicesService.java:749) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.indices.IndicesService.withTempIndexService(IndicesService.java:702) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexWithTemporaryService(MetadataCreateIndexService.java:476) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequestWithV2Template(MetadataCreateIndexService.java:669) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:397) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.action.admin.indices.rollover.MetadataRolloverService.rolloverDataStream(MetadataRolloverService.java:406) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.action.admin.indices.rollover.MetadataRolloverService.rolloverClusterState(MetadataRolloverService.java:163) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.action.admin.indices.rollover.TransportRolloverAction$RolloverExecutor.executeTask(TransportRolloverAction.java:540) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.action.admin.indices.rollover.TransportRolloverAction$RolloverExecutor.execute(TransportRolloverAction.java:460) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:1070) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:1033) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:233) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.lambda$run$2(MasterService.java:1686) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.action.ActionListener.run(ActionListener.java:444) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.run(MasterService.java:1683) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.cluster.service.MasterService$5.lambda$doRun$0(MasterService.java:1278) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.action.ActionListener.run(ActionListener.java:444) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.cluster.service.MasterService$5.doRun(MasterService.java:1257) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.15.3.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] at java.lang.Thread.run(Thread.java:1570) ~[?:?] Caused by: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/modules/x-pack-security/x-pack-security-8.15.3.jar at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?] at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?] at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?] at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:56) ~[?:?] at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:171) ~[?:?] at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) ~[?:?] at java.nio.file.Files.readAttributes(Files.java:1858) ~[?:?] at java.util.zip.ZipFile$Source.get(ZipFile.java:1491) ~[?:?] at java.util.zip.ZipFile$CleanableResource.(ZipFile.java:724) ~[?:?] at java.util.zip.ZipFile.(ZipFile.java:251) ~[?:?] at java.util.zip.ZipFile.(ZipFile.java:180) ~[?:?] at java.util.jar.JarFile.(JarFile.java:345) ~[?:?] at sun.net.www.protocol.jar.URLJarFile.(URLJarFile.java:100) ~[?:?] at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:69) ~[?:?] at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:168) ~[?:?] at sun.net.www.protocol.jar.JarFileFactory.getOrCreate(JarFileFactory.java:91) ~[?:?] at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:117) ~[?:?] at sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:160) ~[?:?] at org.elasticsearch.plugins.spi.SPIClassIterator.loadNextProfile(SPIClassIterator.java:112) ~[elasticsearch-8.15.3.jar:?] ... 27 more

SunnyDay6 commented 1 day ago

I confirm this issue occurs with newer version aswell.

a. Package updated at 00h00:46 b. Service failed à 03:09:09 c. Assumption: I think devs were deploying a nightly package, and elasticsearch was solicitated which crashed the service. d. journalctl doesn't show that the service was restart with upgrade, only at failed point at 03:09:09 e. In /etc/sysconfig/elasticsearch , this is commented #RESTART_ON_UPGRADE=true, ( will test without comment and post back ) f. A manual restart of the service works as expected. g. xpack is set to false in elasticsearch.yml

On Rocky Linux 9.4 Linux 5.14.0-427.42.1.el9_4.x86_64

[AUSER@SERVERNAME ~]# systemctl status elasticsearch × elasticsearch.service - Elasticsearch Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; preset: disabled) Active: failed (Result: exit-code) since Thu 2024-11-14 03:09:09 EST; 5h 38min ago Duration: 1w 4d 9h 38min 31.988s Docs: https://www.elastic.co Process: 1281 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE) Main PID: 1281 (code=exited, status=1/FAILURE) CPU: 1h 20min 28.979s

from dnf.log

2024-11-14T00:00:03-0500 DEBUG ---> Package elasticsearch.x86_64 8.15.3-1 will be upgraded 2024-11-14T00:00:03-0500 DEBUG ---> Package elasticsearch.x86_64 8.16.0-1 will be an upgrade 2024-11-14T00:00:46-0500 DEBUG Upgraded: elasticsearch-8.16.0-1.x86_64

from elasticsearch.log

java.util.ServiceConfigurationError: Error loading SPI class list from URL: jar:file:///usr/share/elasticsearch/modules/x-pack-security/x-pack-security-8.15.3.jar!/META-INF/services/org.elasticsearch.index.SlowLogFieldProvider at org.elasticsearch.plugins.spi.SPIClassIterator.loadNextProfile(SPIClassIterator.java:136) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.plugins.spi.SPIClassIterator.hasNext(SPIClassIterator.java:148) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.plugins.PluginsService.createExtensions(PluginsService.java:379) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.plugins.PluginsService.loadServiceProviders(PluginsService.java:334) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.indices.IndicesService.loadSlowLogFieldProvider(IndicesService.java:1430) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.indices.IndicesService.createIndexService(IndicesService.java:749) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.indices.IndicesService.withTempIndexService(IndicesService.java:702) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexWithTemporaryService(MetadataCreateIndexService.java:476) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequestWithV1Templates(MetadataCreateIndexService.java:600) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:424) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:442) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.action.admin.indices.create.AutoCreateAction$TransportAction$CreateIndexTask.execute(AutoCreateAction.java:335) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.action.admin.indices.create.AutoCreateAction$TransportAction.lambda$new$0(AutoCreateAction.java:120) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:1070) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:1033) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:233) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.lambda$run$2(MasterService.java:1686) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.action.ActionListener.run(ActionListener.java:444) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.run(MasterService.java:1683) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.cluster.service.MasterService$5.lambda$doRun$0(MasterService.java:1278) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.action.ActionListener.run(ActionListener.java:444) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.cluster.service.MasterService$5.doRun(MasterService.java:1257) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.15.3.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] at java.lang.Thread.run(Thread.java:1570) ~[?:?] Caused by: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/modules/x-pack-security/x-pack-security-8.15.3.jar at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?] at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?] at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?] at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:56) ~[?:?] at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:171) ~[?:?] at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) ~[?:?] at java.nio.file.Files.readAttributes(Files.java:1858) ~[?:?] at java.util.zip.ZipFile$Source.get(ZipFile.java:1491) ~[?:?] at java.util.zip.ZipFile$CleanableResource.(ZipFile.java:724) ~[?:?] at java.util.zip.ZipFile.(ZipFile.java:251) ~[?:?] at java.util.zip.ZipFile.(ZipFile.java:180) ~[?:?] at java.util.jar.JarFile.(JarFile.java:345) ~[?:?] at sun.net.www.protocol.jar.URLJarFile.(URLJarFile.java:100) ~[?:?] at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:69) ~[?:?]