search

IHTSDO / snowstorm

Scalable SNOMED CT Terminology Server using Elasticsearch
Other
209 stars 83 forks source link

Bulk indexing has failures - Elasticsearch exception "FORBIDDEN/12/index read-only" #67

Closed StephanMeijer closed 5 years ago

StephanMeijer commented 5 years ago 
2019-08-19 11:17:47.142  INFO 1 --- [ool-11-thread-5] i.k.elasticvc.api.ComponentService       : Saving batch of 29 Concepts
2019-08-19 11:17:47.180 ERROR 1 --- [ool-11-thread-5] o.ihtsdo.otf.snomedboot.ReleaseImporter  : Failed to read or process lines.

org.springframework.data.elasticsearch.ElasticsearchException: Bulk indexing has failures. Use ElasticsearchException.getFailedDocuments() for detailed messages [{jRSYqWwBBvYxDaVU7Gkr=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blo
cked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], kRSYqWwBBvYxDaVU7Gkr=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], iRSYqWwBBvYxDaVU7Gkr=Elasticsearc
hException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], hRSYqWwBBvYxDaVU7Gkr=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12
/index read-only / allow delete (api)];]], fRSYqWwBBvYxDaVU7Gkr=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], gRSYqWwBBvYxDaVU7Gkr=ElasticsearchException[Elasticsear
ch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], mRSYqWwBBvYxDaVU7Gkr=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / all
ow delete (api)];]], jBSYqWwBBvYxDaVU7Gkr=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], lRSYqWwBBvYxDaVU7Gkr=ElasticsearchException[Elasticsearch exception [type=clu
ster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], iBSYqWwBBvYxDaVU7Gkr=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], h
BSYqWwBBvYxDaVU7Gkr=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], lBSYqWwBBvYxDaVU7Gkr=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception,
reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], gBSYqWwBBvYxDaVU7Gkr=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], kBSYqWwBBvYxDaVU7Gkr=El
asticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], mBSYqWwBBvYxDaVU7Gkr=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FO
RBIDDEN/12/index read-only / allow delete (api)];]], khSYqWwBBvYxDaVU7Gkr=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], jhSYqWwBBvYxDaVU7Gkr=ElasticsearchException[E
lasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], ihSYqWwBBvYxDaVU7Gkr=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-
only / allow delete (api)];]], lhSYqWwBBvYxDaVU7Gkr=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], ghSYqWwBBvYxDaVU7Gkr=ElasticsearchException[Elasticsearch exception
 [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], lxSYqWwBBvYxDaVU7Gkr=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (a
pi)];]], hhSYqWwBBvYxDaVU7Gkr=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], kxSYqWwBBvYxDaVU7Gkr=ElasticsearchException[Elasticsearch exception [type=cluster_block_e
xception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], fhSYqWwBBvYxDaVU7Gkr=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], hxSYqWwBBvYxD
aVU7Gkr=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], ixSYqWwBBvYxDaVU7Gkr=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=block
ed by: [FORBIDDEN/12/index read-only / allow delete (api)];]], gxSYqWwBBvYxDaVU7Gkr=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], jxSYqWwBBvYxDaVU7Gkr=ElasticsearchE
xception[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], fxSYqWwBBvYxDaVU7Gkr=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/i
ndex read-only / allow delete (api)];]]}]
        at org.springframework.data.elasticsearch.core.ElasticsearchTemplate.checkForBulkUpdateFailure(ElasticsearchTemplate.java:681)
        at org.springframework.data.elasticsearch.core.ElasticsearchTemplate.bulkIndex(ElasticsearchTemplate.java:662)
        at org.springframework.data.elasticsearch.repository.support.AbstractElasticsearchRepository.saveAll(AbstractElasticsearchRepository.java:184)
        at sun.reflect.GeneratedMethodAccessor339.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments.invoke(RepositoryComposition.java:377)
        at org.springframework.data.repository.core.support.RepositoryComposition.invoke(RepositoryComposition.java:200)
        at org.springframework.data.repository.core.support.RepositoryFactorySupport$ImplementationMethodExecutionInterceptor.invoke(RepositoryFactorySupport.java:629)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
        at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.doInvoke(RepositoryFactorySupport.java:593)
        at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.invoke(RepositoryFactorySupport.java:578)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
        at org.springframework.data.projection.DefaultMethodInvokingMethodInterceptor.invoke(DefaultMethodInvokingMethodInterceptor.java:59)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
        at org.springframework.data.repository.core.support.SurroundingTransactionDetectorMethodInterceptor.invoke(SurroundingTransactionDetectorMethodInterceptor.java:61)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
        at com.sun.proxy.$Proxy99.saveAll(Unknown Source)
        at io.kaicode.elasticvc.api.ComponentService.doSaveBatchComponents(ComponentService.java:81)
        at org.snomed.snowstorm.core.data.services.ConceptUpdateHelper.doSaveBatchConcepts(ConceptUpdateHelper.java:368)
        at org.snomed.snowstorm.core.rf2.rf2import.ImportComponentFactoryImpl$1.persistCollection(ImportComponentFactoryImpl.java:72)
        at org.snomed.snowstorm.core.rf2.rf2import.ImportComponentFactoryImpl$PersistBuffer.flush(ImportComponentFactoryImpl.java:275)
        at java.util.ArrayList.forEach(ArrayList.java:1257)
        at org.snomed.snowstorm.core.rf2.rf2import.ImportComponentFactoryImpl$4.persistCollection(ImportComponentFactoryImpl.java:106)
        at org.snomed.snowstorm.core.rf2.rf2import.ImportComponentFactoryImpl$PersistBuffer.flush(ImportComponentFactoryImpl.java:275)
        at org.snomed.snowstorm.core.rf2.rf2import.ImportComponentFactoryImpl$PersistBuffer.save(ImportComponentFactoryImpl.java:270)
        at org.snomed.snowstorm.core.rf2.rf2import.ImportComponentFactoryImpl.newReferenceSetMemberState(ImportComponentFactoryImpl.java:240)
        at org.ihtsdo.otf.snomedboot.ReleaseImporter$ImportRun.lambda$loadRefsets$3(ReleaseImporter.java:451)
        at org.ihtsdo.otf.snomedboot.ReleaseImporter$ImportRun.readLines(ReleaseImporter.java:530)
        at org.ihtsdo.otf.snomedboot.ReleaseImporter$ImportRun.lambda$readLinesCallable$4(ReleaseImporter.java:470)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

2019-08-19 11:17:47.184  INFO 1 --- [pool-2-thread-6] i.k.elasticvc.api.ComponentService       : Saving batch of 29 Concepts
2019-08-19 11:17:47.193 ERROR 1 --- [pool-2-thread-6] o.s.s.core.rf2.rf2import.ImportService   : Failed RF2 DELTA import on branch MAIN/PFER3. ID e3d6a8f6-9ae5-4d03-8710-ed9c15db97d2

rg.springframework.data.elasticsearch.ElasticsearchException: Bulk indexing has failures. Use ElasticsearchException.getFailedDocuments() for detailed messages [{oBSYqWwBBvYxDaVU7Gk4=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blo
cked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], qxSYqWwBBvYxDaVU7Gk4=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], pBSYqWwBBvYxDaVU7Gk4=Elasticsearc
hException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], rxSYqWwBBvYxDaVU7Gk4=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12
/index read-only / allow delete (api)];]], qBSYqWwBBvYxDaVU7Gk4=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], pxSYqWwBBvYxDaVU7Gk4=ElasticsearchException[Elasticsear
ch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], oxSYqWwBBvYxDaVU7Gk4=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / all
ow delete (api)];]], sxSYqWwBBvYxDaVU7Gk4=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], oRSYqWwBBvYxDaVU7Gk4=ElasticsearchException[Elasticsearch exception [type=clu
ster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], tBSYqWwBBvYxDaVU7Gk4=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], r
hSYqWwBBvYxDaVU7Gk4=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], shSYqWwBBvYxDaVU7Gk4=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception,
reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], nRSYqWwBBvYxDaVU7Gk4=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], thSYqWwBBvYxDaVU7Gk4=El
asticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], pRSYqWwBBvYxDaVU7Gk4=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FO
RBIDDEN/12/index read-only / allow delete (api)];]], rBSYqWwBBvYxDaVU7Gk4=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], sBSYqWwBBvYxDaVU7Gk4=ElasticsearchException[E
lasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], qRSYqWwBBvYxDaVU7Gk4=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-
only / allow delete (api)];]], nBSYqWwBBvYxDaVU7Gk4=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], nhSYqWwBBvYxDaVU7Gk4=ElasticsearchException[Elasticsearch exception
 [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], tRSYqWwBBvYxDaVU7Gk4=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (a
pi)];]], ohSYqWwBBvYxDaVU7Gk4=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], rRSYqWwBBvYxDaVU7Gk4=ElasticsearchException[Elasticsearch exception [type=cluster_block_e
xception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], sRSYqWwBBvYxDaVU7Gk4=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], qhSYqWwBBvYxD
aVU7Gk4=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], phSYqWwBBvYxDaVU7Gk4=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=block
ed by: [FORBIDDEN/12/index read-only / allow delete (api)];]], mhSYqWwBBvYxDaVU7Gk4=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], nxSYqWwBBvYxDaVU7Gk4=ElasticsearchE
xception[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]], mxSYqWwBBvYxDaVU7Gk4=ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/i
ndex read-only / allow delete (api)];]]}]
        at org.springframework.data.elasticsearch.core.ElasticsearchTemplate.checkForBulkUpdateFailure(ElasticsearchTemplate.java:681)
        at org.springframework.data.elasticsearch.core.ElasticsearchTemplate.bulkIndex(ElasticsearchTemplate.java:662)
        at org.springframework.data.elasticsearch.repository.support.AbstractElasticsearchRepository.saveAll(AbstractElasticsearchRepository.java:184)
        at sun.reflect.GeneratedMethodAccessor339.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments.invoke(RepositoryComposition.java:377)
        at org.springframework.data.repository.core.support.RepositoryComposition.invoke(RepositoryComposition.java:200)
        at org.springframework.data.repository.core.support.RepositoryFactorySupport$ImplementationMethodExecutionInterceptor.invoke(RepositoryFactorySupport.java:629)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
        at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.doInvoke(RepositoryFactorySupport.java:593)
        at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.invoke(RepositoryFactorySupport.java:578)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
        at org.springframework.data.projection.DefaultMethodInvokingMethodInterceptor.invoke(DefaultMethodInvokingMethodInterceptor.java:59)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
        at org.springframework.data.repository.core.support.SurroundingTransactionDetectorMethodInterceptor.invoke(SurroundingTransactionDetectorMethodInterceptor.java:61)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
        at com.sun.proxy.$Proxy99.saveAll(Unknown Source)
        at io.kaicode.elasticvc.api.ComponentService.doSaveBatchComponents(ComponentService.java:81)
        at org.snomed.snowstorm.core.data.services.ConceptUpdateHelper.doSaveBatchConcepts(ConceptUpdateHelper.java:368)
        at org.snomed.snowstorm.core.rf2.rf2import.ImportComponentFactoryImpl$1.persistCollection(ImportComponentFactoryImpl.java:72)
        at org.snomed.snowstorm.core.rf2.rf2import.ImportComponentFactoryImpl$PersistBuffer.flush(ImportComponentFactoryImpl.java:275)
        at java.util.ArrayList.forEach(ArrayList.java:1257)
        at org.snomed.snowstorm.core.rf2.rf2import.ImportComponentFactoryImpl.completeImportCommit(ImportComponentFactoryImpl.java:180)
        at org.snomed.snowstorm.core.rf2.rf2import.ImportComponentFactoryImpl.loadingComponentsCompleted(ImportComponentFactoryImpl.java:176)
        at org.ihtsdo.otf.snomedboot.ReleaseImporter$ImportRun.doLoadReleaseFiles(ReleaseImporter.java:202)
        at org.ihtsdo.otf.snomedboot.ReleaseImporter$ImportRun.doLoadReleaseFiles(ReleaseImporter.java:159)
        at org.ihtsdo.otf.snomedboot.ReleaseImporter$ImportRun.access$100(ReleaseImporter.java:145)
        at org.ihtsdo.otf.snomedboot.ReleaseImporter.loadDeltaReleaseFiles(ReleaseImporter.java:51)
        at org.ihtsdo.otf.snomedboot.ReleaseImporter.loadDeltaReleaseFiles(ReleaseImporter.java:85)
        at org.snomed.snowstorm.core.rf2.rf2import.ImportService.importArchive(ImportService.java:101)
        at org.snomed.snowstorm.core.rf2.rf2import.ImportService.lambda$importArchiveAsync$1(ImportService.java:145)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
kaicode commented 5 years ago

Hi @StephanMeijer,

It looks like your Elasticsearch indexes are in a read-only state. Elasticsearch does this when disk space is low - something like below 85%. https://www.elastic.co/guide/en/elasticsearch/reference/6.4/disk-allocator.html

You can check the status of your indices using the Elasticsearch cat API. This gives you an indication of the health of your indices. They will be yellow if you are low on disk: https://www.elastic.co/guide/en/elasticsearch/reference/6.5/cat-indices.html

If you think this could be the problem please change the disk allocation or give more disk space.

I hope this helps!

Kind regards, Kai

StephanMeijer commented 5 years ago

As I suspect it might be indeed disk space, I will close this issue.