search

keeps / dbptk-enterprise

DBPTK Enterprise deployment using docker
https://database-preservation.com
GNU Lesser General Public License v3.0
4 stars 2 forks source link

DBPTK Enterprise Crash #13

Open Laurira opened 3 years ago

Laurira commented 3 years ago

I installed DBPTK Enterprise on Ubuntu. Successfully loaded one siard and used it. Then tried to load another siard (that worked nicely on DBPTK Desktop) then in the middle of loading there were some error message and after that DBPTK Enterprise is useless. image

I am not sure where to find logs via Docker so I used command: sudo docker logs -t --since 10m a4e098dcf0ac (I think these should be last 10 minute logs)

There are something like: 2021-02-02T08:12:23.088931335Z org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://XXX.YY.ZZ.ZZ:8983/solr/dbv-database-dd08a065-9af7-4ee0-b859-e894ad4500ba: Server error writing document id 23676501 to the index 2021-02-02T08:12:23.088938208Z at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:665) 2021-02-02T08:12:23.088944490Z at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:265) 2021-02-02T08:12:23.088950702Z at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248) 2021-02-02T08:12:23.088956673Z at org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:368) 2021-02-02T08:12:23.088962674Z at org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:296) 2021-02-02T08:12:23.088968495Z at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1143) 2021-02-02T08:12:23.088974477Z at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:906) 2021-02-02T08:12:23.088980748Z at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:838) 2021-02-02T08:12:23.088986730Z at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211) 2021-02-02T08:12:23.088992701Z at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:177) 2021-02-02T08:12:23.088998462Z at com.databasepreservation.common.server.index.DatabaseRowsSolrManager.insertDocument(DatabaseRowsSolrManager.java:380) 2021-02-02T08:12:23.089017257Z at com.databasepreservation.common.server.index.DatabaseRowsSolrManager.addRow(DatabaseRowsSolrManager.java:153) 2021-02-02T08:12:23.089023549Z at com.databasepreservation.modules.viewer.DbvtkExportModule.handleDataRow(DbvtkExportModule.java:127) 2021-02-02T08:12:23.089029640Z at com.databasepreservation.model.modules.filters.IdentityFilter.handleDataRow(IdentityFilter.java:88) 2021-02-02T08:12:23.089035652Z at com.databasepreservation.model.modules.filters.ObservableFilter.handleDataRow(ObservableFilter.java:123) 2021-02-02T08:12:23.089041583Z at com.databasepreservation.modules.siard.in.content.SIARD2ContentImportStrategy.endElement(SIARD2ContentImportStrategy.java:374) 2021-02-02T08:12:23.089048736Z at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:609) 2021-02-02T08:12:23.089054858Z at com.sun.org.apache.xerces.internal.impl.xs.XMLSchemaValidator.endElement(XMLSchemaValidator.java:859) 2021-02-02T08:12:23.089060789Z at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1782) 2021-02-02T08:12:23.089066790Z at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2967) 2021-02-02T08:12:23.089072832Z at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:602) 2021-02-02T08:12:23.089078703Z at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:112) 2021-02-02T08:12:23.089084704Z at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:505) 2021-02-02T08:12:23.089090705Z at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:842) 2021-02-02T08:12:23.089096486Z at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:771) 2021-02-02T08:12:23.089102427Z at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141) 2021-02-02T08:12:23.089108208Z at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213) 2021-02-02T08:12:23.089114009Z at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643) 2021-02-02T08:12:23.089119920Z at com.databasepreservation.modules.siard.in.content.SIARD2ContentImportStrategy.importContent(SIARD2ContentImportStrategy.java:184) 2021-02-02T08:12:23.089125981Z at com.databasepreservation.modules.siard.in.input.SIARDImportDefault.migrateDatabaseTo(SIARDImportDefault.java:64) 2021-02-02T08:12:23.089131853Z at com.databasepreservation.DatabaseMigration.migrate(DatabaseMigration.java:123) 2021-02-02T08:12:23.089137663Z at com.databasepreservation.common.server.controller.SIARDController.convertSIARDtoSolr(SIARDController.java:697) 2021-02-02T08:12:23.089143615Z at com.databasepreservation.common.server.controller.SIARDController.loadFromLocal(SIARDController.java:660) 2021-02-02T08:12:23.089155186Z at com.databasepreservation.common.api.v1.CollectionResource.createCollection(CollectionResource.java:188) 2021-02-02T08:12:23.089161338Z at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 2021-02-02T08:12:23.089167780Z at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 2021-02-02T08:12:23.089173751Z at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 2021-02-02T08:12:23.089179572Z at java.lang.reflect.Method.invoke(Method.java:498) 2021-02-02T08:12:23.089185293Z at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) 2021-02-02T08:12:23.089191405Z at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) 2021-02-02T08:12:23.089197376Z at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) 2021-02-02T08:12:23.089203347Z at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) 2021-02-02T08:12:23.089209408Z at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) 2021-02-02T08:12:23.089215440Z at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469) 2021-02-02T08:12:23.089221491Z at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391) 2021-02-02T08:12:23.089227412Z at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80) 2021-02-02T08:12:23.089233253Z at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253) 2021-02-02T08:12:23.089239154Z at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) 2021-02-02T08:12:23.089244875Z at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) 2021-02-02T08:12:23.089250676Z at org.glassfish.jersey.internal.Errors.process(Errors.java:292) 2021-02-02T08:12:23.089256427Z at org.glassfish.jersey.internal.Errors.process(Errors.java:274) 2021-02-02T08:12:23.089262138Z at org.glassfish.jersey.internal.Errors.process(Errors.java:244) 2021-02-02T08:12:23.089267858Z at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) 2021-02-02T08:12:23.089273649Z at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232) 2021-02-02T08:12:23.089279360Z at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680) 2021-02-02T08:12:23.089285141Z at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:392) 2021-02-02T08:12:23.089290811Z at org.glassfish.jersey.servlet.ServletContainer.serviceImpl(ServletContainer.java:385) 2021-02-02T08:12:23.089297484Z at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:560) 2021-02-02T08:12:23.089309066Z at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:501) 2021-02-02T08:12:23.089315087Z at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:438) 2021-02-02T08:12:23.089320848Z at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) 2021-02-02T08:12:23.089326829Z at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) 2021-02-02T08:12:23.089332630Z at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) 2021-02-02T08:12:23.089338591Z at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) 2021-02-02T08:12:23.089344392Z at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) 2021-02-02T08:12:23.089350233Z at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) 2021-02-02T08:12:23.089356084Z at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:114) 2021-02-02T08:12:23.089362035Z at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:104) 2021-02-02T08:12:23.089367937Z at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) 2021-02-02T08:12:23.089373757Z at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) 2021-02-02T08:12:23.089379598Z at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

So if you want to understand in more detail what happened then you should tell me where to get you more information.

Tomorrow I will clean up this mess and try again, so all logs will be vanished in 24 hours.

Laurira commented 3 years ago

I examined this issue further and seems like DBPTK Enterprise is quite chatty... it did create >300GB log file so that it run out of disk space. I am not sure that this application needs to log so much information, but when it is ok, then close this issue.

Laurira commented 3 years ago

And maybe only fix for this issue could be that you create proper error message to the user that "There is no space left on disk"

Laurira commented 3 years ago

Second attempt: image

Now I am sure there is no disk space issue. But error in log is related to heap space solr_1 | Caused by: java.lang.OutOfMemoryError: Java heap space

Laurira commented 3 years ago

My docker gets maximum amount of memory so it can not be docker related RAM issue: image

But I do not know how much memory is assigned to Java inside the docker.

hmiguim commented 3 years ago

Can you post your docker-compose file or sent it to me to check the available options?

Laurira commented 3 years ago

The same problem occurred. This is the docker-compose file `version: "3.2" services: dbptke: image: keeps/dbvtk:latest ports:

Laurira commented 3 years ago

I tried with another bigger database and got the same error.

solr_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781) solr_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917) solr_1 | at java.base/java.lang.Thread.run(Unknown Source) solr_1 | Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed solr_1 | at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:681) solr_1 | at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:695) solr_1 | at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1591) solr_1 | at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1586) solr_1 | at org.apache.solr.update.DirectUpdateHandler2.updateDocOrDocValues(DirectUpdateHandler2.java:977) solr_1 | at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:345) solr_1 | at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:292) solr_1 | at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:239) solr_1 | ... 82 more solr_1 | Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=1856833 actual=347a8e6 (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/var/solr/data/dbv-database-3d71ac32-908f-444d-be82-7fc7dfd607a2_shard1_replica_n1/data/index/_2i4.cfs") [slice=_2i4.fdt])) solr_1 | at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) solr_1 | at org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:526) solr_1 | at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.checkIntegrity(CompressingStoredFieldsReader.java:664) solr_1 | at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.merge(CompressingStoredFieldsWriter.java:574) solr_1 | at org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:229) solr_1 | at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:106) solr_1 | at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4463) solr_1 | at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4057) solr_1 | at org.apache.solr.update.SolrIndexWriter.merge(SolrIndexWriter.java:201) solr_1 | at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625) solr_1 | at org.apache.lucen... dbptke_1 | 2021-08-09 07:36:02.682 ERROR 1 --- [nio-8080-exec-2] o.a.s.c.solrj.impl.BaseCloudSolrClient : Request to collection [dbv-database-3d71ac32-908f-444d-be82-7fc7dfd607a2] failed due to (500) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://172.18.0.3:8983/solr/dbv-database-3d71ac32-908f-444d-be82-7fc7dfd607a2: Server error writing document id 3381177 to the index, retry=0 commError=false errorCode=500 dbptke_1 | 2021-08-09 07:36:02.683 WARN 1 --- [nio-8080-exec-2] c.d.c.s.index.DatabaseRowsSolrManager : Could not insert a document batch in collectiondbv-database-3d71ac32-908f-444d-be82-7fc7dfd607a2. Last response (if any): null dbptke_1 | dbptke_1 | org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://172.18.0.3:8983/solr/dbv-database-3d71ac32-908f-444d-be82-7fc7dfd607a2: Server error writing document id 3381177 to the index dbptke_1 | at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:665) dbptke_1 | at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:265) dbptke_1 | at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248) dbptke_1 | at org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:368) dbptke_1 | at org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:296) dbptke_1 | at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1143) dbptke_1 | at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:906) dbptke_1 | at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:838) dbptke_1 | at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211) dbptke_1 | at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:177) dbptke_1 | at com.databasepreservation.common.server.index.DatabaseRowsSolrManager.insertDocument(DatabaseRowsSolrManager.java:387) dbptke_1 | at com.databasepreservation.common.server.index.DatabaseRowsSolrManager.addRow(DatabaseRowsSolrManager.java:160) dbptke_1 | at com.databasepreservation.modules.viewer.DbvtkExportModule.handleDataRow(DbvtkExportModule.java:134) dbptke_1 | at com.databasepreservation.model.modules.filters.IdentityFilter.handleDataRow(IdentityFilter.java:88) dbptke_1 | at com.databasepreservation.model.modules.filters.ObservableFilter.handleDataRow(ObservableFilter.java:123) dbptke_1 | at com.databasepreservation.modules.siard.in.content.SIARD2ContentImportStrategy.endElement(SIARD2ContentImportStrategy.java:374) dbptke_1 | at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source) dbptke_1 | at org.apache.xerces.impl.xs.XMLSchemaValidator.endElement(Unknown Source) dbptke_1 | at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanEndElement(Unknown Source) dbptke_1 | at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source) dbptke_1 | at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source) dbptke_1 | at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) dbptke_1 | at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)