search

islandora-interest-groups / Islandora-DevOps-Interest-Group

The purpose of the Islandora DevOps Interest Group is to make it easier for endusers, developers, testers, and system administrators to understand, deploy, tune and secure the Islandora stack (and associated tools) in their respective hosting environments.
14 stars 17 forks source link

Ingesting large files into Fedora #47

Open ctgraham opened 6 years ago

ctgraham commented 6 years ago

https://groups.google.com/forum/#!topic/islandora-dev/MiB8PAD3fuA

When ingesting a 14GB MP4 into Fedora via islandora_batch via drush, we receive the following Fedora error message indicating the "uploaded://" file "no longer exists":

ERROR 2018-02-13 16:10:40.868 [http-bio-8080-exec-1220] (BaseRestResource) Unexpected error fulfilling REST API request
org.fcrepo.server.errors.StreamIOException: [DatastreamManagedContent] returned  the error: "org.fcrepo.server.errors.StreamIOException". Reason: Uploaded file uploaded://7885022 no longer exists.
        at org.fcrepo.server.storage.types.DatastreamManagedContent.getContentStream(DatastreamManagedContent.java:187) ~[fcrepo-server-3.8.1.jar:na]
        at org.fcrepo.server.storage.DefaultDOManager.populateDC(DefaultDOManager.java:1090) ~[fcrepo-server-3.8.1.jar:na]
        at org.fcrepo.server.storage.DefaultDOManager.getIngestWriter(DefaultDOManager.java:1011) ~[fcrepo-server-3.8.1.jar:na]
        at org.fcrepo.server.management.DefaultManagement.ingest(DefaultManagement.java:156) ~[fcrepo-server-3.8.1.jar:na]
        at sun.reflect.GeneratedMethodAccessor249.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_60]
        at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_60]
        at org.fcrepo.server.messaging.NotificationInvocationHandler.invoke(NotificationInvocationHandler.java:71) ~[fcrepo-server-3.8.1.jar:na]
        at com.sun.proxy.$Proxy13.ingest(Unknown Source) ~[na:na]
        at org.fcrepo.server.management.ManagementModule.ingest(ManagementModule.java:385) ~[fcrepo-server-3.8.1.jar:na]
        at org.fcrepo.server.rest.FedoraObjectsResource.createObject(FedoraObjectsResource.java:531) [fcrepo-server-3.8.1.jar:na]
        at org.fcrepo.server.rest.FedoraObjectsResource.newObject(FedoraObjectsResource.java:443) [fcrepo-server-3.8.1.jar:na]
        at sun.reflect.GeneratedMethodAccessor248.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_60]
        at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_60]
        at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:180) [cxf-api-2.7.7.jar:2.7.7]
        at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96) [cxf-api-2.7.7.jar:2.7.7]
        at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:192) [cxf-rt-frontend-jaxrs-2.7.7.jar:2.7.7]
        at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:100) [cxf-rt-frontend-jaxrs-2.7.7.jar:2.7.7]
        at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:57) [cxf-api-2.7.7.jar:2.7.7]
        at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:93) [cxf-api-2.7.7.jar:2.7.7]
        at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:272) [cxf-api-2.7.7.jar:2.7.7]
        at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) [cxf-api-2.7.7.jar:2.7.7]
        at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:239) [cxf-rt-transports-http-2.7.7.jar:2.7.7]
        at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:248) [cxf-rt-transports-http-2.7.7.jar:2.7.7]
        at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:222) [cxf-rt-transports-http-2.7.7.jar:2.7.7]
        at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:153) [cxf-rt-transports-http-2.7.7.jar:2.7.7]
        at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:167) [cxf-rt-transports-http-2.7.7.jar:2.7.7]
        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:286) [cxf-rt-transports-http-2.7.7.jar:2.7.7]
        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:206) [cxf-rt-transports-http-2.7.7.jar:2.7.7]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:650) [tomcat-servlet-3.0-api.jar:na]
        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:262) [cxf-rt-transports-http-2.7.7.jar:2.7.7]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) [catalina.jar:7.0.82]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.82]
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat7-websocket.jar:7.0.82]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [catalina.jar:7.0.82]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.82]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:369) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]
        at org.fcrepo.server.security.jaas.AuthFilterJAAS.doFilter(AuthFilterJAAS.java:329) [fcrepo-security-jaas-3.8.1.jar:na]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]
        at org.springframework.security.web.access.channel.ChannelProcessingFilter.doFilter(ChannelProcessingFilter.java:109) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]
        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:168) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) [spring-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) [spring-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [catalina.jar:7.0.82]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.82]
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219) [catalina.jar:7.0.82]
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110) [catalina.jar:7.0.82]
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:506) [catalina.jar:7.0.82]
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169) [catalina.jar:7.0.82]
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) [catalina.jar:7.0.82]
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962) [catalina.jar:7.0.82]
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) [catalina.jar:7.0.82]
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445) [catalina.jar:7.0.82]
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1115) [tomcat-coyote.jar:7.0.82]
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637) [tomcat-coyote.jar:7.0.82]
        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:318) [tomcat-coyote.jar:7.0.82]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_60]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_60]
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-coyote.jar:7.0.82]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
Caused by: org.fcrepo.server.errors.StreamIOException: Uploaded file uploaded://7885022 no longer exists.
        at org.fcrepo.server.storage.types.DatastreamManagedContent.getContentStream(DatastreamManagedContent.java:145) ~[fcrepo-server-3.8.1.jar:na]
        ... 61 common frames omitted
WARN 2018-02-13 16:10:40.905 [http-bio-8080-exec-1220] (WebApplicationExceptionMapper) javax.ws.rs.WebApplicationException: org.fcrepo.server.errors.StreamIOException: [DatastreamManagedContent] returned  the error: "org.fcrepo.server.errors.StreamIOException". Reason: Uploaded file uploaded://7885022 no longer exists.
        at org.fcrepo.server.rest.BaseRestResource.handleException(BaseRestResource.java:203)
        at org.fcrepo.server.rest.BaseRestResource.handleException(BaseRestResource.java:208)
        at org.fcrepo.server.rest.FedoraObjectsResource.createObject(FedoraObjectsResource.java:536)
        at org.fcrepo.server.rest.FedoraObjectsResource.newObject(FedoraObjectsResource.java:443)
        at sun.reflect.GeneratedMethodAccessor248.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:180)
        at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
        at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:192)
        at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:100)
        at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:57)
        at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:93)
        at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:272)
        at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
        at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:239)
        at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:248)
        at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:222)
        at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:153)
        at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:167)
        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:286)
        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:206)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:262)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:369)
        at org.fcrepo.server.security.jaas.AuthFilterJAAS.doFilter(AuthFilterJAAS.java:329)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381)
        at org.springframework.security.web.access.channel.ChannelProcessingFilter.doFilter(ChannelProcessingFilter.java:109)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381)
        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:168)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:506)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1115)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:318)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.fcrepo.server.errors.StreamIOException: [DatastreamManagedContent] returned  the error: "org.fcrepo.server.errors.StreamIOException". Reason: Uploaded file uploaded://7885022 no longer exists.
        at org.fcrepo.server.storage.types.DatastreamManagedContent.getContentStream(DatastreamManagedContent.java:187)
        at org.fcrepo.server.storage.DefaultDOManager.populateDC(DefaultDOManager.java:1090)
        at org.fcrepo.server.storage.DefaultDOManager.getIngestWriter(DefaultDOManager.java:1011)
        at org.fcrepo.server.management.DefaultManagement.ingest(DefaultManagement.java:156)
        at sun.reflect.GeneratedMethodAccessor249.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.fcrepo.server.messaging.NotificationInvocationHandler.invoke(NotificationInvocationHandler.java:71)
        at com.sun.proxy.$Proxy13.ingest(Unknown Source)
        at org.fcrepo.server.management.ManagementModule.ingest(ManagementModule.java:385)
        at org.fcrepo.server.rest.FedoraObjectsResource.createObject(FedoraObjectsResource.java:531)
        ... 51 more
Caused by: org.fcrepo.server.errors.StreamIOException: Uploaded file uploaded://7885022 no longer exists.
        at org.fcrepo.server.storage.types.DatastreamManagedContent.getContentStream(DatastreamManagedContent.java:145)
       ... 61 more

Drush reports:

WD islandora: Failed to ingest object: pitt:7198634code: 500message: [error]
Exception occured: Failed to ingest pitt:7198634.                    [ok]

But no detail on the 500 error is readily available in drush output or the watchdog or the Batch Queue.

Where else should we look for additional information to debug this? Are there timeouts or filesize limitations we should be considering?

ctgraham commented 6 years ago

Peter Lux has a good suggestion on the uploadStoageMinutes setting in the googlegroups thread.