search

archiver-appliance / epicsarchiverap

This is an implementation of an archiver for EPICS control systems that aims to archive millions of PVs.
Other
40 stars 39 forks source link

Exception when fetching pv that contains no data #71

Open NickeZ opened 5 years ago

NickeZ commented 5 years ago

Hey,

I get the following exception in retrieval_catalina.err when the PV does not yet contain any data.

Exception in thread "http-bio-17668-exec-201" java.util.zip.ZipError: zip END header not found
        at com.sun.nio.zipfs.ZipFileSystem.zerror(ZipFileSystem.java:1605)
        at com.sun.nio.zipfs.ZipFileSystem.findEND(ZipFileSystem.java:1021)
        at com.sun.nio.zipfs.ZipFileSystem.initCEN(ZipFileSystem.java:1030)
        at com.sun.nio.zipfs.ZipFileSystem.<init>(ZipFileSystem.java:130)
        at com.sun.nio.zipfs.ZipFileSystemProvider.newFileSystem(ZipFileSystemProvider.java:139)
        at org.epics.archiverappliance.utils.nio.ArchPaths.get(ArchPaths.java:141)
        at org.epics.archiverappliance.utils.nio.ArchPaths.get(ArchPaths.java:76)
        at edu.stanford.slac.archiverappliance.PlainPB.PlainPBPathNameUtility.getParentPath(PlainPBPathNameUtility.java:447)
        at edu.stanford.slac.archiverappliance.PlainPB.PlainPBPathNameUtility.getDirectoryStreamsForPV(PlainPBPathNameUtility.java:488)
        at edu.stanford.slac.archiverappliance.PlainPB.PlainPBPathNameUtility.getAllPathsForPV(PlainPBPathNameUtility.java:273)
        at edu.stanford.slac.archiverappliance.PlainPB.PlainPBStoragePlugin.getFirstKnownEvent(PlainPBStoragePlugin.java:701)
        at org.epics.archiverappliance.retrieval.RetrievalState.getDataSources(RetrievalState.java:88)
        at org.epics.archiverappliance.retrieval.DataSourceResolution.resolveDataSources(DataSourceResolution.java:90)
        at org.epics.archiverappliance.retrieval.DataRetrievalServlet.resolveAllDataSources(DataRetrievalServlet.java:1155)
        at org.epics.archiverappliance.retrieval.DataRetrievalServlet.doGetSinglePV(DataRetrievalServlet.java:447)
        at org.epics.archiverappliance.retrieval.DataRetrievalServlet.doGet(DataRetrievalServlet.java:142)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:624)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:506)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)

It looks something like this in the table:

PV_NAME | Appliance assigned | N/A | N/A | N/A | N/A | N/A | N/A | N/A
NickeZ commented 5 years ago

Solved it with

systemctl stop archiver
find /home/archiver/ -size 0 -delete
systemctl start archiver

And now it looks better in the table:


PV_NAME | Being archived | appliance0 | true | true | 1.0 | Dez/05/2018 12:59:32 +01:00

But why is AA creating empty zipfiles?

slacmshankar commented 5 years ago

Are you by any chance using compress=true? http://slacmshankar.github.io/epicsarchiver_docs/api/edu/stanford/slac/archiverappliance/PlainPB/PlainPBStoragePlugin.html

The support for compression is not great; it's been experimental for quite some time and probably ought to be removed. The .zip file format does not lend itself to this kind of storage easily. I would not use compression; the possibility of data loss is quite real in this case.

We should redo the support for compression at some point in time.

NickeZ commented 5 years ago

Yes we use compression. Without it we need several orders of magnitude more disk space which isn't feasible..

slacmshankar commented 5 years ago

You'd be the first team to try it out in production. Good luck. Did you check out ZFS compression or equivalent? This may have better results and will probably get you significantly better performance.

NickeZ commented 5 years ago

When you put it that way I might reconsider.. we have a very small system with only a single ioc and a few thousand pvs. What kind of failures should I look out for?

slacmshankar commented 5 years ago

What kind of failures should I look out for?

Not sure; we made the mistake of using the .zip format for compression with the idea that this would give us the ability to compress and store multiple files in the same zip file. However, the .zip file format is not suited for append workloads; which is the main kind of workload we have. When we add content to the .zip file, the existing file is copied and a new file with the newly added content is generated. There are multiple potential points of failure in this process; not to mention the terrible performance.

And since compression is quite readily available at the file system level; this feature has not be pursued as much. I have not had the time to refactor the code to pull this feature out; I'll probably do this soon.