search

cmsdaq / DAQExpert

New expert system processing data model produced by DAQAggregator
1 stars 2 forks source link

race condition between DAQExpert and DAQAggregator ? #89

Open andreh12 opened 7 years ago

andreh12 commented 7 years ago

we saw the following in catalina.out of the DAQExpert:

2017-07-09 09:00:08 ERROR StructureSerializer:200 - File incompatible: /daqexpert/snapshots/pro/cdaq/2017/7/9/7/1499583607313.json.gz
java.io.EOFException: Unexpected end of ZLIB input stream
        at java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:240)
        at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
        at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:117)
        at com.fasterxml.jackson.core.json.ByteSourceJsonBootstrapper.ensureLoaded(ByteSourceJsonBootstrapper.java:489)
        at com.fasterxml.jackson.core.json.ByteSourceJsonBootstrapper.detectEncoding(ByteSourceJsonBootstrapper.java:126)
        at com.fasterxml.jackson.core.json.ByteSourceJsonBootstrapper.constructParser(ByteSourceJsonBootstrapper.java:215)
        at com.fasterxml.jackson.core.JsonFactory._createParser(JsonFactory.java:1238)
        at com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:807)
        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2867)
        at rcms.utilities.daqaggregator.persistence.StructureSerializer.deserialize(StructureSerializer.java:197)
        at rcms.utilities.daqaggregator.persistence.StructureSerializer.deserialize(StructureSerializer.java:152)
        at rcms.utilities.daqexpert.processing.ProcessJob.call(ProcessJob.java:79)
        at rcms.utilities.daqexpert.processing.ProcessJob.call(ProcessJob.java:1)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at rcms.utilities.daqexpert.processing.PriorityFuture.run(PriorityFuture.java:47)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
2017-07-09 09:00:08 ERROR ProcessJob:106 - Snapshot not deserialized /daqexpert/snapshots/pro/cdaq/2017/7/9/7/1499583607313.json.gz
2017-07-09 09:00:08 INFO  ProcessJob:120 - This round there was only one snapshot and there was problem with it, aborting..
2017-07-09 09:00:08 INFO  DataPrepareJob:95 - No result this round

Could it be that the DAQExpert discovered the file but the DAQAggregator has not yet closed it ?

(it looks like that there is enough protection against this case so that this issue is very low priority)

gladky commented 6 years ago

DAQAggregator 1.17.4 adds .tmp suffix when writing to file. It renames the file after completion so that described situation of reading unfinished file is avoided. DAQSnapshotService is now ignoring the .tmp files. DAQExpert currently does not use DAQSnapshotService to get snapshots but it accesses the snapshot directory directly. The plan is to use DAQSnapshotService in DAQExpert as well.