search

beast-dev / beast-mcmc

Bayesian Evolutionary Analysis Sampling Trees
http://beast.community
GNU Lesser General Public License v2.1
192 stars 73 forks source link

Missing states in log file #1054

Closed rhagenson closed 5 years ago

rhagenson commented 5 years ago

Discovered upon running Tracer v1.7.1 on the .log output of BEAST v1.8.4 as installed via conda 4.6.8, state 43366000 of 200000000 was missing.

What other details do you need from me to trace why this state might have been dropped from the .log output?

rambaut commented 5 years ago

Is it just that state missing? Is the whole line missing? Is the equivalent tree also missing?

rambaut commented 5 years ago

As a work around for this case, I suggest you just duplicate the line before.

rhagenson commented 5 years ago

Thank you for the workaround. I did just that in order to get my results read.

To answer your questions:

Is it just that state missing?

Yes, it is only that state missing from the .log output. All other states are present.

Is the whole line missing?

Yes, the whole line is missing.

Is the equivalent tree also missing?

Yes, the tree is also missing.

rambaut commented 5 years ago

Is the drive the file is being written to on a network or in the machine running BEAST? Is it close to being full?

rhagenson commented 5 years ago

In the machine running BEAST, and it is not close to being full now, but the disk did run out of space at one time due to a concurrent job that was writing to the same disk. Migrated the other job to write to a larger disk. I captured STDOUT and STDERR for the BEAST job. STDOUT has the state in it, STDERR has no output at all, let alone that the state might have been dropped due to write errors.

rambaut commented 5 years ago

It does sounds like a write error due to a full drive. At the moment these will be silent but it may be a good idea for BEAST to tell you there may be corruption in the log file. It would be handy to have a tool to clean these up afterwards.

rhagenson commented 5 years ago

Sounds good. I think logging an error to state a write error occurred would be helpful. Currently, I have no way of knowing if the full drive is why BEAST failed to output that state, but I think it is fair to say it did not help.

rambaut commented 5 years ago

The fact you had a full disk during the run makes me 99% sure that is what the issue was. BEAST doesn't generally drop states unless there is a disk issue or a network issue (if the drive is mounted on the network).

rhagenson commented 5 years ago

Sounds good. Thank you for your help!