thl unable to read disk log

Following a failed switch, it became clear that the thl could not read the log. 
The log files themselves seemed OK, but trying to read with the thl resulted in 
the following error. The replicator was running at the time. 

This might have happened because the replicator was up and running and there is 
disk log logic that wants to write to the log. In general we need to make the 
thl operate in 'read-only' mode for most operations because it can also mess up 
the log if the replicator happens to be down at the time the thl runs. 

Connecting to storage2011-04-05 23:07:47,325 INFO replicator.thl.DiskLog Using 
directory '/opt/continuent/logs/' for replicator logs 
2011-04-05 23:07:47,325 INFO replicator.thl.DiskLog Checksums enabled for log 
records: true 
2011-04-05 23:07:47,329 INFO replicator.thl.DiskLog Unable to acquire write 
lock; log is read-only 
2011-04-05 23:07:47,334 INFO replicator.thl.DiskLog Loaded event serializer 
class: 
com.continuent.tungsten.enterprise.replicator.thl.serializer.ProtobufSerializer 
2011-04-05 23:07:47,383 INFO replicator.thl.LogIndex Building file index on log 
directory: /opt/continuent/logs 
2011-04-05 23:08:06,836 INFO replicator.thl.LogIndex Constructed index; total 
log files added=2245 
2011-04-05 23:08:06,836 INFO replicator.thl.DiskLog Validating last log file: 
/opt/continuent/logs/thl.data.0000002245 
2011-04-05 23:08:07,699 INFO replicator.thl.DiskLog Last log file ends on 
rotate log event: thl.data.0000002245 
2011-04-05 23:08:08,158 WARN replicator.thl.LogFile Unexpected I/O exception 
while closing log file: name=thl.data.0000002245 exception=sync failed 
com.continuent.tungsten.replicator.thl.THLException: New log file exists 
already: thl.data.0000002245 
at 
com.continuent.tungsten.enterprise.replicator.thl.DiskLog.startNewLogFile(DiskLo
g.java:940) 
at 
com.continuent.tungsten.enterprise.replicator.thl.DiskLog.prepare(DiskLog.java:3
10) 
at 
com.continuent.tungsten.enterprise.replicator.thl.DiskTHLStorage.prepare(DiskTHL
Storage.java:113) 
at 
com.continuent.tungsten.enterprise.replicator.thl.EnterpriseTHLManagerCtrl.conne
ct(EnterpriseTHLManagerCtrl.java:104) 
at 
com.continuent.tungsten.enterprise.replicator.thl.EnterpriseTHLManagerCtrl.main(
EnterpriseTHLManagerCtrl.java:356) 
min seq# = 0 
max seq# = 9223372036854775807 
events = -9223372036854775808 
highest known replicated seq# = -1 
Fatal error: null 
java.lang.NullPointerException 
at 
com.continuent.tungsten.enterprise.replicator.thl.DiskLog.release(DiskLog.java:3
99) 
at 
com.continuent.tungsten.enterprise.replicator.thl.DiskTHLStorage.release(DiskTHL
Storage.java:132) 
at 
com.continuent.tungsten.enterprise.replicator.thl.EnterpriseTHLManagerCtrl.disco
nnect(EnterpriseTHLManagerCtrl.java:135) 
at 
com.continuent.tungsten.enterprise.replicator.thl.EnterpriseTHLManagerCtrl.main(
EnterpriseTHLManagerCtrl.java:365) 
QA.M3 tungsten@qa[log]$ thl info 
Connecting to storage2011-04-05 23:45:06,443 INFO replicator.thl.DiskLog Using 
directory '/opt/continuent/logs/' for replicator logs 
2011-04-05 23:45:06,443 INFO replicator.thl.DiskLog Checksums enabled for log 
records: true 
2011-04-05 23:45:06,775 INFO replicator.thl.DiskLog Unable to acquire write 
lock; log is read-only 
2011-04-05 23:45:06,899 INFO replicator.thl.DiskLog Loaded event serializer 
class: 
com.continuent.tungsten.enterprise.replicator.thl.serializer.ProtobufSerializer 
2011-04-05 23:45:06,957 INFO replicator.thl.LogIndex Building file index on log 
directory: /opt/continuent/logs 
2011-04-05 23:45:16,936 INFO replicator.thl.LogIndex Constructed index; total 
log files added=2268 
2011-04-05 23:45:16,936 INFO replicator.thl.DiskLog Validating last log file: 
/opt/continuent/logs/thl.data.0000002268 
2011-04-05 23:45:17,991 INFO replicator.thl.DiskLog Idle log connection 
timeout: 28800000ms 
2011-04-05 23:45:17,991 INFO replicator.thl.DiskLog Log preparation is complete 
2011-04-05 23:45:17,991 INFO replicator.thl.DiskTHLStorage Adapter preparation 
is complete 
min seq# = 0 
max seq# = 147543383 
events = 147543384
Original issue reported on code.google.com by stephane...@gtempaccount.com on 18 Apr 2011 at 2:00
jakemctigue / tungsten-replicator

thl unable to read disk log #19
