search

project8 / hornet

Hornet is a nearline data processor for the Project 8 experiment
Other
0 stars 0 forks source link

Workers can stick in loop while exiting #20

Open nsoblath opened 9 years ago

nsoblath commented 9 years ago

Sometimes this happens, sometimes it works normally. I think it has to do with the order in which things shutdown.

2015/06/12 14:38:03 [scheduler] stopping on interrupt
2015/06/12 14:38:03 [scheduler] finished.
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 0.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 1.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 2.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 3.0] Finished processing jobs for this file
2015/06/12 14:38:03 [worker 4.0] Finished processing jobs for this file
2015/06/12 14:38:03 [classifier] Adding job:
    {proc-egg egg echo "here's an egg file: {{.Filename}}" 0xc20802db40}
2015/06/12 14:38:03 [classifier] type <egg> will now perform job 0 <proc-egg>: [0]
2015/06/12 14:38:03 [classifier] Adding job:
    {proc-rsa-mat rsa-mat echo "here's an RSA MAT file {{.Filename}}" 0xc208082380}
2015/06/12 14:38:03 [classifier] type <rsa-mat> will now perform job 1 <proc-rsa-mat>: [1]
2015/06/12 14:38:03 Base paths: [/home/nsoblath/go/src/github.com/project8/hornet/build-develop /home/nsoblath/go/src/github.com/project8/hornet]
2015/06/12 14:38:04 [hornet] Timed out waiting for goroutines to finish.