psyllid sometimes not repsonsive after switching files

[cclaessens]

We have observed that during triggering data taking psyllid sometimes stops taking data and ends up in a non-responsive state.

So far all we know about it is that this seems to happen at the transition from one file to another. Both times this occurs the last file written was full (1GB) and the next file was still empty.

hornet logs for run 5980

1c 02:13:17.729 INFO [Mover] ▶ File copy took 990.38178ms to transfer 1007602328 bytes 51d 02:13:17.850 INFO [finishFile] ▶ Completed work on file 51e 02:13:46.791 NOTI [summaryLoop] ▶ Scheduler summary: In the past 1m0s,

• Scheduled 1 file(s)
• Finished 1 file(s) 51f 02:14:15.906 INFO [Scheduler] ▶ Sending to the classifier 520 02:14:15.907 INFO [Classifier] ▶ Classifying file as type 521 02:14:15.907 INFO [Scheduler] ▶ Sending to the mover 522 02:14:16.898 INFO [Mover] ▶ File copy took 990.752612ms to transfer 1007686760 bytes 523 02:14:17.024 INFO [finishFile] ▶ Completed work on file 524 02:14:46.792 NOTI [summaryLoop] ▶ Scheduler summary: In the past 1m0s,
• Scheduled 1 file(s)
• Finished 1 file(s) 525 02:15:24.053 INFO [Scheduler] ▶ Sending to the classifier 526 02:15:24.053 INFO [Classifier] ▶ Classifying file as type 527 02:15:24.053 INFO [Scheduler] ▶ Sending to the mover 528 02:15:24.053 INFO [Mover] ▶ File copy took 186.971µs to transfer 0 bytes 529 02:15:24.053 INFO [finishFile] ▶ Completed work on file 52a 02:15:46.792 NOTI [summaryLoop] ▶ Scheduler summary: In the past 1m0s,
• Scheduled 1 file(s)
• Finished 1 file(s) 52b 02:21:24.261 NOTI [func1] ▶ Added subdirectory to watch [/data/hot/000yyyxxx/000005xxx/000005981]

and from run 6062

7330 20:59:06.766 INFO [Mover] ▶ File copy took 980.864339ms to transfer 1007605664 bytes 7331 20:59:06.882 INFO [finishFile] ▶ Completed work on file 7332 20:59:48.548 NOTI [summaryLoop] ▶ Scheduler summary: In the past 1m0s,

• Scheduled 1 file(s)
• Finished 1 file(s) 7333 21:00:08.518 NOTI [func1] ▶ Added subdirectory to watch [/data/hot/000yyyxxx/000006xxx/000006063] 7334 21:00:08.919 INFO [Scheduler] ▶ Sending to the classifier 7335 21:00:08.920 INFO [Classifier] ▶ Classifying file as type 7336 21:00:08.920 INFO [Scheduler] ▶ Sending to the mover 7337 21:00:09.884 INFO [Mover] ▶ File copy took 963.925266ms to transfer 1007616480 bytes 7338 21:00:10.022 INFO [finishFile] ▶ Completed work on file 7339 21:00:48.548 NOTI [summaryLoop] ▶ Scheduler summary: In the past 1m0s,
• Scheduled 1 file(s)
• Finished 1 file(s) 733a 21:01:11.024 INFO [Scheduler] ▶ Sending to the classifier 733b 21:01:11.024 INFO [Classifier] ▶ Classifying file as type 733c 21:01:11.024 INFO [Scheduler] ▶ Sending to the mover 733d 21:01:11.024 INFO [Mover] ▶ File copy took 227.428µs to transfer 0 bytes 733e 21:01:11.024 INFO [finishFile] ▶ Completed work on file

[cclaessens]

Happened again on August 27th 2018. sterr reads:

ESC[1;31m2018-08-27 11:20:02 [ERROR] _receiver_fpa.cc(246): Exiting due to stream errorESC[0m /var/log/supervisor/psyllid_ch_b-stderr---supervisor-rG6nS8.log (END)