project8 / hornet

Hornet is a nearline data processor for the Project 8 experiment
Other
0 stars 0 forks source link

Slack notification - server information #41

Closed wcpettus closed 6 years ago

wcpettus commented 7 years ago

When a slack notification is sent, we've made hornet sufficiently identical that you can't tell which computer it's running on.

As an example, recently some p8_alerts were spammed:

hornetAPP [12:59 PM] 
ERROR > Error while hashing: exit status 1

CRITICAL > [classifier] file </data/hot/000yyyxxx/000002xxx/000002866/rid000002866.msk> does not exist

ERROR > File copy failed! (/data/hot/000yyyxxx/000002xxx/000002866/rid000002866_meta.json -> /data/warm/000yyyxxx/000002xxx/000002866/rid000002866_meta.json) [open /data/hot/000yyyxxx/000002xxx/000002866/rid000002866_meta.json: no such file or directory]

ERROR > Error copying (/data/hot/000yyyxxx/000002xxx/000002866/rid000002866_meta.json -> /data/warm/000yyyxxx/000002xxx/000002866/rid000002866_meta.json) [Failed to copy file]

But you have to check both ignatius and zeppelin to tell which.

This problem will go away when we only run one DAQ, but that may take a while(?)

nsoblath commented 6 years ago

This is already done