search

Stanford-ChEMH-MCAC / protocols

Laboratory protocols used by the Metabolite Chemistry Analysis Center, a ChEM-H Knowledge Center at Stanford University.
BSD 3-Clause "New" or "Revised" License
1 stars 4 forks source link

email on worklist failure #9

Open wdwvt1 opened 6 years ago

wdwvt1 commented 6 years ago

I wonder if we can setup a system that monitors worklist failure and sends an email to the 6545 mailing list or some other status checking system. Right now, Shuo has been having to monitor the system at a high frequency.

The acquisition logbook is in some binary format that is unreadable by less or zless/zcat etc. If it were readable, we could just have a cron job monitoring the logbook looking in the EventType column and look for the text 'Error'. We'd have to keep the terminal with cron running always on in the background apparently.

An alternative might be to start a script when a workload starts that looks at the output directory size every X minutes (you'd set X based on method) and emails the list if the directory didn't grow through two ticks of X.

@brymerr921 suggested this which looks very promising. However, I can't get wevtutil to read the Agilent log. I think the Agilent log is not a standard Windows log format.

If we could get this worked out I think we'd save a fair amount of lost machine time due to failures not being visible.

curt-f commented 6 years ago

There is also https://www.agilent.com/en/products/crosslab-enterprise-services/laboratory-analytics/remote-advisor . I'm sure Agilent charges lots for it.

Anything that can read the Agilent log is preferred, for cost and also scalability / portability reasons.

My guess is the system logbook viewer is in a proprietary binary format. I further guess that figuring out the format would be only a small bit of work for folks with the right skills.

wdwvt1 commented 6 years ago

I bet we can hack this together - I will do a little digging and report back.

On Mon, Aug 6, 2018 at 1:33 PM curt-f notifications@github.com wrote:

There is also https://www.agilent.com/en/products/crosslab-enterprise-services/laboratory-analytics/remote-advisor . I'm sure Agilent charges lots for it.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Stanford-ChEMH-MCAC/protocols/issues/9#issuecomment-410844102, or mute the thread https://github.com/notifications/unsubscribe-auth/AA__-RqR6I7dinbwzUH1fqIhVPsyeoKRks5uOKgogaJpZM4VxBum .