DAE PVs Disconnected in the Archiver

DominicOram commented 7 years ago

Occasionally, although the DAE is running fine, the archiver believes that some of the PVs are disconnected. This is particularly annoying as it's where the web dashboard gets it's data from.

DominicOram commented 7 years ago

This has been seen on IMAT and ZOOM. It was fixed on ZOOM through an instrument reset.

kjwoodsISIS commented 7 years ago

Well get NAGIOS to watch it first. Then, when we can reproduce it, we can tackle this ticket.

FreddieAkeroyd commented 7 years ago

The blocks say "reconnecting" in the archiver, some PVs do mention the "60 tries" that Dominic referred to yesterday, but there are also one on pure monitor that are not working (e.g. title). I'm guessing it is a java channel access issue and all we can do is restart the inst archiver. There is a check for missing PVs in nagios called WEB-DASHBOARD-ARCHIVER

To test ticket:

[x] Look at current ibex web dashboard http://dataweb.isis.rl.ac.uk/Dashboards/default.htm IMAT should be giving problems with disconnected PVS
[ ] Log into varanus.nd.rl.ac.uk confirm WEB-DASHBOARD-ARCHIVER is showing error
[x] Restart inst archiver on IMAT using URL http://ndximat.isis.cclrc.ac.uk:4812/restart - should be safe to do this anytime, but wait for instrument in setup if possible
[x] Go to http://ndximat.isis.cclrc.ac.uk:4812/main and check archiver says "RUNNING"
[x] check nagios again and wait for an email

@GDH-ISIS could you please review this ticket

GDH-ISIS commented 7 years ago

Did not need to log onto varanus.nd.rl.ac.uk - not sure how and I do not have any user-name or password. Checked IMAT was in Setup. Clicked on the two links above. Dataweb appears to have come back. Forced Nagios to check. Got the email.

ISISComputingGroup / IBEX

DAE PVs Disconnected in the Archiver #2155