Viewing of log file data in IBEX (similar to Journal data)

[ChrisM-S]

As a scientist choosing how to schedule data collection for the next experimental run and also as a person diagnosing an apparent instrument anomaly I would like to be able to view collected log file data from the IBEX NeXus/log files. This is the definitive record of instrument scientific output and it is important for me to be able to establish after the end of each run what has been stored.

In ticket #4960 it was clear that the logging in the journal viewer did not represent the logged data stored by the instrument in the NeXus files and perhaps worse, it provided a "plausible" but alternative view of the history of the data which suggested issues with the experiment which might require re-taking of sample data.

Being able to switch the log viewer to view and concatenate (if necessary) data from log/nexus files would ensure that this decision to re-take data is "fail-safe"

[ChrisM-S]

When I talked to the user and summarized to SG for ticket #4960 both said that they convinced themselves about the conflicting temperature data (from the log plots) by using the Journal Viewer. Is anyone aware of this facility in the IBEX journal viewer or could this be something which has enterprisingly added to the independent ISIS journal viewer...?? At the time I expected that the journal viewer would only show the expected required setpoints from a title at best and not a summary of logged values (other than beam current). So I was cautious about making this assumption without checking.

[kjwoodsISIS]

@ChrisM-S - I think there may be a little confusion in terminology here. IBEX has the Log Plotter view. You can use it to view the history of any block or PV. The IBEX Journal Viewer allows you to browse journal entries, nothing more. I suspect the scientist uses the term "journal viewer" to refer to both facilities.

[Tom-Willemsen]

When the scientists refered to "journal viewer" here, I think they were probably talking about Tris Youngs' journal viewer program ( https://www.projectaten.com/jv ) which does plot data from nexus files, not the IBEX journal viewer (which does not plot data in any way).

[John-Holt-Tessella]

Email sent to Mantid:

One of our users is using Tris Youngs’ journal viewer to look at history of blocks. Is this something which is easily accomplished in Mantid if so can we advertise it if not should it be a feature? I get the impression that when it works Tri’s program is good but when it doesn’t there is no support and that can be frustrating.

(User was S on Sandals)

[ChrisM-S]

Also, just to avoid losing the requirement, on SANS2D which is due for conversion soon (and possibly others, reflectometers?). GH’s “log file viewer” VI certainly has been used extensively for plotting of detailed SECI diagnostic logs of motion and other parameters where several logs need to be concatenated over time (several months years). On SECI instruments, these logs are written with dated names but in a different format from normal SECI (block) log files. I don’t think we have this tool in IBEX.

The question which should be answered, and this is for discussion, is what should someone be using when looking at a series of diagnostic data over 1-6 months or a year or two be using to concatenate and view logs when an issue (like the MAPS beamline or SANS2D front end positioning questions get asked). This strictly speaking (and in practice) is less used by the scientists but is important and I don’t think yet answered in IBEX – Could/can Mantid be used for example to plot all motion diagnostics on MERLIN for the last 18 months easily (i.e. with a simple algorithm and easily locate-able data trail?)

Secondly, of the Mantid and Diagnostic issue, is plotting “off instrument” usage really the best option? It is not at all easy to access a Mantid machine off the instrument VM from home or office. Putting a limited version of Mantid on the instrument is certainly not out of the question, and has been done before where out of the box diagnostic on the VM are useful (detectors have asked about this before).

[KathrynBaker]

Paragraph 1 – we don’t need it, the archiver and data browser provides the functionality. It’s also worth noting that GHs tool can NOT be run on an instrument, it is too resource hungry.

Paragraph 2: We have had numerous discussions relating to data retention and viewing it. So long as the value is listed for archive the instrument archive logs it to the MYSQL database in IBEX. The summation of those conversations can be found at https://github.com/ISISComputingGroup/ibex_developers_manual/wiki/Data-Generation-and-Storage The decision made for the archive data which would provide this information was that it would be stored on the instrument for one cycle, on easy access storage for two cycles, and then somewhere else forever as it is an infrequent request. I don’t think we actually do this, but that was the agreed plan for it.

Paragraph 3 There are various plans/desires for integrating Mantid and IBEX, we just don’t have the time to do them or define them well enough at the moment, at least from our side.

[ChrisM-S]

Did anyone reply to your email @John-Holt-Tessella about log plotting in Mandid? Or do we just leave it that the scientists best bet for plotting older logs is the Tris viewer (supported or not)?

[John-Holt-Tessella]

They did I am still chatting to them (slowly though)

[ChrisM-S]

I chatted with a muon instrument scientist today and they assured me that they have Mantid algorithms which can extract and loop through log data (multiple variables - if required) taking the run start/end times from Nexus files and using these to concatenate variables and dates/times into a single mantid workspace for plotting (or export).

This seems to satisfy the requirement of being able to retrieve/plot any data (over some period of a historic investigation - e.g. cryogenic equipment performance degradation, beamguide/jaw "creap" issues etc. against time) as long as there are Nexus files stored on the archive and appropriate blocks have been created during all experiments.

I do wonder if we should be including a small set of some basic AC: prefixed PVs in most/all NeXus files as default (Hidden?) blocks. This is assuming that these are not large amounts of data but should particularly include what we already show on the Beam Status OPI.

Maybe a good question for the Scientific advisory group? This could help to formalise our long term diagnostics archive policy (things not always taken into account by newer ISs).