Journal Viewer

[kjwoodsISIS]

As a scientist, I would like the ability to view journal files in IBEX, so that I can view results from previous experiments.

Acceptance Criteria

1. To be defined (after discussion of the points below).

Notes

1. This ticket arises from a discussion with the IRIS team. They noted that SECI has a journal viewer, but IBEX doesn't.
2. The purpose of the journal viewer is to view to read and display journal files from ISIS instruments (i.e. to view data from completed experiments).
3. SECI currently has a built-in journal viewer.
4. There is an independent Journal Viewer (see https://www.projectaten.com/jv).
5. Question: should we implement a journal viewer in IBEX?
6. Question: if there is an independent journal viewer, why create a new one? Why replicate some or all of that functionality in IBEX?
7. Question: the existing, independent journal viewer reads SECI logs and NeXus files. Can it read logs produced by IBEX?
8. Question: would it not be better if the independent journal viewer was made available to all scientists?
   1. Maybe it is?
   2. If so, what is the compelling reason to have one in IBEX?

[FreddieAkeroyd]

We should avoid duplicating features of the independent Journal Viewer, the journal viewer in seci was just a convenient place for a user to see what they had done recently. One i provide on the instrument is also available as a web page e.g. https://data.isis.rl.ac.uk/journals/ndxlarmor/journal_main.html which just renders an XML file written by the control program. I think we re currently avoiding embedding web pages directly for portability reasons, but serving and rendering the XML journal file would probably not be too big a job and give them the recent summary view i think they need.

[kjwoodsISIS]

If the journal is available on the web, why not simply provide a web-link to the web journal viewer?

[FreddieAkeroyd]

I think they like having it integrated in the GUI for easy access, but you can suggest just an external web link to them - however they might consider it a loss of functionality compared to SECI. The web page is fed-id protected as you can look across instruments, as IBEX is client - server maybe we need to be a bit more information sensitive with journals like we are with run titles etc. In seci you had to log onto an instrument and just saw that instrument, so it was self contained.

[ChrisM-S]

One thing to add in favour of an embedded XML viewer in the client is that the current (browser specific) method doesn’t handle XML parsing errors well. Rendering the page into the client under control of a piece of rendering code in Java (even if only the client on the server machine) would hopefully allow much better error checking. I spent a long time manually running the MS-XML parser to track down an invalid UTF-8 character in a title within an old journal XML on LOQ last year. The problem manifested itself as the page not showing any combined journal which included this particular cycle’s log file (now fixed☺).

[Tom-Willemsen]

+1 from Vesuvio in their demo today.

[AdrianPotter]

+1 From MERLIN. This has been requested by several instrument scientists now. We should revisit it

[kjwoodsISIS]

I am not sure that the questions posed in the original description have been fully answered. I think we need to have a better understanding of what is wanted/required/needed for a journal viewer.

1. It seems that scientists want some kind of quick, convenient way of viewing journal files.
   • I am guessing, therefore, that some scientists think the current, independent Journal Viewer is not quick & convenient.
2. What functionality should a "quick, convenient" journal viewer have?
   • we should have a proper statement of these requirements, which implies speaking to the scientists and understanding what the current SECI viewer does.
3. Why does it have to be built in to IBEX?
   • Why not have it as a stand-alone application that can be launched from IBEX?
4. If we build a "quick, convenient" journal viewer, what's to stop people asking for it to have more and more features?
   • If they want more features, they can use the independent Journal Viewer.

[AdrianPotter]

From Helen Walker (Merlin)

More generally, I was asking other people what they thought about the loss of the journal functionality moving between SECI and IBEX. Given the problems we’ve been having on Merlin this startup, we’ve had a lot of people remote desktopping in to help, and when you are looking from a small laptop screen, it really does make it a loss easier to follow what’s going on if the journal is directly viewable within the same piece of software

I'm not sure of the answer to several of the questions above. It needs elaborating. Was there a VI that did this? I get the impression scientists used to do it in SECI

[kjwoodsISIS]

I don't believe the SECI functionality was provided by a VI, but by some other component (some kind of XML viewer). I suggest we sit down in front of a copy of SECI and watch the journal viewer in action, making a note of the functionality it provides (these notes will provide the requirements for an equivalent system in IBEX). Then we can do a quick survey to see if there are existing XML viewers that we could use to meet the requirements and choose one to be the basis of our journal viewer.

[ChrisM-S]

There is one using a VI and one using an embedded browser. Also, before the XML journal (and still used occasionally now because they are complete - containing many cycles - and quick to search) people would look at JOURNAL.TXT and SUMMARY.TXT. These simply reflected the top 80 characters of the old .RAW file with an added RB number at the end column only in SUMMARY.TXT. These are still written and are almost instantaneous to grep on the instrument for a user or sample e.g.

Run     User             Title                      Time                    uA    RBNo

...
IRS36693Fernandez-Alonso,BusDCl d2 T=157K           15-JUL-2008 20:20:46    15.1  720070
IRS36694Fernandez-Alonso,BusDCl d2 T=158K           15-JUL-2008 20:36:46    15.0  720070
IRS36695Fernandez-Alonso,BusDCl d2 T=159K           15-JUL-2008 20:52:51    15.1  720070
IRS36696Fernandez-Alonso,BusDCl d2 T=160K           15-JUL-2008 21:08:57    15.1  720070
IRS36697Fernandez-Alonso,BusDCl d2 T=161K           15-JUL-2008 21:25:03    15.0  720070
IRS36698Fernandez-Alonso,BusDCl d2 T=162K           15-JUL-2008 21:41:04    15.1  720070
IRS36699Fernandez-Alonso,BusDCl d2 T=163K           15-JUL-2008 21:56:59    15.1  720070
IRS36700Fernandez-Alonso,BusDCl d2 T=164K           15-JUL-2008 22:12:55    15.1  720070
IRS36701Fernandez-Alonso,BusDCl Cooling             15-JUL-2008 22:33:15    15.1  720070
IRS36702Fernandez-Alonso,BusDCl Cooling             15-JUL-2008 22:38:47    15.1  720070
IRS36703Fernandez-Alonso,BusDCl Cooling             15-JUL-2008 22:44:18    15.1  720070
IRS36704Fernandez-Alonso,BusDCl Cooling             15-JUL-2008 22:49:54    15.1  720070
IRS36705Fernandez-Alonso,BusDCl Cooling             15-JUL-2008 22:55:25    15.1  720070
IRS36706Fernandez-Alonso,BusDCl Cooling             15-JUL-2008 23:01:01    15.1  720070
IRS36707Fernandez-Alonso,BusDCl Cooling             15-JUL-2008 23:06:32    15.1  720070
IRS36708Fernandez-Alonso,BusDCl Cooling             15-JUL-2008 23:12:03    15.1  720070
IRS36709Fernandez-Alonso,BusDCl Cooling             15-JUL-2008 23:17:34    15.1  720070
IRS36710Fernandez-Alonso,BusDCl Cooling             15-JUL-2008 23:23:05    15.1  720070
IRS36711Fernandez-Alonso,BusDCl Cooling             15-JUL-2008 23:28:37    15.1  720070
IRS36712Fernandez-Alonso,BusDCl Cooling             15-JUL-2008 23:34:08    15.1  720070
IRS36713Fernandez-Alonso,BusDCl Cooling             15-JUL-2008 23:39:40    15.1  720070
...

[kjwoodsISIS]

OK, so I now count 5 ways of looking at journal files:

1. Tristan Youngs journal viewer
2. a VI-based journal file viewer
3. an XML based browser
4. grepping SUMMARY.TXT
5. grepping JOURNAL.TXT

To my way of thinking that's 4 too many.

We should implement one solution to take the place of items 2 & 3 (above). Tris can continue to develop and support his Journal Viewer as he sees fit. Anyone who wishes to continue using grep is welcome to do so (I assume they know how to use man pages ...).

[FreddieAkeroyd]

we have MySQL on the instruments, we can populate a database table with journal information and then query this in IBEX. We could start with a simple table, move on to something about like the XML web page xstl, and then go beyond that and link up with the archive engine for block plotting / averaging.

[kjwoodsISIS]

OK. These sound like good suggestions. To make them a reality, we need a plan. It is going to take more than one ticket to implement the full Journal Viewer. I will arrange a meeting.

[kjwoodsISIS]

Here are the essential requirements captured from the meeting:

IBEX Journal Viewer Requirements

Pre-amble

• Journal files are created by the ICP (Instrument Control program - aka the DAE control program).
• Journal files are stored in C$\Data\
• Three types of journal file are created:
  • SUMMARY.TXT (one instance of this)
  • JOURNAL.TXT (one instance of this)
  • XML journal files (one per cycle) All three types of journal file are assumed to be refreshed after each run (FAA to confirm)

XML journal files are named according to the following convention Name: journal_YY_cc.xml where, YY = the last two digits of the cycle year, and cc = the cycle number (within the year) For example, a journal file called journal_17_04.xml will have been created in cycle 2017/04. Note: the cycle year refers to the ISIS financial year. Thus, cycle 2017/04, refers to financial year 2017-2018. And, in fact, cycle 2017/04, being at the end of the financial year, actually takes place in the first quarter of calendar year 2018.

Requirements

1. The IBEX journal viewer will only process XML journal files.
   • for the avoidance of doubt, the IBEX journal viewer will not process the SUMMARY.TXT or JOURNAL.TXT files
2. The IBEX journal viewer will operate independently of any other journal viewer in use at ISIS
   • there will be no dependencies, no communication, no interaction with other journal viewers
3. The IBEX journal viewer will display journal files for the instrument at which the client is currently pointing
4. The IBEX journal viewer must display the journal for the current cycle
5. The IBEX journal viewer could display the journal for other cycles (but not in the first version)
6. The contents of XML journal files will be rendered as a simple scrolling table - the journal table.
   • each row of the journal table will display the details of a single run
   • the journal table will display the following columns (in the order listed below):
     • Run Number
     • Title
     • Start Time
     • Duration
     • uAmp Hours
     • RB Number
     • Users
7. By default, the journal viewer must display the entries in date/time order (most recent run first).
8. The user must be able to sort the journal table by clicking on a column header
   • clicking on the header a second time should reverse the sort order
   • default sorting orders (i.e. on first click on the header) are:
     • Run Number (from highest to lowest - so that table is sorted by most recent run)
     • Title (alphabetically, a - z)
     • Start Time (from most recent to oldest)
     • Duration (from shortest to longest)
     • uAmp Hours (from lowest to highest)
     • RB Number (from highest to lowest)
     • Users (alphabetically, a - z)

Notes

1. The current web-based XML journal viewer allows the user to view more than just the current cycle. It allows the user to view the journal

   1. for the current cycle
   2. for all cycles (meaning journal files for all cycles stored on the control PC)
   3. for a specified cycle
   4. between any two specified run numbers These capabilities will not be present in the IBEX journal viewer (at least not in the first version).

2. There are often multiple Run Numbers for a single RB Number. When sorting by RB Number, it would be useful to sort by Run Number as a secondary sort.

3. It has been suggested that the IBEX journal viewer respect the "Show/Hide Title" toggle. However, this would lead to inconsistent behaviour.

   • the purpose of the "Show/Hide Title" toggle is to control the display of the experiment title on the web dashboard
   • the "Show/Hide Title" toggle only applies to the current experiment; it has no effect on completed experiments (which, by definition, are not shown on the web dashboard). Journal files, on the other hand, are primarily about completed experiments.
   • I can view the current experiment title simply looking at the Dashboard pane in the main IBEX window, or the DAE view. Hiding the title in the journal viewer won't make any difference to that and would only serve to confuse users
   • the IBEX journal viewer has nothing to do with the web dashboard; we should not try to couple them in any fashion. Therefore, I believe we should ignore the "Show/Hide Title" toggle in the journal viewer.

[FreddieAkeroyd]

Re: Requirement 1: it would not be difficult to place the journal information into the MySQL database already on the instrument, this would improve client performance (especially remote clients) and also easily allow searching and other enhanced features.

[kjwoodsISIS]

@FreddieAkeroyd what needs to change in order for the journal data to be placed in the MySQL database? Presumably, it is the ICP that generates these files; will it be modified to add records to a database table instead?
If we do put the data in the MySQL database, what happens to the journal_YY_cc.xml, SUMMARY.TXT and JOURNAL.TXT files? I assume they would continue to be created (and therefore archived). Which, therefore, suggests the contents of the database table can be purged periodically?

[FreddieAkeroyd]

@kjwoodsISIS yes the ICP would be modified to write the journal information to the database, it is already MySQL aware as it reads sample environment from the database, so it should be a reasonably simple change. The journal text files would continue to be created as some other applications rely on them. The journal database table could be purged periodically, but it will not grow at a large rate and being able to search back quite a few cycles might be useful to the scientists. Though it involves more changes, I think reading the journal information from MySQL will provide a better user experience and also be easier to program on the client side (not needing to worry about multiple XML files to parse)

[FreddieAkeroyd]

If we take the MySQL route, as part of the process we should write a python program to ingest previous XML files into the database.

[FreddieAkeroyd]

Actually, I don't strictly need to modify the ISISICP. To provide slack journal alerts without needing to update every ISISICP I created a separate JournalParser.exe program that is executed from end_of_run on each instrument, as well as sending the journal details to slack this program could MySQL it or otherwise

[John-Holt-Tessella]

Where is the journal parsing program held?

[kjwoodsISIS]

Yes, I agree - using MySQL will provide a better user experience. But, in order to make use of it, we need to understand (and document) the flow of data from the ISISICP.

[FreddieAkeroyd]

@John-Holt-Tessella journal parser is in github and checked out to C:\Instrument\Apps\EPICS\ISIS\JournalParser

[kjwoodsISIS]

Superseded by other tickets: #1978, #2901, #2703, #2704, #2706, #2723, #2837, #22900, #2901, #2902, #2903, #2904, #2905, #2952, 2953.