Closed kjwoodsISIS closed 6 years ago
We should avoid duplicating features of the independent Journal Viewer, the journal viewer in seci was just a convenient place for a user to see what they had done recently. One i provide on the instrument is also available as a web page e.g. https://data.isis.rl.ac.uk/journals/ndxlarmor/journal_main.html which just renders an XML file written by the control program. I think we re currently avoiding embedding web pages directly for portability reasons, but serving and rendering the XML journal file would probably not be too big a job and give them the recent summary view i think they need.
If the journal is available on the web, why not simply provide a web-link to the web journal viewer?
I think they like having it integrated in the GUI for easy access, but you can suggest just an external web link to them - however they might consider it a loss of functionality compared to SECI. The web page is fed-id protected as you can look across instruments, as IBEX is client - server maybe we need to be a bit more information sensitive with journals like we are with run titles etc. In seci you had to log onto an instrument and just saw that instrument, so it was self contained.
One thing to add in favour of an embedded XML viewer in the client is that the current (browser specific) method doesn’t handle XML parsing errors well. Rendering the page into the client under control of a piece of rendering code in Java (even if only the client on the server machine) would hopefully allow much better error checking. I spent a long time manually running the MS-XML parser to track down an invalid UTF-8 character in a title within an old journal XML on LOQ last year. The problem manifested itself as the page not showing any combined journal which included this particular cycle’s log file (now fixed☺).
+1 from Vesuvio in their demo today.
+1 From MERLIN. This has been requested by several instrument scientists now. We should revisit it
I am not sure that the questions posed in the original description have been fully answered. I think we need to have a better understanding of what is wanted/required/needed for a journal viewer.
From Helen Walker (Merlin)
More generally, I was asking other people what they thought about the loss of the journal functionality moving between SECI and IBEX. Given the problems we’ve been having on Merlin this startup, we’ve had a lot of people remote desktopping in to help, and when you are looking from a small laptop screen, it really does make it a loss easier to follow what’s going on if the journal is directly viewable within the same piece of software
I'm not sure of the answer to several of the questions above. It needs elaborating. Was there a VI that did this? I get the impression scientists used to do it in SECI
I don't believe the SECI functionality was provided by a VI, but by some other component (some kind of XML viewer). I suggest we sit down in front of a copy of SECI and watch the journal viewer in action, making a note of the functionality it provides (these notes will provide the requirements for an equivalent system in IBEX). Then we can do a quick survey to see if there are existing XML viewers that we could use to meet the requirements and choose one to be the basis of our journal viewer.
There is one using a VI and one using an embedded browser. Also, before the XML journal (and still used occasionally now because they are complete - containing many cycles - and quick to search) people would look at JOURNAL.TXT
and SUMMARY.TXT
. These simply reflected the top 80 characters of the old .RAW file with an added RB number at the end column only in SUMMARY.TXT
. These are still written and are almost instantaneous to grep on the instrument for a user or sample e.g.
Run User Title Time uA RBNo
...
IRS36693Fernandez-Alonso,BusDCl d2 T=157K 15-JUL-2008 20:20:46 15.1 720070
IRS36694Fernandez-Alonso,BusDCl d2 T=158K 15-JUL-2008 20:36:46 15.0 720070
IRS36695Fernandez-Alonso,BusDCl d2 T=159K 15-JUL-2008 20:52:51 15.1 720070
IRS36696Fernandez-Alonso,BusDCl d2 T=160K 15-JUL-2008 21:08:57 15.1 720070
IRS36697Fernandez-Alonso,BusDCl d2 T=161K 15-JUL-2008 21:25:03 15.0 720070
IRS36698Fernandez-Alonso,BusDCl d2 T=162K 15-JUL-2008 21:41:04 15.1 720070
IRS36699Fernandez-Alonso,BusDCl d2 T=163K 15-JUL-2008 21:56:59 15.1 720070
IRS36700Fernandez-Alonso,BusDCl d2 T=164K 15-JUL-2008 22:12:55 15.1 720070
IRS36701Fernandez-Alonso,BusDCl Cooling 15-JUL-2008 22:33:15 15.1 720070
IRS36702Fernandez-Alonso,BusDCl Cooling 15-JUL-2008 22:38:47 15.1 720070
IRS36703Fernandez-Alonso,BusDCl Cooling 15-JUL-2008 22:44:18 15.1 720070
IRS36704Fernandez-Alonso,BusDCl Cooling 15-JUL-2008 22:49:54 15.1 720070
IRS36705Fernandez-Alonso,BusDCl Cooling 15-JUL-2008 22:55:25 15.1 720070
IRS36706Fernandez-Alonso,BusDCl Cooling 15-JUL-2008 23:01:01 15.1 720070
IRS36707Fernandez-Alonso,BusDCl Cooling 15-JUL-2008 23:06:32 15.1 720070
IRS36708Fernandez-Alonso,BusDCl Cooling 15-JUL-2008 23:12:03 15.1 720070
IRS36709Fernandez-Alonso,BusDCl Cooling 15-JUL-2008 23:17:34 15.1 720070
IRS36710Fernandez-Alonso,BusDCl Cooling 15-JUL-2008 23:23:05 15.1 720070
IRS36711Fernandez-Alonso,BusDCl Cooling 15-JUL-2008 23:28:37 15.1 720070
IRS36712Fernandez-Alonso,BusDCl Cooling 15-JUL-2008 23:34:08 15.1 720070
IRS36713Fernandez-Alonso,BusDCl Cooling 15-JUL-2008 23:39:40 15.1 720070
...
OK, so I now count 5 ways of looking at journal files:
To my way of thinking that's 4 too many.
We should implement one solution to take the place of items 2 & 3 (above). Tris can continue to develop and support his Journal Viewer as he sees fit. Anyone who wishes to continue using grep is welcome to do so (I assume they know how to use man pages ...).
we have MySQL on the instruments, we can populate a database table with journal information and then query this in IBEX. We could start with a simple table, move on to something about like the XML web page xstl, and then go beyond that and link up with the archive engine for block plotting / averaging.
OK. These sound like good suggestions. To make them a reality, we need a plan. It is going to take more than one ticket to implement the full Journal Viewer. I will arrange a meeting.
Here are the essential requirements captured from the meeting:
C$\Data\
SUMMARY.TXT
(one instance of this)JOURNAL.TXT
(one instance of this)XML journal files are named according to the following convention
Name: journal_YY_cc.xml
where, YY
= the last two digits of the cycle year, and cc
= the cycle number (within the year)
For example, a journal file called journal_17_04.xml
will have been created in cycle 2017/04.
Note: the cycle year refers to the ISIS financial year. Thus, cycle 2017/04, refers to financial year 2017-2018. And, in fact, cycle 2017/04, being at the end of the financial year, actually takes place in the first quarter of calendar year 2018.
SUMMARY.TXT
or JOURNAL.TXT
filesThe current web-based XML journal viewer allows the user to view more than just the current cycle. It allows the user to view the journal
There are often multiple Run Numbers for a single RB Number. When sorting by RB Number, it would be useful to sort by Run Number as a secondary sort.
It has been suggested that the IBEX journal viewer respect the "Show/Hide Title" toggle. However, this would lead to inconsistent behaviour.
Re: Requirement 1: it would not be difficult to place the journal information into the MySQL database already on the instrument, this would improve client performance (especially remote clients) and also easily allow searching and other enhanced features.
@FreddieAkeroyd what needs to change in order for the journal data to be placed in the MySQL database? Presumably, it is the ICP that generates these files; will it be modified to add records to a database table instead?
If we do put the data in the MySQL database, what happens to the journal_YY_cc.xml
, SUMMARY.TXT
and JOURNAL.TXT
files? I assume they would continue to be created (and therefore archived). Which, therefore, suggests the contents of the database table can be purged periodically?
@kjwoodsISIS yes the ICP would be modified to write the journal information to the database, it is already MySQL aware as it reads sample environment from the database, so it should be a reasonably simple change. The journal text files would continue to be created as some other applications rely on them. The journal database table could be purged periodically, but it will not grow at a large rate and being able to search back quite a few cycles might be useful to the scientists. Though it involves more changes, I think reading the journal information from MySQL will provide a better user experience and also be easier to program on the client side (not needing to worry about multiple XML files to parse)
If we take the MySQL route, as part of the process we should write a python program to ingest previous XML files into the database.
Actually, I don't strictly need to modify the ISISICP. To provide slack journal alerts without needing to update every ISISICP I created a separate JournalParser.exe program that is executed from end_of_run on each instrument, as well as sending the journal details to slack this program could MySQL it or otherwise
Where is the journal parsing program held?
Yes, I agree - using MySQL will provide a better user experience. But, in order to make use of it, we need to understand (and document) the flow of data from the ISISICP.
@John-Holt-Tessella journal parser is in github and checked out to C:\Instrument\Apps\EPICS\ISIS\JournalParser
Superseded by other tickets: #1978, #2901, #2703, #2704, #2706, #2723, #2837, #22900, #2901, #2902, #2903, #2904, #2905, #2952, 2953.
As a scientist, I would like the ability to view journal files in IBEX, so that I can view results from previous experiments.
Acceptance Criteria
Notes