kjwoodsISIS
commented
8 years ago Root cause of problems is that if Blocks are disconnected, it takes time to re-connect them. Need to explore options for an alternative mechanism - new issues.
Closed kjwoodsISIS closed 7 years ago
kjwoodsISIS
commented
8 years ago Root cause of problems is that if Blocks are disconnected, it takes time to re-connect them. Need to explore options for an alternative mechanism - new issues.
mattclarke
commented
8 years ago Freddie and I have discussed this and think the webpage should connect to the MySQL database rather than use CA. There are a few options for how we do this:
It probably makes sense for the webpage to be aspx based as that is what we used for Seci, but alternative suggestions are welcome.
KathrynBaker
commented
8 years ago I think my instinct would be option 2.
Aspx should be fine.
From: Matt Clarke [notifications@github.com] Sent: 20 May 2016 08:01 To: ISISComputingGroup/IBEX Subject: Re: [ISISComputingGroup/IBEX] Dataweb dashboard page is slow to load (#1191)
Freddie and I have discussed this and think the webpage should connect to the MySQL database rather than use CA. There are a few options for how we do this:
It probably makes sense for the webpage to be aspx based as that is what we used for Seci, but alternative suggestions are welcome.
� You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHubhttps://github.com/ISISComputingGroup/IBEX/issues/1191#issuecomment-220532329
John-Holt-Tessella
commented
8 years ago Could we create a view directly on the current database which pulls out the data we need?
kjwoodsISIS
commented
8 years ago We should definitely exploit technologies designed for extracting data from databases to create the web-page. I suggest we have a quick chat with Ed Vowles (who is working with Hannah's team - databases are Ed's area of expertise).
KathrynBaker
commented
8 years ago The archive database isn't the easiest to work with, otherwise I'd agree a view would be an idea to pursue. I'm competent with SQL, and I wouldn't want to write that query, even as a view.
From: John Holt [notifications@github.com] Sent: 20 May 2016 09:14 To: ISISComputingGroup/IBEX Cc: Baker, Kathryn (STFC,RAL,ISIS); Comment Subject: Re: [ISISComputingGroup/IBEX] Dataweb dashboard page is slow to load (#1191)
Could we create a view directly on the current database which pulls out the data we need?
� You are receiving this because you commented. Reply to this email directly or view it on GitHubhttps://github.com/ISISComputingGroup/IBEX/issues/1191#issuecomment-220543379
FreddieAkeroyd
commented
8 years ago I have a query in the ICP that would return the information required as I need to pull block details at end of run, however it would not be the most efficient way to get the most recent value. Maybe a database trigger to create a new table of the most recent values? The only caveat is that the archive engine currently only knows about blocks that you have asked to log, if you've asked it to display but not log I think it doesn't know about it.
KathrynBaker
commented
8 years ago Before I consider taking this on as my next ticket, can I clarify the behaviour we want to see at a higher level would be:
The config has a list of blocks to display The dataweb dashboard echoes that display of blocks (in groups)
Some extra facts relating to this: We need a mechanism to speed this up, and we feel that interacting with a database is a good option. The only thing that knows about which blocks are being displayed (not just logged) at the moment is the blockserver. The archive database knows about the blocks being logged, and only those, and is a horrid query to consider, so something new is needed. A trigger could be used, but the non-logged blocks would be unavailable on the dashboard, limiting the usefulness.
A suggestion to sort this (which could then be the work to do for the ticket, building on the discussion above):
A python service (part of the dbsvr?, possibly better as a separate one) pays attention to config changes, gets the groups and blocks list and populates them into a new table, monitors the blocks for value changes and updates any new values into the table- I haven't quite thought schema yet so there may be multiple tables needed for the most reliable database. Python might not be the best option for this, it just seems to be our language of choice for interfacing with our databases.
If we aren't worried about the blocks not being logged, then a trigger is a good option from the archiver, if we want to mess around with that database.
The dataweb server page would just query the database table directly to get the current snapshot.
It could be that there is a view within the table so that the dataweb server can keep the SQL it is using as simple as a select * from VIEW; (fields would probably need specifying and dealing with, but if more than one table is being used the view would allow the dataweb to just deal with one 'table')
It could be worth having a timestamp for the dashboard as a whole, which is updated by the python service periodically, and may be a separate query, but would show easily if something has failed, or the instrument is stopped, which would be a separate process to the database, which would keep running whether the instrument was stopped or not.
Whatever method we choose of getting the data where we want it, we probably need at least a mini-schema, so I'll have a think about this in the mean time.
KathrynBaker
commented
8 years ago Below is a schema that should cover the necessary info for the display, it may be that there are three queries (one for visible blocks only, one for all, and one for the RUN_INFO could have a field as below to cope with hiding the title etc. via a single query)
Please note this is a suggestion, and is open to discussion until we decide how we want to populate it, as until then, it may not be possible to do all that we want.
RUN_INFO Table
| Field | Source Record Name/PV | Display as | Notes |
|---|---|---|---|
| RUNSTATE | $(P)DAE:RUNSTATE_STR | Run Status | |
| RUNNUMBER | $(P)DAE:RUNNUMBER | Run Number | |
| RBNUMBER | $(P)ED:RBNUMBER | RB Number | Needs to be possible to hide this value |
| SURNAME | $(P)ED:SURNAME | User(s) | Needs to be possible to hide this value |
| TITLE | $(P)DAE:TITLE | Title | Needs to be possible to hide this value |
| HIDE_TITLE | ???? | Used to hide or not hide the RBNUMBER, SURNAME and TITLE | |
| STARTTIME | $(P)DAE:STARTTIME | Start Time | |
| RUNDURATION | $(P)DAE:RUNDURATION | Total Run Time | |
| RUNDURATION_PD | $(P)DAE:RUNDURATION_PD | Period Run Time | |
| GOODFRAMES | $(P)DAE:GOODFRAMES | Good Frames (Total) | |
| GOODFRAMES_PD | $(P)DAE:GOODFRAMES_PD | Good Frames (Period) | |
| RAWFRAMES | $(P)DAE:RAWFRAMES | Raw Frames (Total) | |
| RAWFRAMES_PD | $(P)DAE:RAWFRAMES_PD | Raw Frames (Period) | |
| PERIOD | $(P)DAE:PERIOD | Current Period | |
| NUMPERIODS | $(P)DAE:NUMPERIODS | Number Of Periods | |
| PERIODSEQ | $(P)DAE:PERIODSEQ | Period Sequence | |
| BEAMCURRENT | $(P)DAE:BEAMCURRENT | Beam Current | |
| TOTALUAMPS | $(P)DAE:TOTALUAMPS | Total Uamps | |
| COUNTRATE | $(P)DAE:COUNTRATE | Count-Rate | |
| DAEMEMORYUSED | $(P)DAE:DAEMEMORYUSED | DAE Memory Used | |
| TOTALCOUNTS | $(P)DAE:TOTALCOUNTS | Total DAE Counts | |
| DAETIMINGSOURCE | $(P)DAE:DAETIMINGSOURCE | DAE Timing Source | |
| MONITORCOUNTS | $(P)DAE:MONITORCOUNTS | Monitor Counts | |
| MONITORSPECTRUM | $(P)DAE:MONITORSPECTRUM | Monitor Spectrum | |
| MONITORFROM | $(P)DAE:MONITORFROM | Monitor From | |
| MONITORTO | $(P)DAE:MONITORTO | Monitor To | |
| NUMTIMECHANNELS | $(P)DAE:NUMTIMECHANNELS | Number Of Time Channels | |
| NUMSPECTRA | $(P)DAE:NUMSPECTRA | Number Of Spectra | |
| SHUTTER | IN:$(INST):SHTR:STAT | Shutter Status | INST is defined by the server, and will be based on the instrument |
| CONFIG | $(P)CS:BLOCKSERVER: GET_CURR_CONFIG_DETAILS | Configuration | Is buried in the configuration JSON, will need to be updated on config change |
GROUPS Table
| Field | Source Record Name/PV | Display as | Notes |
|---|---|---|---|
| ID | Internal field to make it easier to sort the blocks using a PK rather than a text field | ||
| Name | $(P)CS:BLOCKSERVER: GET_CURR_CONFIG_DETAILS | {Value in the table} | Each name would be displayed, with the blocks underneath, the groups are buried in the configuration JSON, will need to be updated on config change |
BLOCKS Table
| Field | Source Record Name/PV | Display as | Notes |
|---|---|---|---|
| ID | A PK is always useful | ||
| NAME | $(P)CS:BLOCKSERVER: GET_CURR_CONFIG_DETAILS | {Value in the table} | Each block is displayed below the appropriate group, the block information is buried in the configuration JSON, and will need updating on config change |
| VISIBLE | $(P)CS:BLOCKSERVER: GET_CURR_CONFIG_DETAILS | This will be an informational/meta field relating to whether or nothe block should be display, if this is false, then the block should not be displayed unless the state of show hidden blocks has been selected | |
| GROUP_ID | This is just to allow the link between the group name and the block | ||
| SOURCE | $(P)CS:BLOCKSERVER: GET_CURR_CONFIG_DETAILS | The value of this field would be the block PV for lookup, this would be the source of the VALUE field | |
| VALUE | $(P)CS:SC:$(NAME) | This is displayed alongside the block name, the NAME in the source is the name of the block, if triggering entries from another table, that source can be ignored | |
| SETPOINT_EXISTS | This may or may not be useful, but could help define whether or not to dsiplay a setpoint (if we have a way of checking for one) | ||
| SETPOINT_SOURCE | This would need to be gernerated programatically, as the SP is not maintanined by the block server, this would be the source of the SETPOINT_VALUE | ||
| SETPOINT_VALUE | $(P)CS:SC:$(NAME):SP | This is displayed as (Setpoint=value) alongside the block value, if there is a setpoint and value available |
The PV to monitor for a config change is: $(P)CS:BLOCKSERVER:SERVER_STATUS
A suggested flow for a separate program: On starting the system, each of the DAE PVs and the shutter status are read, the values updated, and a monitor placed on them to update the values in the database as they change. It is assumed that the configuration is changed. After a change of configuration: Clear the tables Get the information on the current configuration, and unpack the JSON Update the CONFIG in RUN_INFO Update the tables with the information in the config details, checking for SPs if necessary Monitor the PVs for the blocks (value and setpoint), and update teh database on value change Monitor the configuration status PV, if that changes state to "LOADING_CONFIG", or anything other than "", wait for it to finish, and there has been a config change trigger the after config change code
FreddieAkeroyd
commented
8 years ago Quite a lot of the above, such as adjusting blocks to config changes and monitoring values, can be done by the archive engine technology – can we make use of a third archive engine (which would know about all blocks, logged or otherwise) configured by the blockserver? Adding an on-update trigger to the archive sample table could update a separate current values block table, this table would also have “visible” column etc. column mentioned above. The “display as” field could come from the desc field of the channel table generated by the block server, or be set elsewhere.
John-Holt-Tessella
commented
8 years ago I am not entirely sure how the setpoint thing works, but if you are happy with this then excellent.
I like the idea of a separate program and the rest looks awesome.
KathrynBaker
commented
8 years ago Freddie: If we go down a third archive engine route, then I’m not likely to take this ticket up, I don’t think I have the time to learn the archive engine side of things and get this ticket done this sprint. There’s also the fact that is a current snapshot rather than an archive, so how do the archive engines cope with the kind of clear down each time the server is restarted that this would need? I don’t think we’ll want ‘old’ blocks hanging around. All of this might be easy if you know the archive engine bits, though so I am quite happy for the concept to be corrected.
John: Last updated time, yes, I think I had that in my head, wasn’t sure where to put it. Can certainly add the precision in. Run information as a special group – I did think about this, and the main reason I didn’t go with that option, is basically for the change of config. We only want the current blocks in this snapshot of the system, so that table is likely to get a reasonable number of “delete * from” commands run on it, now it can be limited to only deleting from outside of the run_info group, I was just after keeping the SQL interactions as simple as possible. But using the group style schema might not be a bad idea for adding extra values in the future. Setpoints – SECI instruments display a setpoint if they have one available, it strikes me as still being a good idea, you ask for a setpoint of 10 and the read back is stuck at 6, you know you want to look and see what is going on on the instrument.
FreddieAkeroyd
commented
8 years ago The third archive engine might be slightly more complicated than i first though as a channel can only be owned by one engine, so the block server already owns the blocks. The archive engine has the concept of channel groups, so non-logged values could be logged in a different channel group so as not to get archived with the raw data. I agree the archive engine is not the simplest being, but I was just trying to weigh it against writing a new service that does a lot of the same things it is already doing. Old Blocks in the channel table could also get put into another "group" to stop them showing up in the dashboard. I think it may be possible to get the blockserver to set the block archive up after a config change in a way it could be used by an external dashboard, that then saves us having to manage the logging. Maybe managing a separate logging service isn't so bad, but it just felt like duplication at some level so i wanted to be sure it was the correct approach.
KathrynBaker
commented
8 years ago If I felt the solution to the requirement was simple I would have carried on rather than opening up a debate, so I certainly appreciate looking at this in different ways.
It does feel like duplication, but I think that might be down to the oddities of the archive engine – if the ASP query were a plausible option, we could have gone directly there, and just added in the non-logged blocks somehow, but I think that was discredited as an SQL nightmare.
So, do we have a conclusion over which way to try and enact this?
FreddieAkeroyd
commented
8 years ago Another option is just reformatting the web page already provided by the archive engine see e.g. http://control-svcs.isis.cclrc.ac.uk/archive/group?name=BEAM though HTML it seems to parse as valid XML, also it has "current values" as well as "last archived value" so it would appear that non-logged values could be added as low frequency logged values by the block server. Currently this is hidden on localhost the instrumens but http://localhost:4813/group?name=BLOCKS on NDXDEMO will give the details there. Unavailable blocks are shown as disconnected.
KathrynBaker
commented
8 years ago Does it give us the hidden blocks? If not how difficult will it be to add them? How difficult will it be to add in the run information? If the answers to those questions are “not at the moment”, “easy” and “easy”, then I’d say we go with this option, as it probably causes the least hassle. If the answers are something else, we might need to think about it a little more still, as giving less functionality than is available now is probably not a good option.
ChrisM-S
commented
8 years ago Can I just clarify the protocol chain, is it
HTTP(s) -->ASPX -->CA --> "Ibex Client Protocol" (Blocks or naked PVs) (external) (webserver) (Instrument PC)
don't answer if I've got it right :-)
ChrisM-S
commented
8 years ago Sorry not used to the interface.
KathrynBaker
commented
8 years ago Just to confirm the plan after further discussions this morning:
We need to make the html page of the archive engine for the blocks visible beyond the instrument. The blockserver will be altered, so that as well as the existing group of BLOCKS, there will be a log of the RUN_INFO PVs, and the non-logged blocks, put into a separate group in the archive engine called DATAWEB (or two groups, one for RUN_INFO and one for NON_LOGGED). The RUN_INFO will have to use the direct PVs not block format PVs to get around the fact that there may be a block showing DAE information. The asp server would periodically get the CURR_CONFIG_DETAILS PV (for the information to match blocks to groups and whether or not they are hidden), and the XML (via the archive engine html, for each group). It would then parse the config details, and using that and the XML build the dataweb page for display.
I will split this ticket, as there is quite a bit of work there into three: 1 – Make the page available more widely 2 – The blockserver changes 3 – The aspx changes
FreddieAkeroyd
commented
8 years ago Actually 1 is already done http://ndxlarmor:4813/group?name=BLOCKS
KathrynBaker
commented
8 years ago Sub-tickets have been created as follows: Blockserver changes #1370 aspx changes #1371
KathrynBaker
commented
8 years ago To avoid needing a PV at all a ticket to cover the blockserver providing the current config details via a URL has been created #1380
This means that the page can be generated in a completely different way, and so #1371 has been replaced with #1382
KathrynBaker
commented
8 years ago It has also been suggested that #1380 and #1382 be undertaken by the work experience student Dom is overseeing during SPRINT_2016_07_14, so shall change the milestone and status of this ticket too
AdrianPotter
commented
8 years ago Request has been covered by recent dataweb work. Only outstanding ticket is #1382
As a scientist, I want to view the dataweb dashboard page for my instrument, so that I can quickly check the status of my instrument when I am not in the lab.
Acceptance Criteria
Notes