Baton Passing: Analysis

[kjwoodsISIS]

As a support person, I want to investigate the requirements for a baton passing system, so that we have some option available if we decide to implement such a scheme.

'''Acceptance Criteria'''

1. A document describing the investigation - results of analysis, pros/cons, conclusions & recommendations.

'''Notes'''

1. The essence of a baton passing system is that the system maintains an object called a “baton”. Users of the system can request “ownership” or “possession” of the baton. The user who has possession of the baton has control of the instrument (i.e. can change PVs, etc.). Users who do not have possession of the baton do not have control (i.e. can view, but not change PVs). Some things to think about:
   • Only authorised users can request possession of the baton.
   • A user can request possession of the baton (if they don’t already have it)
   • The user who has the baton can agree to relinquish possession, or they can refuse to do so.
   • If a user who has the baton has been inactive for a period of time, then the system automatically reclaims the baton.
   • If no-one has the baton, then the next person to request it, gets it.
   • Each instrument has its own baton; there is only one baton per instrument.
   • In an emergency, a privileged user needs to be able to demand instant possession of the baton.
     1. I think it would be good to sketch out how to implement a baton system in EPICS and to point out where we will need to think about security.
   • What types of permissions/roles do users need?
   • How do we assign these permissions/roles to users?
   • How do we identify individual users? - e.g. via their federal ID?
   • How might we implement this in EPICS?
   • SNS use a scheme in which the baton is held, in effect, by the Script Server. Is this a better model?
   • How do we ensure that only one user can be in possession of the baton (i.e. two or more people can't all be in possession at the same time).
   • How long before the system reclaims an inactive baton? (10 mins?, 20mins?, ...). Probably needs to be configurable.
   • What happens if the system loses track of the baton? (e.g. the service that manages the baton crashes)
   • How do we make it "fail safe"? (i.e. fails so that no-one has control until the system can be restored to normal service).
   • It would probably be helpful if the GUI displayed who currently holds the baton.

[kjwoodsISIS]

Imported from trac issue 295. Created by skn09965 on 2014-02-25T10:53:35, last modified: 2014-03-26T13:42:59

[kjwoodsISIS]

Trac comment by sjb99183 on 2014-02-26 11:01:59:

• The gateway enforces control by restricting write permission to the baton holder (identified by logged in user and/or machine).
• We have an IOC that controls who has the baton. When ownership changes, it changes the gateway’s access.txt and tell it to reread it. On startup, it should check the access.txt to see who currently has the baton.
• There is a PV that clients can write to to request the baton. This needs to be in its own ASG in the gateway so non-baton holders can write to it.
  • If they are in the permitted list (a file somewhere in settings) and the baton is free then they get the baton.
  • If there is a baton holder, the IOC sets a PV which the GUI will see and prompt it’s user to relinquish the baton, or reject the request. There needs to be a (configurable) timeout on this process so a live but unattended GUI cannot hog the baton.
• There needs to be a timeout mechanism, so the baton is automatically relinquished if the holder disconnects. This could be done by regularly incrementing a PV which the GUI monitors. A script in the GUI would respond by writing the new value back to another PV. If the two PVs differ by too much, the IOC assumes the client is dead and reclaims the baton. The timeout period needs to be configurable and different from that for requesting the baton.
• If we use the SNS Script Server, it will be privileged so it can seize the baton at any time. It is not subject to timeouts or requests to relinquish the baton. If the script server is running on NDXLARMOR, its access will not be controlled by the gateway, but we need to ensure that it does take the baton so everybody else is locked out. If it is not on NDXLARMOR, the mechanism by which it seizes the baton needs to be exposed by the gateway and we will need to figure out how to prevent it being abused (eg run it as a special user or make its machine privileged?).

== Local users == Since security is enforced by the gateway, any user on NDXLARMOR will be able to write to PV with or without the baton.

• We should discourage people from logging on to NDXLARMOR. With EPICS there is no longer any need to do so (except for support staff)
• We will need to provide a mechanism to manually claim and release the baton (eg a python script), but there is no way to force local users to use it.
• The normal timeout mechanism will not apply, so if they forget to release the baton, somebody will have to log in and release it.

== Requesting the baton == If the baton is free and two clients request it at the same time, the process needs to be atomic so one gets it and the other’s request is not processed until the baton has been claimed.

If the baton is not free (and not held by the script server) and a client requests it, that client’s id needs to be cached, so if the baton holder relinquishes it, that client gets it rather than some other client who happened to request it while the first was waiting for it to be relinquished.

Multiple simultaneous requests are unlikely so there is no need for a proper queue. Instead, while a request is in progress, we just remember the most recent new request and when the current request succeeds or is rejected, we initiate that request.

In the GUI the user should see who has the baton. If we don’t implement a proper queue, they should also be able to see who, if anyone, is requesting the baton. If not held by the script server, the user can press a button to request it. The user then needs to see what is happening to their request:

• You have the baton
• The script server has taken it (it could have started at the same time the user was pressing the button)
• The holder has been asked to relinquish it
• The holder has refused to relinquish it
• Another request is in progress (and yours has been rejected/queued)

If a user gets a request to relinquish the baton, they should be told who is requesting it.

== Grabbing the baton in an Emergency == A variant use-case is to make an emergency request to seize the baton. This should only be available to privileged users (e.g. the instrument scientist himself or support personnel, rather than visiting scientists). In an emergency, you simply request the baton (maybe enter your privileged password) and you get immediate control (the current holder gets no opportunity to refuse your request, the script server loses the baton too).

== Implementation == Implementing this would mean:

• adding code to the inst etc IOC to handle the baton setting
• the timeout and baton request mechanisms can be handled almost entirely by database records
• The baton needs adding to the GUI which includes adding a couple of small python scripts to keep the hold of the baton and to respond to requests to relinquish it.
• The script server would also need changing to grab the baton.
• We would need to define three classes of user:
  • Users who can grab the baton
  • Users who can request the baton
  • Users who cannot take control

[kjwoodsISIS]

Trac comment by skn09965 on 2014-02-28 14:17:24:

'''Note:''' A baton-passing scheme has been implemented in the past at ISIS (e.g. on the old VMS-based control system). While many people seem to like the idea of a baton-passing system, experience shows that few people actually use it. In practice, it may be simpler to limit the use of a baton to the Script Server. In this scheme:

• when the Script Server is asked to run a script (and that script is ready to run), it grabs the baton and holds it until the script has completed. If there is more than one ready-to-run script in the queue, then the Script Server retains the baton until all the scripts have run. Once all scripts have run, the Script Server relinquishes the baton.
  • when a user wants to change a PV (not using a script) he/she is free to do so, unless the Script Server holds the baton. Otherwise, the user has to wait until the Script Server has completed executing any queued scripts.

[kjwoodsISIS]

Trac comment by sjb99183 on 2014-03-03 09:42:44:

If only the script server holds the baton then things are much simpler.

There is still a PV that the script server writes to to gain and release the baton and the inst etc IOC still monitors it, but now it would modify access.txt to give write access to everyone (or all registered users) or to restrict it to the script server.

The GUI would just have a label to indicate whether the script server had the baton.

Seizing the baton in an emergency would be a matter of stopping the script server, although we would need to make sure that it always set the PV to release the baton.

[kjwoodsISIS]

Trac comment by skn09965 on 2014-03-20 16:57:19:

Ready for Review

[kjwoodsISIS]

Trac comment by mjc23 on 2014-03-25 13:30:17:

Can we put this in a proper document (wiki or !SharePoint) rather than in the comments of a ticket please?

Done! Content of this ticket placed here: [wiki:BatonPassing Some Ideas for Implementing Baton Passing]