SANS2D: Commission Aperture and Guides into IBEX

[DominicOram]

As a user on SANS2D I would like to be able to move my apertures and guides in IBEX. Accuracy of movement is very important as bringing the system to vacuum while items are in the wrong place could damage the instrument. The axes are also buried so care will have to be taken to ensure their position is well known.

Acceptance Criteria

• A commissioning plan has been written that will confirm the axis have the same functionality in IBEX as in SECI
• The plan has been completed and the scientists are happy that motion is the same under IBEX/SECI
• Switching control back and forth between SECI/IBEX is as smooth as possible

[davidkeymer]

It is possible that the homing routines for the front end axes will need modifications if they don't fit one of the standards, as was found for the rear detector during previous testing. See PR for details.

[DominicOram]

Testing scheduled for 29th October onwards.

Feel free to edit the below:

Draft Commissioning Plan

Do the following with all axes:

• [x] The following data will be collected under SECI:
  • Home position (mm)
  • Position of high limit (mm) - as measured from moving starting from home
  • Position of low limit (mm) - as measured from moving starting from home
  • Distance between limits (mm)
  • Time taken between home and both limit switches
  • Time taken between limit switches
  • Which limits are high/low on the controller
• [x] SECI will be converted to redefine home as zero (https://github.com/ISISComputingGroup/IBEX/issues/5639) - Ensuring old settings are still kept
• [x] Rehome and confirm distances to limits are still the same
• [x] Switch to IBEX and copy over SECI settings
• [x] Note current positions
• [x] Confirm axes are still reading the same position in IBEX as SECI
• [x] Move axes a small amount in IBEX, confirm they appear to move correctly
• [x] Move axes to both limits and confirm the following are the same in IBEX as in SECI:
  • [x] The limit position
  • [x] The distance between limits
  • [x] The time taken to move between limits
• [x] Move axes slightly off a limit, switch back to SECI and confirm positions are all the same
• [x] Rehome axes under SECI and confirm distances to limits are still the same

------- At this point we can say that motion under IBEX is the same as SECI, other than homing. At this point the beamline could be run with homing under SECI and all other motion under IBEX -----

• [x] Rehome axes under SECI
• [x] Switch to IBEX and rehome
• [x] Confirm all axes reach the same home position as SECI and stop there
• [ ] Note the difference for each axis as stated in https://github.com/ISISComputingGroup/EPICS-galil/pull/41. Convert this value to mm and confirm with scientists whether it's acceptable
• [ ] From home travel to both limit switches, confirm the distance and time taken is the same as in SECI

[Tom-Willemsen]

If we have spare time, also test https://github.com/ISISComputingGroup/IBEX/issues/5735 (in particular does the fast deceleration work correctly, or does it use a slow deceleration?). If we run out of time, add this to the next testing ticket.

[Tom-Willemsen]

Definition of zero migrated over. Old and new config files stored in \\isis\shares\ISIS_Experiment_Controls_Public\SANS2D front end galils in case they are needed for reference.

[DominicOram]

Data from SECI:

	Block 1 (G1)	Block 2 (G2)	Block 3 (G3)	Block 4 (G4)	Block 5 (G5)
Home position / mm	58.60	61.06	62.02	58.61	60.16
Position top LS engages / mm	283.840	284.840	284.840	284.060	284.880
Position bottom LS engages / mm	-16.400	-4.860	-14.920	-4.920	-4.840
Distance bewtween switches / mm	300.24	289.7	299.76	288.98	289.72
Distance from home to top LS / mm	225.24	223.78	222.82	225.45	224.72
Distance from home to bottom LS / mm	75	65.92	76.94	63.53	65
Time taken from LS to LS	59.96s	58.52s	61.19s	57.39s	58.19s
Is the top LS in SECI high or low on the Galil?	HIGH	HIGH	HIGH	HIGH	HIGH

	Plate 1 (S1)	Plate 2 (S2)	Plate 3 (S3)	Plate 4 (S4)	Plate 5 (S5)
Home position / mm	889.51	867.82	873.33	867.70	872.32
Position top LS engages / mm	955.425	984.882	981.563	976.245	978.227
Position bottom LS engages / mm	-8.534	-17.748	-20.977	-25.210	-21.694
Distance bewtween switches / mm	963.959	1002.63	1002.54	1001.455	999.921
Distance from home to top LS / mm	65.915	117.062	108.233	108.545	105.907
Distance from home to bottom LS / mm	898.044	885.568	894.307	892.91	894.014
Time taken from LS to LS	2min 32.41s	2min 39.38s	2min 38.41s	2min 39.51s	2min 38.11s
Is the top LS in SECI high or low on the Galil?	LOW	LOW	LOW	LOW	LOW

[DominicOram]

Data from IBEX:

	Block 1 (G1)	Block 2 (G2)	Block 3 (G3)	Block 4 (G4)	Block 5 (G5)
Home position / mm	58.60	61.06		58.61	60.16
Home position diff from SECI (see https://github.com/ISISComputingGroup/EPICS-galil/pull/41) / mm	-0.02	-0.02		-0.02	-0.02
Position top LS engages / mm	283.900	283.980	283.980	283.830	283.960
Position bottom LS engages / mm	-15.64	-8.800	-15.120	-8.470	-8.380
Distance between switches / mm	299.54	292.78		292.30	292.34
Distance from home to top LS / mm	225.3	222.92		225.22	223.8
Distance from home to bottom LS / mm	74.24	69.86		67.08	68.54
Time taken from LS to LS	1min00.71	57.93s	57.96s	58.91s	58.08
Is the top LS in IBEX high or low on the Galil?	HIGH	HIGH	HIGH	HIGH	HIGH

	Plate 1 (S1)	Plate 2 (S2)	Plate 3 (S3)	Plate 4 (S4)	Plate 5 (S5)
Home position / mm	889.493	867.796	873.296	867.693	872.304
Position top LS engages / mm	951.896	981.358	978.101	972.919	965.571
Home position diff from SECI (see https://github.com/ISISComputingGroup/EPICS-galil/pull/41) / mm	0.017	0.031	0.047	0.022	0.022
Position bottom LS engages / mm	-4.927	-17.354	-18.448	-24.662	-21.660
Distance between switches / mm	956.823	998.712	996.549	997.58	987.231
Distance from home to top LS / mm	62.403	113.562	104.805	105.226	93.267
Distance from home to bottom LS / mm	894.42	885.15	891.744	892.355	893.964
Time taken from LS to LS	2min31.21	2min38.18	2min37.49	2min37.24	2min36.86
Is the top LS in IBEX high or low on the Galil?	LOW	LOW	LOW	LOW	LOW

Other IBEX issues:

• [x] S4 behaves weirdly at the limit, it reaches the limit then moves off of it slightly This seems to be due to the blades falling under gravity, keeping them energised fixed this
• [x] G3 stalled (mechanical problem) Whilst a mechanical problem it may have been caused by guides staying energised
• [x] Limit switch not correct whilst homing (created https://github.com/ISISComputingGroup/IBEX/issues/5868)
• [x] Homing all the guides at the same time doesn't work (created https://github.com/ISISComputingGroup/IBEX/issues/5869)

[DominicOram]

To review:

• [ ] https://github.com/ISISComputingGroup/EPICS-ioc/pull/551
• [ ] https://github.com/ISISComputingGroup/EPICS-galil/pull/47
• [ ] Confirm numbers above match
• [ ] Confirm additional bits are covered by https://github.com/ISISComputingGroup/IBEX/issues/5882

[DominicOram]

When switching back and forth between SECI movements work and they report the same values. However, the homing routine used in IBEX does not work in SECI so when moving to sECI you also have to download the SECI homing routines. I have sent the instrument scientists the following:

Homing in SECI/IBEX

Whilst SECI and IBEX do the same steps to perform a home (and end up in the same place as we saw yesterday) the actual code doing the homing routines is slightly different so you will have to reapply the routines when moving from IBEX to SECI. Hopefully you won’t be doing this that often but if you would like this process to be smoother then we can try and automate it more, please let us know. This will also only be a problem if you want to home in SECI, you could home in IBEX then go from IBEX -> SECI and move everything as usual without rehoming.

Homing routines will need to be reapplied for all of the Galils. This will mean that if you switch from IBEX to SECI (using C:\Instrument\Apps\EPICS\stop_ibex_start_seci.bat) SECI will give you a number of warnings boxes that the homing routines are incorrect. Once you close these boxes SECI will start normally but before you home in SECI you will have to reapply the homing routines by doing the following:

1. Go to the table of motors VI
2. Press Ctrl + Shift + F10, more tabs should appear
3. Click the Program tab
4. Select a Galil on the left
5. Press load
6. Select the file that has the same name as the Galil you have selected
7. Press download
8. Repeat for all Galils in the list

[kjwoodsISIS]

Summary of Commissioning on SANS2D (as of 6th November 2020)

What have we achieved

1. We have successfully used IBEX to control the detector and baffle trolleys in the vacuum tank.
2. We have successfully used IBEX to control the positioning of the beam-stops & detectors on their trolleys.
3. We have successfully used IBEX to control the large jaws in the vacuum tank.
4. We have successfully used IBEX to move the plates in the front-end.
5. We have successfully used IBEX to move the guides in the front-end.
6. We have re-defined zeros in SECI to make it easier to switch between SECI & IBEX.
7. We have created a table of home positions for the plates in guides in SECI.
8. We have reproduced the table of home positions for the plates in guides using IBEX (apart from some very minor differences, which we don't believe are significant).

What remains to be done

We need to test the following

1. control of the sample changer
2. control of the sample positioner
3. control of the little jaws
4. control of the S6 aperture
5. live view (primarily a matter of configuration)
6. Fast deceleration in tank
7. Fast shutter on overcount
8. Named aperture/guide positions We believe that testing these will be straightforward - they are all very similar to other devices we know we can control. We also need to complete migration of some configurations & scripts.

Problems, Obstacles, Risks

The primary problem we have encountered is with the Baldor drives, which move the front-end guides. The guides are very heavy and the Baldors are operating at or near their limits. The Baldors have a tendency to stall and overheat. This tendency is present in SECI too; IBEX seems to make it worse. Unfortunately, we don't know what it is about IBEX that makes the Baldors more prone to stalling & overheating. [Can we ask Gareth for his advice?]

• On day 1 of testing, we were able to move the guides but encountered some stalls.
  We fixed this by not keeping the Baldors energised when stopped.
• On day 2 of testing, we were able to move the guides and encountered no stalls.
• On day 3 of testing, we were able to move the guides but we encountered frequent stalls on all the Baldors.
  Switching back to SECI did not resolve the problem, they continued to stall. We tried power cycling the Galils (which control the Baldors), just in case an incorrect parameter had been inadvertently set.
  This made no difference.
• Eventually, Baldors 1 & 4 started to behave again.
• Sarah, who has a lot of experience persuading the Baldors to behave, was able to use SECI to get Baldors 2 and 3 working again. But Baldor 5 continued to remain stalled.
• Jamie then rewired the amplifier card for Baldor 4 into Baldor 5. This was sufficient to move Baldor 5 into a position when it could remain for the coming cycle (Cycle 2020/03). He then re-connected Baldor 4 back to its amplifier card. Baldor 5 remains inoperable.

We would like to do more testing of IBEX with the Baldor drives, to see if we can resolve the stalling/overheating problem. However, there are no available Baldor drives (other than those in SANS2D) we can use for testing. There is a very high risk that further testing could damage the Baldors beyond repair before we find a solution.

Options for proceeding:

1. SANS2D continues to operate under SECI until we can be confident that that the Baldors are reliable in IBEX
   • the least risky option. We know it will work, because it works now.
   • it is likely that this will not happen before the long shutdown
2. SANS2D uses IBEX for cycle 2020/04, but we create lvDCOM components to act as proxies to control the Baldors (the lvDCOM components would relay command from IBEX to LabVIEW VIs (the same ones used in SECI))
   • a possible temporary solution
3. If the Baldors are moved only infrequently, use SECI to move the Baldors. Then switch to IBEX to run experiments.
   • another possible temporary solution, but potentially very inconvenient
4. Replace the Baldors with alternative drives
   • there is already a proposal to do this; it may need to be brought forward
   • this is not a quick solution; it can't happen before the long shutdown
   • if this project goes ahead, then we need to think whether SANS2D should be re-commissioned using just IBEX or both SECI and IBEX. If we re-commission with just IBEX, there is no path back to SECI. If we re-commission with both, that’s a lot of extra work.