Minimal functionality when using reset after power cycling camera

[GDYendell]

Hi, @MarkRivers !

I have found that when we power cycle a GigE camera, the reset process seems to work and it can acquire frames, but most of the parameters do not work. They get reset to 0 / No / N.A., so we can't see what the values on the camera are, and setting things has no effect, for example AcquireTime, although the readbacks update in the IOC.

Is this something you have seen before?

We are running a version based on R2-1

[MarkRivers]

I don't believe I have tested power-cycling the camera without restarting the IOC.

I will try it soon. It seems strange that you can't set parameters.

In terms of restoring the current settings to the device after a power-cycle, it is a more general problem in EPICS, not just restricted to GigE cameras or ADAravis. When a device is power-cycled without rebooting the IOC we need to push the most recent EPICS PV values down to the device. I have been thinking that this is probably something we could use the autosave module for. The autosave file is designed to have all of the important settings that need to be pushed to the device to restore operation on an IOC reboot. It seems like we could just use autosave to read that file (before it gets corrupted with the new values) and push to the device without doing a reboot.

I believe that Michael Davidsaver's branch of aravisGigE had a fix for rebooting GigE cameras, but specific to that driver. I want a more general solution.

[JimboMonkey]

Hello,

I have the same problem with a Prosilica camera (the reconnect button has no effect on a disconnected camera), and additionally, if reconnect is pressed whilst the camera is connected, it locks up the camera parameters. Images are still streamed, but parameter changes no longer have an effect. When this happens the following error message appears in the IOC console:

(process:29661): GLib-GObject-^[[1;33mWARNING^[[0m **: ^[[34m08:47:17.193^[[0m: invalid unclassed pointer in cast to 'ArvGcFeatureNode'
** (process:29661): ^[[1;35mCRITICAL^[[0m **: ^[[34m08:47:17.193^[[0m: arv_gc_feature_node_is_available: assertion 'ARV_IS_GC_FEATURE_NODE (gc_feature_node)' failed

I can confirm that the reconnect functionality works with the same camera when using the aravisGigE module.

Thanks,

James

[MarkRivers]

I can confirm that the reconnect functionality works with the same camera when using the aravisGigE module.

What version of the aravisGigE module are you using? Is it the master branch or one of the releases in https://github.com/areaDetector/aravisGigE, or is it a branch from Michael Davidsaver?

[JimboMonkey]

It appears to be a very old version from March 2015. It is using a Diamond release number (0-3-5dls6) so I am not sure how well it maps to the versions from GitHub. Looking in the git log, the merges seem to come from the master branch.

[MarkRivers]

I have changed the master branch of ADAravis and ADCore so that one can now do the following without restarting the IOC:

• Power-cycle the camera
• Reconnect the ADAravis driver to the camera
• Download all of the current EPICS settings to the camera so that it operates the same as before the power-cycle

ADAravis required the following changes:

• Added arvFeature::initialize() and an std::vector of arvFeatures to allow re-initializing when the camera reconnects.
• Add autosave command create_manual_set(ADAutoSaveMenu.req, ...) to iocBoot/iocAravis/st.cmd.base.
  • This can be used to save and load the camera and plugin PV when the camera reconnects.
  • It can also be used to save and load different configurations for different setups.

ADCore required the following changes:

• New file ADAutoSaveMenu.req to be used with the configMenu utility in autosave.
  • It saves all of the camera and plugin settings defined in iocBoot/iocXXX/auto_settings.req.
  • This allows manually saving and loading different configurations for different setups with the same IOC.
  • It also allows saving and then loading the settings when a camera is reset or restarted, without restarting the IOC.
• Modified ADPlugins OPI sub-screen.
  • Added a related display "Plugins/Other 2/AS configMenu" which allows manually saving and loading configurations that use ADAutoSaveMenu.req described above.

The following steps are needed to do this:

• Power cycle the camera
• Wait a few seconds for the camera to finish initializing, and then press the "Reset camera" button on this screen:

• At this point ADAravis can control the camera again. However, many camera settings will no longer match the EPICS settings.
• Open this screen from "ADAravis/Plugins/Other 2/AS configMenu"

• On this screen you can name different configurations you want to save, in this case just test1 and test2.
• Press the Save button for the configuration name you want to save. This will save all of the camera and plugin settings to a file called ADAutoSave_test1.cfg. It will also create a backup of the current settings before creating the new file with a name like ADAutoSave_test1.cfg_210628-133135, where the end of the file name is the current date and time.
• Press the Load button for the same configuration you just saved. This will download all of the EPICS camera settings to the camera. It will also set all of the plugin settings to the same values you just loaded, so they should not actually change.
• This tool can also be used to save multiple configurations with different settings, for example different binning or camera orientation with NDPluginTransform. It provides a convenient way to then reload these configurations at a later date.

[MarkRivers]

There are some improvements that should be done.

• A sequence record can be used to automatically do these steps by effectively doing the following:

  • Press the Reset camera button
  • Wait for a fixed time
  • Set the configuration file name
  • Press the configMenu Save button and wait for it to complete
  • Press the configMenu Load button and wait for it to complete

• The ADAravis driver uses the driver-specific ARResetCamera record and parameter to reconnect.

  • This is not needed. asynPortDriver already provides the connect() and disconnect() methods.
  • All drivers should implement these methods.
  • disconnect() will disconnect the camera from the driver
  • connect() will connect the camera to the driver

• All of the OPI screens already contain connect() and disconnect() widgets that will result in the disconnect() and connect() methods being called.

[GDYendell]

Thanks @MarkRivers this is working great! The beamline and can now move their cameras around as they please and then reconnect in the IOC with no issues.

Using autosave to restore settings on reconnect is a nice addition, but just being able to reconnect is a already huge UX improvement in our case.

[devilkingsatan666]

I have changed the master branch of ADAravis and ADCore so that one can now do the following without restarting the IOC:

• Power-cycle the camera
• Reconnect the ADAravis driver to the camera
• Download all of the current EPICS settings to the camera so that it operates the same as before the power-cycle

ADAravis required the following changes:

• Added arvFeature::initialize() and an std::vector of arvFeatures to allow re-initializing when the camera reconnects.

• Add autosave command create_manual_set(ADAutoSaveMenu.req, ...) to iocBoot/iocAravis/st.cmd.base.

  • This can be used to save and load the camera and plugin PV when the camera reconnects.
  • It can also be used to save and load different configurations for different setups.

ADCore required the following changes:

• New file ADAutoSaveMenu.req to be used with the configMenu utility in autosave.

  • It saves all of the camera and plugin settings defined in iocBoot/iocXXX/auto_settings.req.
  • This allows manually saving and loading different configurations for different setups with the same IOC.
  • It also allows saving and then loading the settings when a camera is reset or restarted, without restarting the IOC.

• Modified ADPlugins OPI sub-screen.

  • Added a related display "Plugins/Other 2/AS configMenu" which allows manually saving and loading configurations that use ADAutoSaveMenu.req described above.

The following steps are needed to do this:

• Power cycle the camera
• Wait a few seconds for the camera to finish initializing, and then press the "Reset camera" button on this screen:

• At this point ADAravis can control the camera again. However, many camera settings will no longer match the EPICS settings.
• Open this screen from "ADAravis/Plugins/Other 2/AS configMenu"

• On this screen you can name different configurations you want to save, in this case just test1 and test2.
• Press the Save button for the configuration name you want to save. This will save all of the camera and plugin settings to a file called ADAutoSave_test1.cfg. It will also create a backup of the current settings before creating the new file with a name like ADAutoSave_test1.cfg_210628-133135, where the end of the file name is the current date and time.
• Press the Load button for the same configuration you just saved. This will download all of the EPICS camera settings to the camera. It will also set all of the plugin settings to the same values you just loaded, so they should not actually change.
• This tool can also be used to save multiple configurations with different settings, for example different binning or camera orientation with NDPluginTransform. It provides a convenient way to then reload these configurations at a later date.

Dear Mark, It seems that you directly used the configMenu adl file shipped with autosave, but there are no database-related changes made. Should people add lines like the following in the commonPlugins.cmd file of ADCore? dbLoadRecords("$(AUTOSAVE)/db/configMenu.db","P=$(PREFIX),CONFIG=ADAutoSave,ENABLE_SAVE=0")

Might it be a good idea to include "configMenu_settings.req" in "ADAutoSaveMenu.req"?

Best regards, Wu-Cheng Chiang

[MarkRivers]

Should people add lines like the following in the commonPlugins.cmd file of ADCore? dbLoadRecords("$(AUTOSAVE)/db/configMenu.db","P=$(PREFIX),CONFIG=ADAutoSave,ENABLE_SAVE=0")

@devilkingsatan666 thanks for pointing that out. I had added configMenu.db to my local commonPlugins.cmd, but not to EXAMPLE_commonPlugins.cmd. I have now added that and made some other minor fixes and pushed to the master branch.

Might it be a good idea to include "configMenu_settings.req" in "ADAutoSaveMenu.req"?

I was including configMenu.req rather than configMenu_settings.req in ADAutoSaveMenu.req. This is the difference between those files:

corvette:~/devel/autosave/db>diff -U2 configMenu.req configMenu_settings.req
--- configMenu.req      2021-03-12 14:38:47.823241034 -0600
+++ configMenu_settings.req     2021-03-12 14:38:47.818241066 -0600
@@ -1,2 +1,5 @@
 $(P)$(CONFIG)Menu:currName
 $(P)$(CONFIG)Menu:currDesc
+$(P)$(CONFIG)Menu:enableSave
+# configMenu will search for .cfg files, so no need to autosave them
+#file configMenuNames.req P=$(P),CONFIG=$(CONFIG)

So the difference is that configMenu_settings.req also saves the enableSave PV, while configMenu.req does not. Since that PV is present and exposed in the OPI "more" screen I have changed ADAutoSaveMenu.req to include configMenu_settings.req instead.

[MarkRivers]

This issue is resolved, closing it.