All pixel values are zero

[LeeYangLBLBCS]

IOC shell shows the following error at start up:
=================================================
#!../../bin/linux-x86_64/kinetixApp
< /epics/common/xf31id1-lab3-ioc1-netsetup.cmd
**Can't open /epics/common/xf31id1-lab3-ioc1-netsetup.cmd: No such file or directory**
errlogInit(20000)
< envPaths
==========================================
Apparently this file  /epics/common/xf31id1-lab3-ioc1-netsetup.cmd  is only available in developer's own environment.
I don't know if that's causing all image results with 0 pixels though.
The first few lines of the IOC shell start up are attached below:
==========================================================
../../bin/linux-x86_64/kinetixApp st.cmd
#!../../bin/linux-x86_64/kinetixApp
< /epics/common/xf31id1-lab3-ioc1-netsetup.cmd
Can't open /epics/common/xf31id1-lab3-ioc1-netsetup.cmd: No such file or directory
errlogInit(20000)
< envPaths
epicsEnvSet("IOC","iocKinetix")
epicsEnvSet("TOP","/opt/epics/modules/synApps_6_1_epics7/support/areaDetector-R3-7/ADKinetix/iocs/kinetixIOC")
epicsEnvSet("ADKINETIX","/opt/epics/modules/synApps_6_1_epics7/support/areaDetector-R3-7/ADKinetix/iocs/kinetixIOC/../..")
epicsEnvSet("SUPPORT","/opt/epics/base-7.0.4/../modules/synApps_6_1_epics7/support")
epicsEnvSet("ASYN","/opt/epics/base-7.0.4/../modules/synApps_6_1_epics7/support/asyn-R4-44-2")
epicsEnvSet("AREA_DETECTOR","/opt/epics/base-7.0.4/../modules/synApps_6_1_epics7/support/areaDetector-R3-7")
epicsEnvSet("ADSUPPORT","/opt/epics/base-7.0.4/../modules/synApps_6_1_epics7/support/areaDetector-R3-7/ADSupport")
epicsEnvSet("ADCORE","/opt/epics/base-7.0.4/../modules/synApps_6_1_epics7/support/areaDetector-R3-7/ADCore")
epicsEnvSet("AUTOSAVE","/opt/epics/base-7.0.4/../modules/synApps_6_1_epics7/support/autosave-R5-10")
epicsEnvSet("BUSY","/opt/epics/base-7.0.4/../modules/synApps_6_1_epics7/support/busy-R1-7-2")
epicsEnvSet("CALC","/opt/epics/base-7.0.4/../modules/synApps_6_1_epics7/support/calc-R3-7-4")
epicsEnvSet("SNCSEQ","/opt/epics/base-7.0.4/../modules/synApps_6_1_epics7/support/seq-2-2-9")
epicsEnvSet("SSCAN","/opt/epics/base-7.0.4/../modules/synApps_6_1_epics7/support/sscan-R2-11-3")
epicsEnvSet("DEVIOCSTATS","/opt/epics/base-7.0.4/../modules/synApps_6_1_epics7/support/iocStats-3-1-16")
epicsEnvSet("STD","/opt/epics/base-7.0.4/../modules/synApps_6_1_epics7/support/std-R3-6-1")
epicsEnvSet("EPICS_BASE","/opt/epics/base-7.0.4")
dbLoadDatabase("/opt/epics/modules/synApps_6_1_epics7/support/areaDetector-R3-7/ADKinetix/iocs/kinetixIOC/dbd/kinetixApp.dbd")
kinetixApp_registerRecordDeviceDriver(pdbbase)
# Prefix for all records
# epicsEnvSet("PREFIX", "XF:31ID1-ES{Kinetix-Det:1}")
epicsEnvSet("PREFIX", "OML1:")
# The port name for the detector
epicsEnvSet("PORT",   "KTX")
# The queue size for all plugins
epicsEnvSet("QSIZE",  "20")
# The maximim image width; used for row profiles in the NDPluginStats plugin
epicsEnvSet("XSIZE",  "3200")
# The maximim image height; used for column profiles in the NDPluginStats plugin
epicsEnvSet("YSIZE",  "3200")
# The maximum number of time series points in the NDPluginStats plugin
epicsEnvSet("NCHANS", "2048")
# The maximum number of frames buffered in the NDPluginCircularBuff plugin
epicsEnvSet("CBUFFS", "500")
# The search path for database files
epicsEnvSet("EPICS_DB_INCLUDE_PATH", "/opt/epics/base-7.0.4/../modules/synApps_6_1_epics7/support/areaDetector-R3-7/ADCore/db")
ADKinetixConfig(0, "KTX", 3200, 3200, 3, 0, 0)
2024/04/11 10:27:53.216 ADKinetix::ADKinetix: Opened camera...
2024/04/11 10:27:53.218 ADKinetix::ADKinetix: Model: Kinetix | Resolution: 3200x3200
2024/04/11 10:27:53.218 ADKinetix::ADKinetix: Configuring default region to full sensor size...
2024/04/11 10:27:53.218 ADKinetix::updateCameraRegion: Configured Region: (0, 0) to (3199, 3199) | Binning: (1, 1)
  Speed table:
  - port 'Sensitivity', value 0
    - speed index 0, running at 100.000000 MHz
      - gain index 1, 'Standard', bit-depth 12 bpp
  - port 'Speed', value 1
    - speed index 0, running at 200.000000 MHz
      - gain index 1, 'Standard', bit-depth 8 bpp
  - port 'Dynamic Range', value 2
    - speed index 0, running at 100.000000 MHz
      - gain index 1, 'Standard', bit-depth 16 bpp
  - port 'Sub-Electron', value 3
    - speed index 0, running at 100.000000 MHz
      - gain index 1, 'Standard', bit-depth 16 bpp

2024/04/11 10:27:54.379 ADKinetix::selectSpeedTableMode: Readout port set to 'Sensitivity'
2024/04/11 10:27:54.480 ADKinetix::selectSpeedTableMode: Readout speed set to 10 ns per pix
2024/04/11 10:27:54.482 ADKinetix::selectSpeedTableMode: Gain set to Standard
2024/04/11 10:27:54.482 ADKinetix::ADKinetix: Registered EOF callback function.
2024/04/11 10:27:54.482 ADKinetix::ADKinetix: Spawning camera monitor thread...
dbLoadRecords("/opt/epics/modules/synApps_6_1_epics7/support/areaDetector-R3-7/ADKinetix/iocs/kinetixIOC/../../db/ADKinetix.template", "P=OML1:,R=cam1:,PORT=KTX,ADDR=0,TIMEOUT=1")
2024/04/11 10:27:54.482 ADKinetix::monitorThread: Temperature monitor thread active.
# Create a standard arrays plugin, set it to get data from pvCamera driver.
NDStdArraysConfigure("Image1", 1, 0, "KTX", 0, 10000000)
........

[jwlodek]

The error at the top is regarding a configuration file we use at NSLS-II to restrict the network interface EPICS uses to broadcast UDP messages. It can safely be removed.

In the pvcam sdk, you should have an example LiveImageCallbacks. When you run this example do you also see all zeros? Do you have a lens installed or is the camera set up with the lens cover on? I think when I was initially testing with the lens cover on we saw all zeros.

[LeeYangLBLBCS]

The vendor test program, PVCamTest works fine, no 0 pixel problem. I attached the MEDM screen for information, no obvious problem I can see on the screen though.

[LeeYangLBLBCS]

Does the following error from "ImageJ" log indicate problems with camera interface IO?

11/4/2024 10:59:12.74: UpdateImage got exception: pendIO timed out 11/4/2024 10:59:23.78: UpdateImage got exception: pendIO timed out 11/4/2024 10:59:34.73: UpdateImage got exception: pendIO timed out 11/4/2024 10:59:45.75: UpdateImage got exception: pendIO timed out 11/4/2024 10:59:56.79: UpdateImage got exception: pendIO timed out 11/4/2024 11:0:7.81: UpdateImage got exception: pendIO timed out 11/4/2024 11:0:18.78: UpdateImage got exception: pendIO timed out 11/4/2024 11:0:29.77: UpdateImage got exception: pendIO timed out 11/4/2024 11:0:40.78: UpdateImage got exception: pendIO timed out 11/4/2024 11:0:50.83: UpdateImage got exception: pendIO timed out 11/4/2024 11:1:1.78: UpdateImage got exception: pendIO timed out 11/4/2024 11:1:11.873: Image display stopped 11/4/2024 11:1:12.80: UpdateImage got exception: pendIO timed out 11/4/2024 11:1:13.35: Image display started 11/4/2024 11:1:23.78: UpdateImage got exception: pendIO timed out 11/4/2024 11:1:33.373: UpdateImage got exception: pendIO timed out

[jwlodek]

Are you saving the images through imageJ? I don't use the imagej viewer much so I do not recognize those errors. We can check if the IOC is getting non-zero values by adding a print above this line: https://github.com/NSLS-II/ADKinetix/blob/259526b9eeef906ae885fd11a9f48888edef2717/kinetixApp/src/ADKinetix.cpp#L907

to print the first few pixels of the image.

[LeeYangLBLBCS]

I'm not trying to save images when these error occurred. These ImageJ errors also correlate to the following IOC output: 2024/04/11 12:07:40.294 ADKinetix::acquisitionThread: Timeout waiting for next frame, trying again! 2024/04/11 12:07:40.605 ADKinetix::writeInt32: function=62, value=0.000000 Camera 0 timed out waiting for a frame 2024/04/11 12:07:41.297 ADKinetix::acquisitionThread: Timeout waiting for next frame, trying again! 2024/04/11 12:07:41.605 ADKinetix::writeInt32: function=62, value=0.000000 Camera 0 timed out waiting for a frame 2024/04/11 12:07:42.297 ADKinetix::acquisitionThread: Timeout waiting for next frame, trying again! 2024/04/11 12:07:42.605 ADKinetix::writeInt32: function=62, value=0.000000 Camera 0 timed out waiting for a frame 2024/04/11 12:07:43.295 ADKinetix::acquisitionThread: Timeout waiting for next frame, trying again! 2024/04/11 12:07:43.605 ADKinetix::writeInt32: function=62, value=0.000000 2024/04/11 12:07:43.687 ADKinetix::acquireStop: Waiting for acquisition thread to join... Processing aborted on camera 0 2024/04/11 12:07:43.795 ADKinetix::acquisitionThread: Timeout waiting for next frame, trying again! 2024/04/11 12:07:43.795 ADKinetix::acquisitionThread: Acquisition done. Freeing framebuffers. 2024/04/11 12:07:43.826 ADKinetix::acquireStop: Acquisition stopped.

The CommonPlugins screen shows only the "NDPluginStdArrays" is Enabled:

[LeeYangLBLBCS]

After I plugged in the "Dolphin" PCIe card, the image looks normal. https://www.dolphinics.com/download/PX/OPEN_DOC/PXH832_users_guide.pdf I left the card out initially, because I thought it was optional. Their own test program, PVCamTest, work fine without the extra card though.

[jwlodek]

Interesting, so you are using the USB connection option? I had only tested with the PCI card option.

[LeeYangLBLBCS]

Yes, I've using the USB connection only. The PCIe cable does not work. I'm still talking to Teledyne support to find out why. I also found out that the reason it suddenly worked with the USB port is because I have changed the "Region size" from 3200x3200 to 320x320. It appeared there were some kind of throughput issue such that larger ROI can not be acquired. In addition, I noticed the frame rate setting in the MEDM screen has no effect on the actual acquisition rate.

[jwlodek]

Yes, the AcquirePeriod PV does not have an effect currently. This is because the way that the PVCam SDK handles acquisitions it only accepts a single (exposure time) value to regulate framerate. I tried doing something myself with sleeps, but the camera continues to acquire in the background when you do this, so the framerate becomes somewhat irregular. I asked Teledyne on how I could implement this and waiting to hear back.

For now we are using the edge trigger mode to regulate framerate with low exposure times.

As for why it didn't work at high resolutions - I think Teledyne outlines two different operating modes (Imaging/Monitoring or something like that), and if the connection speed is lower than some value that qualifies the camera for imaging mode, some higher resolutions and speeds are disabled in firmware. Likely this is what you were seeing.

[LeeYangLBLBCS]

Thanks for explaining. If the exposure time is the only factor to impact the frame rate, does it mean the external trigger’s periodicity will not work correctly?

On Fri, Apr 12, 2024 at 7:24 AM Jakub Wlodek @.***> wrote:

Yes, the AcquirePeriod PV does not have an effect currently. This is because the way that the PVCam SDK handles acquisitions it only accepts a single (exposure time) value to regulate framerate. I tried doing something myself with sleeps, but the camera continues to acquire in the background when you do this, so the framerate becomes somewhat irregular. I asked Teledyne on how I could implement this and waiting to hear back.

For now we are using the edge trigger mode to regulate framerate with low exposure times.

As for why it didn't work at high resolutions - I think Teledyne outlines two different operating modes (Imaging/Monitoring or something like that), and if the connection speed is lower than some value that qualifies the camera for imaging mode, some higher resolutions and speeds are disabled in firmware. Likely this is what you were seeing.

— Reply to this email directly, view it on GitHub https://github.com/NSLS-II/ADKinetix/issues/1#issuecomment-2051862867, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADSYGNEDDCEQUNHK7DJYV53Y47VCXAVCNFSM6AAAAABGCW7HK6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANJRHA3DEOBWG4 . You are receiving this because you authored the thread.Message ID: @.***>

[jwlodek]

It works fine with an external trigger, since the camera waits for the external TTL signal to collect the next frame. Only the internal trigger does not seem to have an option to control time between pulses, and will just collect a new frame as soon as exposure is completed.

[LeeYangLBLBCS]

Is the camera index in the first arguement for selecting one of the two cameras, if both USB and PCIe ports are plugged in? camera 1: ADKinetixConfig(0, "$(PORT)", $(XSIZE), $(YSIZE), 3, 0, 0) vs camera2: ADKinetixConfig(1, "$(PORT)", $(XSIZE), $(YSIZE), 3, 0, 0)

[LeeYangLBLBCS]

What's the largest image size can you acquire without error? For region greater than 2000x2000, I always get 0 pixels starting at row 2000. This problem does not change when I change the pixel data size, such as between Uint16 and Uint8. There is also no problem with vendor's test program PVCamTest, so I can't get any help from Teledyne support on this issue.

[jwlodek]

I am able to acquire full frame (3200x3200) images, but I have not tested over USB. Also yes, the deviceIndex is used to select between multiple cameras. I have tested with running two on the same server with the same PCI card from dolphin.

[LeeYangLBLBCS]

sorry, I forgot to mention that I have switched to using PCIe cable since last week. The problem I described is with the PCIe port. USB port is disconnected to make things simple.

[jwlodek]

Working in-person at ALS last week we were able to confirm full (3200x3200) resolution acquisition. We are not sure what the issue was with the zero pixel values