search

epics-modules / dxp

EPICS support for digital x-ray spectroscopy electronics from X-ray Instrumentation Associates (XIA)
2 stars 3 forks source link

Some errors when using dxp-R6-1 module on ketek SDD detector #10

Open 1458861693 opened 11 months ago

1458861693 commented 11 months ago

There is some errors when using dxp-R6-1 module on ketek SDD detector，the following is the information of my environment. operating system ： CentOS7 architecture ： X86_64 iocBoot Directory ： /home/xanes/softIOC/support/dxp-R6-1/iocBoot/iocMicroDXP When i start IOC，there are many errors as log.txt in attached files，i really want some help，Thanks. The attached files contain log.txt，st.cmd and START_IOC，log.txt is printed information by IOCShell. log.txt st.cmd.txt START_IOC.txt

1458861693 commented 11 months ago

I don't determine which IOC should i use，may be iocMicroDXP？ I have restarted the IOC，the printed error of iocshell seems to be different. As this figure. image Looking forward to your reply.

1458861693 commented 11 months ago

when i use the prospect software on Windows10，the detector works OK. The prospect software also load KETEK DPP2_usb2.ini file，which located in $(TOP)/iocBoot/iocMicroDXP on centos7.

MarkRivers commented 11 months ago

Have you set up udev correctly to assign world rw permission to that USB device on Linux?

Please send the output of the following commands:

lsusb

This is the output for one of my systems where I have Measurement Computing USB modules installed and using udev. Note that the Measurement Computing devices are on bus 001, devices 006, 005, 004.

[epics@13idc ~]$ lsusb
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 009: ID 1604:10c0 Tascam Dell Integrated Hub
Bus 001 Device 008: ID 413c:a102 Dell Computer Corp. iDRAC Virtual NIC
Bus 001 Device 007: ID 1604:10c0 Tascam Dell Integrated Hub
Bus 001 Device 003: ID 1604:10c0 Tascam Dell Integrated Hub
Bus 001 Device 006: ID 09db:013e Measurement Computing Corp. USB-1808X
Bus 001 Device 005: ID 09db:009d Measurement Computing Corp. USB-3104
Bus 001 Device 004: ID 09db:0127 Measurement Computing Corp. USB-CTR08
Bus 001 Device 002: ID 0409:005a NEC Corp. HighSpeed Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

lsusb will show you what USB bus the Ketek is using. Then show the permissions using the following command, but perhaps changing 001 to another number depending on what lsusb showed.

[epics@13idc ~]$ ls -l /dev/bus/usb/001
total 0
crw-rw-r-- 1 root root 189, 0 Oct 31 13:28 001
crw-rw-r-- 1 root root 189, 1 Oct 31 13:28 002
crw-rw-r-- 1 root root 189, 2 Oct 31 13:28 003
crw-rw-rw- 1 root adm  189, 3 Dec 21 08:55 004
crw-rw-rw- 1 root adm  189, 4 Dec 21 08:55 005
crw-rw-rw- 1 root adm  189, 5 Dec 21 08:55 006
crw-rw-r-- 1 root root 189, 6 Oct 31 13:28 007
crw-rw-r-- 1 root root 189, 7 Oct 31 13:28 008
crw-rw-r-- 1 root root 189, 8 Oct 31 13:28 009

Note that devices 004, 005, and 006 have world rw permission because of my udev configuration file.

1458861693 commented 11 months ago

Thanks for your reply. This is the output information when i type lsusb command in the terminal.

[xanes@localhost iocMicroDXP]$ lsusb
Bus 002 Device 002: ID 0bda:0328 Realtek Semiconductor Corp. 
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 008: ID 03f0:1f4a HP, Inc 
Bus 001 Device 002: ID 03f0:2f4a HP, Inc 
Bus 001 Device 012: ID 20bd:0020  
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
[xanes@localhost iocMicroDXP]$ ls -lh /dev/bus/usb/001/
total 0
crw-rw-r--. 1 root root 189,  0 Dec 21 08:56 001
crw-rw-r--. 1 root root 189,  1 Dec 21 08:56 002
crw-rw-r--. 1 root root 189,  7 Dec 21 08:56 008
crw-rw-rw-. 1 root root 189, 11 Dec 21 12:29 012
[xanes@localhost iocMicroDXP]$

The usb device for ketek detector is on bus 001, device 012, which has rw permission for all people.

MarkRivers commented 11 months ago

I have sent a message to the XIA support team requesting their help with this. This is my message to them

Folks,

I have an EPICS user who is getting errors when trying to run a Ketek MicroDXP on Linux.

I am trying to understand how xia_usb2_open() can return -16, since it only seems to return positive integer error numbers?

He is running the latest EPICS dxp release (R6-1), which uses Handel 1.2.28.

I believe I have another user at the APS who has seen similar problems.

Thanks, Mark

1458861693 commented 11 months ago

Thanks for your help. Looking forward to your reply.

1458861693 commented 11 months ago

Looking forward to your reply. Merry Christmas and a Happy New Year!

1458861693 commented 11 months ago

May be libusb library or the ini file KETEK_DPP2_USB2.ini cause this error？ Looking forward to your reply very much.

1458861693 commented 11 months ago

I want to know which version of libusb that I should use. I have tried different versions of libusb，for example libusb-0.1.12、libusb-1.0.21 with libusb-compat-0.1.8 and others，but this doesn't solve problem. May be other reasons cause this error？ I also commit the issue to libusb team，the link is above. I really wish you can give me more help to solve this problem. Thanks very much.

MarkRivers commented 11 months ago

I think that we need XIA to tell us what the problem could be. I don’t think the problem is the .ini file or the version of the USB library.

MarkRivers commented 11 months ago

Most people in the US are on vacation this week so we may have to wait until after Jan. 2 for an answer.

1458861693 commented 11 months ago

OK，I'm sorry，thanks again for your reply. Best wishes for vacation.

1458861693 commented 11 months ago

Happy new year！ Could you give me some help to solve this problem，it has been bothering me for a long time. Looking forward to your reply！

MarkRivers commented 10 months ago

I asked XIA if they have had a chance to look at this issue yet. They told me today (Jan. 5) they had not, but would try to look at it soon.

1458861693 commented 10 months ago

OK，Thanks for your help very much. Please tell me the newest reply if they look at this issue. Thanks again.

xia-stan commented 10 months ago

Just wanted to confirm that we're looking into the issue and will update both here and our support ticket when we have feedback.

xia-stan commented 10 months ago

I started with the log file. I wanted to confirm the operating system is CentOS 7. The log file name listed in the log says Ubuntu5. I suspect this is just a naming hold over from a previous system but want to be sure. My recommendations assume that this is just a file name issue.

The first logged error is Error 403, which is an MD layer error. The next error is 4002, indicating that there was a communications error with the device. Those errors tend to be related. I also see error 4202, which indicates that the device is trying to set a DSP parameter that the firmware doesn’t know about. Whether or not this is because of the initial communication issue, I can’t be sure at this time.

Based on what I’ve found so far, I recommend the following course of action.

  1. Verify that the device plugged into a computer port that supports USB 2.0+. Or try a different port if the current one does support USB 2.0+
  2. Update to the latest version of Handel 1.2.30 and verify that the issue persists.

I am working with the team to setup a test system to tray and reproduce the issue. In the mean time, could you please provide the handel ini file so that we can ensure we're using the same configuration?

1458861693 commented 10 months ago

First，thanks for your help very much. The operating system that i use is CentOS7. This is the output information when type 'uname -a' command.

[xanes@localhost iocMicroDXP]$ cat /etc/redhat-release
CentOs Linux release 7.9.2009(Core)
[xanes@localhost ~]$ uname -a
Linux localhost.localdomain 3.10.0-1160.105.1.el7.x86_64 #1 SMP Thu Dec 7 15:39:45 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

I have tried several different USB port including USB2.0 and USB3.0，but it is also wrong. The ini file(KETEK_DPP2_usb2.ini) that i have used in iocMicroDXP is uploaded as an attachment(named KETEK_DPP2_usb2_ini.txt). KETEK_DPP2_usb2_ini.txt The Silicon Drift Detector System that I have used is Ketek AXAS-M2，which belongs to MicroDXP？ How I update the handle library to the newest version in CentOS7 System，the current handle library is obtained by build the epics dxp-R6-1 module. I find handel.dll and handel.lib library in handel-microdxp-1.2.30-x64.zip file，which is downloaded by this link： https://github.com/xiallc/Handel-Releases/releases/tag/1.2.30, but i don't know how to use it in CentOS7，could you help me. Thanks again for your reply.

MarkRivers commented 10 months ago

I can load Handel 1.2.30 source code into the master branch of the EPICS dxp module tomorrow and you can build it from source for Centos7.

1458861693 commented 10 months ago

OK thanks. I will try it tomorrow. May i get the Handel 1.2.30 source code from this link？ https://github.com/xiallc/Handel-Releases/releases/tag/1.2.30 but I don't know how to build Handel library from this source code，i will try it again.

MarkRivers commented 10 months ago

@1458861693 I have updated this module to have the handel 1.2.30 source code. You should pull the master branch and rebuild this module. I have tested that it builds OK on 64-bit Linux and Windows. I have not tested whether it works.

1458861693 commented 10 months ago

Thanks for your help very much. I have tested the master branch and rebuild dxp module just now，but it still doesn't work. The attached file is new log file. Thanks again. log.txt

MarkRivers commented 10 months ago

I think there is a Ketek detector with MicroDXP at another beamline here at the APS. I am trying to borrow it so I can test to see if I also have a problem. I can test with CentOS 7 and CentOS 9.

MarkRivers commented 10 months ago

I just tried with a Hitachi Vortex MicroDXP that I borrowed from Soenke Seifert at APS sector 12.

I have a dual-boot system with Windows 10 and Centos 9. When I boot Windows and run the latest Prospect software it works fine.

I then boot Centos 9, with the MicroDXP plugged into the same USB port. When I start EPICS it fails, with very similar messages to the original report in this issue.

xiaSetLogLevel(2)
#xiaSetLogLevel(4)
#xiaSetLogOutput("Handel_MicroDXP_Ubuntu5.txt")
# Edit the .ini file to select the desired USB device number and other settings
xiaInit("KETEK_DPP2_usb2.ini")
[WARN ] 2024-01-09 17:36:31,951 xiaFindEntryLimits (handel_file.c:720)                   : Unable to find section default definitions
[WARN ] 2024-01-09 17:36:31,951 xiaReadIniFile (handel_file.c:421)                       : Section missing from ini file: default definitions
xiaStartSystem
[ERROR] 2024-01-09 17:36:37,240 xia_usb2_read (xia_usb_linux.c:415)                      : [ 403] usb_bulk_read error, driver reports: -110
[ERROR] 2024-01-09 17:36:37,240 dxp_md_usb2_io (md_linux.c:1343)                         : [4002] Error reading 32 bytes from 0x1000000 for camChan 0, driver reports 6
[ERROR] 2024-01-09 17:36:37,240 dxp_read_response (udxp_common.c:685)                    : [4002] Error reading response from ioChan 0
[ERROR] 2024-01-09 17:36:37,240 dxp__update_version_cache (udxp_common.c:925)            : [4002] Error reading response to board information command for getting the variant for ioChan 0
[ERROR] 2024-01-09 17:36:37,240 dxp_command (udxp_common.c:210)                          : [4002] Error updating the variant cache for ioChan 0
[ERROR] 2024-01-09 17:36:37,240 dxp_init_dspparams (udxp.c:417)                          : [4002] Error reading number of DSP parameters for ioChan 0
[ERROR] 2024-01-09 17:36:37,240 dxp_download_dspconfig (udxp.c:299)                      : [4002] Error initializing DSP parameters for ioChan 0
MarkRivers commented 10 months ago

I should add that I am running Handel 1.2.30 (latest).

I am using the following .ini file. I have added the .txt extension to allow it to upload to Github. KETEK_DPP2_usb2.ini.txt

MarkRivers commented 10 months ago

I just swapped the MicroDXP with a Saturn, serial number 1232. I am using the same USB cable and port.

It works fine with EPICS on CentOS 9. There are no errors during initialization, I can start/stop acquisition, etc.

# Initialize the XIA software
# Set logging level (1=ERROR, 2=WARNING, 3=XXX, 4=DEBUG)
xiaSetLogLevel(2)
# Edit saturn.ini to match your Saturn speed (20 or 40 MHz), 
# pre-amp type (reset or RC), and interface type (EPP, USB 1.0, USB 2.0)
xiaInit("saturn.ini")
xiaStartSystem
NDDxpConfig("DXP1", 1, 10, 42000000)
asynSetTraceIOMask("DXP1", 0, 2)
#asynSetTraceMask("DXP1", 0, 255)
dbLoadRecords("/corvette/home/epics/devel/dxp-6-0/dxpApp/Db/dxpSystem.template",   "P=dxpSaturn:, R=dxp1:,IO=@asyn(DXP1 0 1)")
dbLoadRecords("/corvette/home/epics/devel/dxp-6-0/dxpApp/Db/dxpHighLevel.template","P=dxpSaturn:, R=dxp1:,IO=@asyn(DXP1 0 1)")
dbLoadRecords("/corvette/home/epics/devel/dxp-6-0/dxpApp/Db/dxpSaturn.template",   "P=dxpSaturn:, R=dxp1:,IO=@asyn(DXP1 0 1)")
dbLoadRecords("/corvette/home/epics/devel/dxp-6-0/dxpApp/Db/dxpLowLevel.template", "P=dxpSaturn:, R=dxp1:,IO=@asyn(DXP1 0 1)")
dbLoadRecords("/corvette/home/epics/devel/dxp-6-0/dxpApp/Db/dxpSCA_16.template",   "P=dxpSaturn:, R=dxp1:,IO=@asyn(DXP1 0 1)")
dbLoadRecords("/corvette/home/epics/devel/dxp-6-0/dxpApp/Db/mcaCallback.template", "P=dxpSaturn:, R=mca1, IO=@asyn(DXP1 0 1)")
dbLoadRecords("/corvette/home/epics/devel/mca-7-9/mcaApp/Db/mca.db",               "P=dxpSaturn:, M=mca1, DTYP=asynMCA,INP=@asyn(DXP1 0),NCHAN=2048")
# Template to copy MCA ROIs to DXP SCAs
dbLoadTemplate("roi_to_sca.substitutions")
# Setup for save_restore
< ../save_restore.cmd
### save_restore setup
#
# The rest this file does not require modification for standard use, but...
# If you want save_restore to manage its own NFS mount, specify the name and
# IP address of the file server to which save files should be written.
# This currently is supported only on vxWorks.
#save_restoreSet_NFSHost("oxygen", "164.54.52.4")
# status-PV prefix
#save_restoreSet_status_prefix("xxx:")
# Debug-output level
save_restoreSet_Debug(0)
# Ok to save/restore save sets with missing values (no CA connection to PV)?
save_restoreSet_IncompleteSetsOk(1)
# Save dated backup files?
save_restoreSet_DatedBackupFiles(1)
# Number of sequenced backup files to write
save_restoreSet_NumSeqFiles(3)
# Time interval between sequenced backups
save_restoreSet_SeqPeriodInSeconds(300)
# specify where save files should be
set_savefile_path("autosave")
###
# specify what save files should be restored.  Note these files must be
# in the directory specified in set_savefile_path(), or, if that function
# has not been called, from the directory current when iocInit is invoked
#set_pass0_restoreFile("auto_positions.sav")
#set_pass0_restoreFile("auto_settings.sav")
#set_pass1_restoreFile("auto_settings.sav")
# load general-purpose interpolation tables with local, user-editable file
# (if interp_settings.req is included in auto_settings.req, the next line
# will overwrite those restored values)
#set_pass1_restoreFile("interp.sav")
###
# specify directories in which to to search for included request files
set_requestfile_path("./")
set_requestfile_path("/corvette/home/epics/devel/autosave-5-10-2", "asApp/Db")
set_requestfile_path("/corvette/home/epics/devel/areaDetector-3-13/ADCore",   "ADApp/Db")
set_requestfile_path("/corvette/home/epics/devel/calc-3-7-4",     "calcApp/Db")
macLib: macro CAMAC is undefined (expanding string set_requestfile_path("$(CAMAC)",    "camacApp/Db"))
set_requestfile_path("/corvette/home/epics/devel/dxp-6-0",      "dxpApp/Db")
set_requestfile_path("/corvette/home/epics/devel/mca-7-9",      "mcaApp/Db")
set_requestfile_path("/corvette/home/epics/devel/sscan-2-11-5",    "sscanApp/Db")
#dbLoadRecords("$(AUTOSAVE)/asApp/Db/save_restoreStatus.db", "P=xxx:")
save_restoreSet_status_prefix("dxpSaturn:")
dbLoadRecords("/corvette/home/epics/devel/autosave-5-10-2/asApp/Db/save_restoreStatus.db", "P=dxpSaturn:")
set_pass0_restoreFile("auto_settings.sav")
set_pass1_restoreFile("auto_settings.sav")
### Scan-support software
# crate-resident scan.  This executes 1D, 2D, 3D, and 4D scans, and caches
# 1D data, but it doesn't store anything to disk.  (See 'saveData' below for that.)
dbLoadRecords("/corvette/home/epics/devel/sscan-2-11-5/sscanApp/Db/scan.db","P=dxpSaturn:,MAXPTS1=2000,MAXPTS2=1000,MAXPTS3=10,MAXPTS4=10,MAXPTSH=2048")
iocInit()
Starting iocInit
############################################################################
## EPICS R7.0.7
## Rev. R7.0.7-dirty
## Rev. Date Git: 2022-09-07 13:50:35 -0500
############################################################################
reboot_restore: entry for file 'auto_settings.sav'
reboot_restore: Found filename 'auto_settings.sav' in restoreFileList.
*** restoring from 'autosave/auto_settings.sav' at initHookState 6 (before record/device init) ***
reboot_restore: done with file 'auto_settings.sav'

reboot_restore: entry for file 'auto_settings.sav'
reboot_restore: Found filename 'auto_settings.sav' in restoreFileList.
*** restoring from 'autosave/auto_settings.sav' at initHookState 7 (after record/device init) ***
reboot_restore: done with file 'auto_settings.sav'

iocRun: All initialization complete
### Start up the autosave task and tell it what to do.
# Save settings every thirty seconds
create_monitor_set("auto_settings.req", 30, P=dxpSaturn:)
### Start the saveData task.
saveData_Init("saveData.req", "P=dxpSaturn:")
saveData: message queue created
saveData:maxAllowedRetries = 10
saveData:retryWaitInSecs = 15
saveData: section [basename] not found
epics> auto_settings.sav: 848 of 848 PV's connected

epics>
MarkRivers commented 10 months ago

@xia-stan note that I reported basically the same issues to XIA in 2019, case XIA17040. That issue was with CentOS 7. Back then the communication with the MicroDXP seemed a little better. Sometimes it would run for a while without failing. it worked better on the computer in my office than the one that Soenke was using at sector 12. Now I cannot get it to communicate at all with Centos 9. But it works fine on the same computer with Windows, which was also the case back in 2019.

1458861693 commented 10 months ago

Thanks very much for your help. What should I do now to solve this problem？Please give me some suggestions. Thanks again.

MarkRivers commented 10 months ago

I don’t think there is anything you can do. We need XIA to solve this. I have demonstrated that the Saturn works fine on Linux, and the MicroDXP works fine with Windows on the same hardware where Linux fails.

1458861693 commented 10 months ago

OK，Looking forward the solution from XIA.

MarkRivers commented 10 months ago

I just connected a Mercury-4 to the same USB cable and port on the CentOS 9 system. The EPICS IOC works fine, it starts up with no errors, and I can start and stop acquisition. There are a few warnings about deprecated and unsupported acquisition values, but the USB communications works fine.

# Set logging level (1=ERROR, 2=WARNING, 3=INFO, 4=DEBUG)
xiaSetLogLevel(2)
xiaInit("mercury4.ini")
xiaStartSystem
[WARN ] 2024-01-10 14:16:53,706 pslSetAcquisitionValues (mercury_psl.c:1025)             : ignoring deprecated acquisition value: adc_percent_rule
[WARN ] 2024-01-10 14:16:53,861 psl__SetTriggerType (mercury_psl.c:9795)                 : Skipping trace_trigger_type, not supported by non Mercury OEM variant
[WARN ] 2024-01-10 14:16:53,861 psl__SetTriggerPosition (mercury_psl.c:9846)             : Skipping trace_trigger_position, not supported by non Mercury OEM variant
[WARN ] 2024-01-10 14:16:53,861 psl__SetAdcOffset (mercury_psl.c:9985)                   : Skipping adc_offset, not supported by non Mercury OEM variant
[WARN ] 2024-01-10 14:16:53,861 psl__SetOffsetDac (mercury_psl.c:10037)                  : Skipping offset_dac, not supported by non Mercury OEM variant
[WARN ] 2024-01-10 14:16:53,861 psl__SetBaselineFactor (mercury_psl.c:10088)             : Skipping baseline_factor, not supported by non Mercury OEM variant
[WARN ] 2024-01-10 14:16:54,088 pslSetAcquisitionValues (mercury_psl.c:1025)             : ignoring deprecated acquisition value: adc_percent_rule
[WARN ] 2024-01-10 14:16:54,239 psl__SetTriggerType (mercury_psl.c:9795)                 : Skipping trace_trigger_type, not supported by non Mercury OEM variant
[WARN ] 2024-01-10 14:16:54,239 psl__SetTriggerPosition (mercury_psl.c:9846)             : Skipping trace_trigger_position, not supported by non Mercury OEM variant
[WARN ] 2024-01-10 14:16:54,239 psl__SetAdcOffset (mercury_psl.c:9985)                   : Skipping adc_offset, not supported by non Mercury OEM variant
[WARN ] 2024-01-10 14:16:54,239 psl__SetOffsetDac (mercury_psl.c:10037)                  : Skipping offset_dac, not supported by non Mercury OEM variant
[WARN ] 2024-01-10 14:16:54,239 psl__SetBaselineFactor (mercury_psl.c:10088)             : Skipping baseline_factor, not supported by non Mercury OEM variant
[WARN ] 2024-01-10 14:16:54,467 pslSetAcquisitionValues (mercury_psl.c:1025)             : ignoring deprecated acquisition value: adc_percent_rule
[WARN ] 2024-01-10 14:16:54,618 psl__SetTriggerType (mercury_psl.c:9795)                 : Skipping trace_trigger_type, not supported by non Mercury OEM variant
[WARN ] 2024-01-10 14:16:54,619 psl__SetTriggerPosition (mercury_psl.c:9846)             : Skipping trace_trigger_position, not supported by non Mercury OEM variant
[WARN ] 2024-01-10 14:16:54,619 psl__SetAdcOffset (mercury_psl.c:9985)                   : Skipping adc_offset, not supported by non Mercury OEM variant
[WARN ] 2024-01-10 14:16:54,619 psl__SetOffsetDac (mercury_psl.c:10037)                  : Skipping offset_dac, not supported by non Mercury OEM variant
[WARN ] 2024-01-10 14:16:54,619 psl__SetBaselineFactor (mercury_psl.c:10088)             : Skipping baseline_factor, not supported by non Mercury OEM variant
[WARN ] 2024-01-10 14:16:54,846 pslSetAcquisitionValues (mercury_psl.c:1025)             : ignoring deprecated acquisition value: adc_percent_rule
[WARN ] 2024-01-10 14:16:54,997 psl__SetTriggerType (mercury_psl.c:9795)                 : Skipping trace_trigger_type, not supported by non Mercury OEM variant
[WARN ] 2024-01-10 14:16:54,997 psl__SetTriggerPosition (mercury_psl.c:9846)             : Skipping trace_trigger_position, not supported by non Mercury OEM variant
[WARN ] 2024-01-10 14:16:54,997 psl__SetAdcOffset (mercury_psl.c:9985)                   : Skipping adc_offset, not supported by non Mercury OEM variant
[WARN ] 2024-01-10 14:16:54,997 psl__SetOffsetDac (mercury_psl.c:10037)                  : Skipping offset_dac, not supported by non Mercury OEM variant
[WARN ] 2024-01-10 14:16:54,997 psl__SetBaselineFactor (mercury_psl.c:10088)             : Skipping baseline_factor, not supported by non Mercury OEM variant
# DXPConfig(serverName, ndetectors, maxBuffers, maxMemory)
NDDxpConfig("DXP1",  4, -1, -1)
[WARN ] 2024-01-10 14:16:55,170 pslSetAcquisitionValues (mercury_psl.c:1138)             : ignoring deprecated acquisition value: adc_percent_rule
[WARN ] 2024-01-10 14:16:55,223 pslSetAcquisitionValues (mercury_psl.c:1138)             : ignoring deprecated acquisition value: adc_percent_rule
[WARN ] 2024-01-10 14:16:55,275 pslSetAcquisitionValues (mercury_psl.c:1138)             : ignoring deprecated acquisition value: adc_percent_rule
[WARN ] 2024-01-10 14:16:55,328 pslSetAcquisitionValues (mercury_psl.c:1138)             : ignoring deprecated acquisition value: adc_percent_rule
[WARN ] 2024-01-10 14:16:55,329 pslSetAcquisitionValues (mercury_psl.c:1138)             : ignoring deprecated acquisition value: adc_percent_rule
[WARN ] 2024-01-10 14:16:55,330 pslSetAcquisitionValues (mercury_psl.c:1138)             : ignoring deprecated acquisition value: adc_percent_rule
[WARN ] 2024-01-10 14:16:55,331 pslSetAcquisitionValues (mercury_psl.c:1138)             : ignoring deprecated acquisition value: adc_percent_rule
[WARN ] 2024-01-10 14:16:55,332 pslSetAcquisitionValues (mercury_psl.c:1138)             : ignoring deprecated acquisition value: adc_percent_rule
dbLoadTemplate("4element.substitutions")
# Create a netCDF file saving plugin
NDFileNetCDFConfigure("DXP1NetCDF", 100, 0, DXP1, 0)
dbLoadRecords("/corvette/home/epics/devel/areaDetector-3-13/ADCore/db/NDFileNetCDF.template","P=dxpMercury:,R=netCDF1:,PORT=DXP1NetCDF,ADDR=0,TIMEOUT=1,NDARRAY_PORT=DXP1")
# Create a TIFF file saving plugin
NDFileTIFFConfigure("DXP1TIFF", 20, 0, "DXP1", 0)
dbLoadRecords("/corvette/home/epics/devel/areaDetector-3-13/ADCore/db/NDFileTIFF.template",  "P=dxpMercury:,R=TIFF1:,PORT=DXP1TIFF,ADDR=0,TIMEOUT=1,NDARRAY_PORT=DXP1")
# Create a NeXus file saving plugin
NDFileNexusConfigure("DXP1Nexus", 20, 0, "DXP1", 0, 0, 80000)
dbLoadRecords("/corvette/home/epics/devel/areaDetector-3-13/ADCore/db/NDFileNexus.template", "P=dxpMercury:,R=Nexus1:,PORT=DXP1Nexus,ADDR=0,TIMEOUT=1,NDARRAY_PORT=DXP1")
#xiaSetLogLevel(4)
#asynSetTraceMask DXP1 0 0x11
#asynSetTraceIOMask DXP1 0 2
### Scan-support software
# crate-resident scan.  This executes 1D, 2D, 3D, and 4D scans, and caches
# 1D data, but it doesn't store anything to disk.  (See 'saveData' below for that.)
dbLoadRecords("/corvette/home/epics/devel/sscan-2-11-5/sscanApp/Db/scan.db","P=dxpMercury:,MAXPTS1=2000,MAXPTS2=1000,MAXPTS3=10,MAXPTS4=10,MAXPTSH=2048")
# Load devIocStats
dbLoadRecords("/corvette/home/epics/devel/devIocStats-3-1-16/db/iocAdminSoft.db", "IOC=dxpMercury:")
iocInit
Starting iocInit
############################################################################
## EPICS R7.0.7
## Rev. R7.0.7-dirty
## Rev. Date Git: 2022-09-07 13:50:35 -0500
############################################################################
reboot_restore: entry for file 'auto_settings4.sav'
reboot_restore: Found filename 'auto_settings4.sav' in restoreFileList.
*** restoring from 'autosave/auto_settings4.sav' at initHookState 6 (before record/device init) ***
reboot_restore: done with file 'auto_settings4.sav'

reboot_restore: entry for file 'auto_settings4.sav'
reboot_restore: Found filename 'auto_settings4.sav' in restoreFileList.
*** restoring from 'autosave/auto_settings4.sav' at initHookState 7 (after record/device init) ***
reboot_restore: done with file 'auto_settings4.sav'

iocRun: All initialization complete
seq dxpMED, "P=dxpMercury:, DXP=dxp, MCA=mca, N_DETECTORS=4, N_SCAS=32"
sevr=info Sequencer release 2.2.5, compiled Mon Oct  2 12:22:39 2023
sevr=info Spawning sequencer program "dxpMED", thread 0x3f60420: "dxpMED"
### Start up the autosave task and tell it what to do.
# Save settings every thirty seconds
create_monitor_set("auto_settings4.req", 30, "P=dxpMercury:")
dxpMED: All channels connected.
### Start the saveData task.
saveData_Init("saveData.req", "P=dxpMercury:")
saveData: message queue created
saveData:maxAllowedRetries = 10
saveData:retryWaitInSecs = 15
saveData: section [basename] not found
# Sleep for 15 seconds to let initialization complete and then turn on AutoApply and do Apply manually once
epicsThreadSleep(15.)
[WARN ] 2024-01-10 14:16:56,223 pslSetAcquisitionValues (mercury_psl.c:1138)             : ignoring deprecated acquisition value: adc_percent_rule
[WARN ] 2024-01-10 14:16:56,238 pslSetAcquisitionValues (mercury_psl.c:1138)             : ignoring deprecated acquisition value: adc_percent_rule
[WARN ] 2024-01-10 14:16:56,240 pslSetAcquisitionValues (mercury_psl.c:1138)             : ignoring deprecated acquisition value: adc_percent_rule
[WARN ] 2024-01-10 14:16:56,241 pslSetAcquisitionValues (mercury_psl.c:1138)             : ignoring deprecated acquisition value: adc_percent_rule
[WARN ] 2024-01-10 14:16:56,242 pslSetAcquisitionValues (mercury_psl.c:1138)             : ignoring deprecated acquisition value: adc_percent_rule
...
[WARN ] 2024-01-10 14:17:02,317 pslSetAcquisitionValues (mercury_psl.c:1138)             : ignoring deprecated acquisition value: adc_percent_rule
# Turn on AutoApply
dbpf("dxpMercury:AutoApply", "Yes")
DBF_STRING:         "Yes"     
# Manually do Apply once
dbpf("dxpMercury:Apply", "1")
DBF_LONG:           1 = 0x1   
# Seems to be necessary to resend AutoPixelsPerBuffer to read back correctly from Handel
dbpf("dxpMercury:AutoPixelsPerBuffer.PROC", "1")
DBF_UCHAR:          1 = 0x1   
epics> [WARN ] 2024-01-10 14:17:11,295 pslSetAcquisitionValues (mercury_psl.c:1138)             : ignoring deprecated acquisition value: adc_percent_rule

epics> 
epics> 
epics> dbgrep *Start*
dxpMercury:EraseStart
dxpMercury:StartAll
epics> dbpf dxpMercury:EraseStart 1
DBF_STRING:         "Erase"   
epics> dbpf dxpMercury:StopAll 1
DBF_STRING:         "Stop"    
epics>

The ini files for the Mercury, Saturn, and MicroDXP all contain this line:

interface = usb2

So they are all using the same interface, usb2. That means they share common code in Handel for much of the communication.

The puzzle is why the Saturn and Mercury communicate fine, but the MicroDXP fails. Possible causes:

xv8tjc4d commented 10 months ago

I am facing a similar error "usb_bulk_write returned -110 should be 9", while making it work XIA MicroDXP on CentOS 7.

More Info:

xv8tjc4d commented 10 months ago

@1458861693 Did you have any luck trying to run on Ubuntu instead of CentOS?

MarkRivers commented 10 months ago

@1458861693 and @xv8tjc4d thanks to @xia-stan I just added a test program dxp/dxpApp/src/hqsg-microdxp.c and modified the Makefile to build that test program on Linux. That test program runs fine for me on CentOS 9. This is very interesting, since it means there is something in the EPICS driver that does not work, but the test program does.

Can you both please run this test program? Building it by pulling the master branch and rebuilding. You can run it by cd'ing to dxp/iocBoot/iocMicroDXP and typing

../../bin/linux-x86_64/hqsg-microdxp KETEK_DPP2_usb2.ini

You could also test with your own .ini files.

xv8tjc4d commented 10 months ago

@.*** iocMicroDXP]# ../../bin/linux-x86_64/hqsg-microdxp KETEK_DPP2_usb2.ini Configuring the Handel log file. Loading the .ini file. Error encountered! Status = 4002

On Fri, Jan 12, 2024 at 11:07 AM Mark Rivers @.***> wrote:

@1458861693 https://github.com/1458861693 and @xv8tjc4d https://github.com/xv8tjc4d thanks to @xia-stan https://github.com/xia-stan I just added a test program dxp/dxpApp/src/hqsg-microdxp.c and modified the Makefile to build that test program on Linux. That test program runs fine for me on CentOS 9. This is very interesting, since it means there is something in the EPICS driver that does not work, but the test program does.

Can you both please run this test program? Building it by pulling the master branch and rebuilding. You can run it by cd'ing to dxp/iocBoot/iocMicroDXP and typing

../../bin/linux-x86_64/hqsg-microdxp KETEK_DPP2_usb2.ini

You could also test with your own .ini files.

— Reply to this email directly, view it on GitHub https://github.com/epics-modules/dxp/issues/10#issuecomment-1889811860, or unsubscribe https://github.com/notifications/unsubscribe-auth/BCTRTK3LYQPOC6YRNAZPDZ3YOGCWFAVCNFSM6AAAAABA5ZBALCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBZHAYTCOBWGA . You are receiving this because you were mentioned.Message ID: @.***>

xv8tjc4d commented 10 months ago

/ I/O level error codes 4001-4100 /

44 | #define DXP_MDOPEN 4001 / Error opening port for communication / 45 | #define DXP_MDIO 4002 / Device IO error / 46 | #define DXP_MDINITIALIZE 4003 / Error configurting port during initialization / 47 | #define DXP_MDSIZE 4007 / Incoming data packet length larger than requested / 48 | #define DXP_MDUNKNOWN 4009 / Unknown IO function /

MarkRivers commented 10 months ago

@xv8tjc4d please attach the Handel.log file so we can see more what happened. You will need to rename it to Handel.log.txt to attach it.

MarkRivers commented 10 months ago

I spoke too soon. I have run the test program more times, and I find that it fails about half the time for me on CentOS 9. I have attached 3 log files:

This time the program ran, but it does print errors trying to set the IO priority. handel.log.good.txt

This time the program failed downloading the firmware. handel.log.bad.txt

This time the program failed reading SLOWLEN in pslReadoutPeakingTimes. handel.log.bad2.txt

xv8tjc4d commented 10 months ago

here are my logs on CentOS 7: "[403] usb_bulk_write returned -110 should be 9" I also had : [4002] Error downloading firmware via Xerxes. handel.log.txt

MarkRivers commented 10 months ago

That is the same error I got in handel.log.bad2.txt above.

MarkRivers commented 10 months ago

So now we know that a simple simple test program from XIA reproduces the problem on multiple systems.

xv8tjc4d commented 10 months ago

I tried on Ubuntu and same error happens 4002.

1458861693 commented 10 months ago

@1458861693 Did you have any luck trying to run on Ubuntu instead of CentOS?

I have tried on Ubuntu 20.04，but it also doesn't work.

1458861693 commented 10 months ago

Hi， @MarkRivers ，I have downloaded the newest master branch and rebuild epics dxp module to try this test program. The attachment is the log file named log.txt. Test program seems run OK on my PC，which is CentOS7 System.
log.txt I have run the test program many times on my CentOS7 PC，it works OK and no error！

1458861693 commented 10 months ago

I have tested this test program several times again，it works failed. This is the error information named handel.bad.log.txt. handel.bad.log.txt

MarkRivers commented 10 months ago

@1458861693 the file you uploaded is just the console output. What is more important is the handel.log file that the test program produces. That is the file you need to rename to handel.log.txt and then upload.

1458861693 commented 10 months ago

OK，thanks your help first，I run the test program just now，unfortunately it failed. This attachment is the log file named handel.bad.log.txt. handel.bad.log.txt Thanks again.

MarkRivers commented 10 months ago

I just tried running hqsg-microdxp.c many times in a row to get some statistics. These are the error codes printed each time I ran it:

4205
4144
4205
4002
(no error, success)
4144
4002
4001
4001
...

Once I got error 4001 the device is unusable, I get that error every time I run the program. I believe I need to power-cycle the device, but I am working from home so I can't do that. That handel.log file ends with this:

[INFO ] 2024-01-13 08:49:10,551 dxp_add_board_item (xerxes.c:533)                        : New module '0': interface = '0', # channels = 1
[INFO ] 2024-01-13 08:49:10,551 dxp_add_board_item (xerxes.c:565)                        : Opening interface ioname=0 iface_dllname=usb2 funcs=0x1487e90
[INFO ] 2024-01-13 08:49:10,551 xia_usb2_open (xia_usb_linux.c:163)                      : Entry: device_number = 0, static handle = (nil)
[INFO ] 2024-01-13 08:49:10,551 xia_usb2_open (xia_usb_linux.c:168)                      : No handle
[INFO ] 2024-01-13 08:49:10,554 xia_usb2_open (xia_usb_linux.c:190)                      : Opening device 0x10e9:0xb01 number 0
[INFO ] 2024-01-13 08:49:10,554 xia_usb2_open (xia_usb_linux.c:200)                      : setting configuration: 1
[INFO ] 2024-01-13 08:49:10,554 xia_usb2_open (xia_usb_linux.c:229)                      : usb_set_configuration failed: -2
[ERROR] 2024-01-13 08:49:10,555 dxp_md_usb2_open (md_linux.c:1235)                       : [4001] Error opening USB device '0', where the driver reports a status of -2
[ERROR] 2024-01-13 08:49:10,555 dxp_add_board_item (xerxes.c:572)                        : [4001] Error opening module '0' with interface '0'
[ERROR] 2024-01-13 08:49:10,555 xia__AddXerxesModule (handel_xerxes.c:1321)              : [4001] Error adding module to Xerxes for alias 'module1'
[ERROR] 2024-01-13 08:49:10,555 xiaBuildXerxesConfig (handel_xerxes.c:170)               : [4001] Error adding module for alias = 'module1'
[ERROR] 2024-01-13 08:49:10,555 xiaStartSystem (handel_system.c:154)                     : [4001] Error configuring Xerxes.

Before the device got hung up with error 4001 I tried running the test program as root. That does remove the error about not being able to set the priority, but it does not improve the failure rate later on.

MarkRivers commented 10 months ago

I just learned something quite interesting.

My tests so far have been on a Dell Precision 5860. This is a dual-boot system with Windows 10 and CentOS 9 stream. The MicroDXP fails on CentOS 9.

Today I tested on a Dell Precision 5820. This is a triple-boot system with Windows 10, CentOS 7, and Ubuntu 22. To my surprise the MicroDXP worked fine with both CentOS 7 and Ubuntu 22 on this machine. I ran the hqsg-microdxp test program over 50 times on each OS with no errors. I also ran the EPICS IOC over 20 times on each OS with no errors.

The other users have reported failures with CentOS 7 and Ubuntu 20.

I am wondering if the problem might actually be related to the computer hardware (i.e. USB controller) rather than the OS.

1458861693 commented 10 months ago

@MarkRivers Could you provide me the IOC here that you ran successfully on Dell Precision 5820. I want to test this IOC on my computer which is also Dell Precision and CentOS7 System. Thanks！