search

paulscherrerinstitute / ADZMQ

ZMQ ADDriver and NDPlugin for areaDetector
1 stars 2 forks source link

build ADSimDetector and ANZMQ into docker image #2

Closed Tinanuaa closed 3 years ago

Tinanuaa commented 3 years ago

Hi , I want to install ADZMQ with ADSimDetector, and I am using linux alpine, epics-base:3.14.12.6 for building areadetector, ADSimDetector and ADZMQ into a docker image.

I followed https://areadetector.github.io/master/install_guide.html to install everything except ADZMQ, I built areadetector and ADSimDetector successfully, but when I added ADZMQ into the dockerfile, I got an error as following when running docker build

31 584.2 make[6]: *** No rule to make target '../../../lib/linux-x86_64/libADZMQ.a', needed by 'simDetectorApp'. Stop.

31 584.2 make[5]: *** [/EPICS/base/configure/RULES_ARCHS:61: install.linux-x86_64] Error 2

31 584.2 make[4]: *** [/EPICS/base/configure/RULES_DIRS:83: src.install] Error 2

31 584.2 make[3]: *** [/EPICS/base/configure/RULES_DIRS:83: simDetectorApp.install] Error 2

31 584.2 make[2]: *** [/EPICS/base/configure/RULES_DIRS:83: simDetectorIOC.install] Error 2

31 584.2 make[1]: *** [/EPICS/base/configure/RULES_DIRS:83: iocs.install] Error 2

31 584.2 make: *** [/EPICS/base/configure/RULES_DIRS:83: /EPICS/support/areaDetector/ADSimDetector.install] Error 2

my steps for adding ANZMQ is as following:

  1. create RELEASE.local and put into ADZMQ/configure/RELEASE.local 
    
SUPPORT=/EPICS/support
EPICS_BASE=/EPICS/base

asyn

ASYN=$(SUPPORT)/asyn BUSY=$(SUPPORT)/busy CALC=$(SUPPORT)/calc SSCAN=$(SUPPORT)/sscan AUTOSAVE=$(SUPPORT)/autosave

AREA_DETECTOR is needed for base and plugins

AREA_DETECTOR=/EPICS/support/areaDetector ADBINARIES=$(AREA_DETECTOR)/ADBinaries ADCORE=$(AREA_DETECTOR)/ADCore


2. change arch to linux in ADZMQ/iocs/zmqIOC/iocBoot/iocZMQ/Makefile
`ARCH = linux-x86_64`

3. add ADZMQ inside areaDetector/Makefile

ifdef ADZMQ DIRS := $(DIRS) $(ADZMQ) $(ADZMQ)_DEPEND_DIRS += $(ADCORE) endif


4. add ADZMQ in areaDetector/configure/RELEASE_PRODS.local

ADZMQ

ADZMQ=$(AREA_DETECTOR)/ADZMQ


5. add ADZMQ in areaDetector/ADSimDetector/iocs/simDetectorIOC/configure/RELEASE
`ADZMQ=$(AREA_DETECTOR)/ADZMQ`

6. add ADZMQ in areaDetector/ADSimDetector/iocs/simDetectorIOC/simDetectorApp/src/Makefile

PROD_LIBS += ADZMQ

ifeq ($(OS_CLASS),WIN32) LIBZMQ = libzmq else LIBZMQ = zmq endif PROD_LIBS += $(LIBZMQ)


7. add ADZMQ in areaDetector/ADSimDetector/iocs/simDetectorIOC/iocBoot/iocSimDetector/st_base.cmd

add ADZMQ

NDZMQConfigure("NDZMQ1", "tcp://*:1234", 3, 0, "$(PORT)", 0) dbLoadRecords("$(ADCORE)/ADApp/Db/NDPluginBase.template","P=$(PREFIX),R=ZMQ1:,PORT=NDZMQ1,ADDR=0,TIMEOUT=1,NDARRAY_PORT=$(PORT),NDARRAY_ADDR=0")

doAfterIocInit "dbpf 13SIM1:ZMQ1:EnableCallbacks,1"


8. add ADZMQ in areaDetector/Makefile

ifdef ADZMQ DIRS := $(DIRS) $(ADZMQ) $(ADZMQ)_DEPEND_DIRS += $(ADCORE) endif



am I missing something?   Do I need to make ANZMQ before SimDetector? Cause if I don't change the configurations related to simdetector, I can build the image successfully, and the files in /EPICS/support/areaDetector/ADZMQ/lib/linux-x86_64 are libADZMQ.a   libADZMQ.so  libzmq.a.
xiaoqiangwang commented 3 years ago

Did you check wether ADZMQ has got built?

$ ls /EPICS/support/areaDetector/ADZMQ

To get it automatically built, add ADZMQ definition also to /EPICS/support/areaDetector/configure/RELEASE.local

ADZMQ=$(AREA_DETECTOR)/ADZMQ
Tinanuaa commented 3 years ago

Yes, I added in the /EPICS/support/areaDetector/configure/RELEASE.local as well, sorry I forgot to mention this

Tinanuaa commented 3 years ago

I debug the docker build and found when the command 

make -C areaDetector install

failed , the ADZMQ was not compiled, so no lib folder in ADZMQ folder image

Tinanuaa commented 3 years ago

I added dependency in areaDetector/Makefile for simDetector 

# Build simulation drivers next
ifdef ADSIMDETECTOR
DIRS := $(DIRS) $(ADSIMDETECTOR)
$(ADSIMDETECTOR)_DEPEND_DIRS += $(ADCORE)
$(ADSIMDETECTOR)_DEPEND_DIRS += $(ADZMQ)
endif

then I can build the image but when I run it, it complains port NDZMQ1 not found, then I add the following code in ADSimDetector/iocs/simDetectorIOC/simDetectorApp/src/Makefile

$(PROD_NAME)_DBD += ADZMQSupport.dbd

The error gone but I can't see anything coming out from zmq_client? image

xiaoqiangwang commented 3 years ago

Did you start the acquisition and make sure array callbacks are enabled?

$ caput 13SIM1:cam1:ArrayCallbacks 1
$ caput 13SIM1:cam1:Acquire 1
Tinanuaa commented 3 years ago

I started the acquisition and enable the array callbacks, but still got nothing? image

xiaoqiangwang commented 3 years ago

For test purpose, is the detector in continuous mode? simDetector_acquire

Also your iocsh output would be helpful.

Tinanuaa commented 3 years ago

when you say iocsh ,do you mean epics> this? there is no output from epics> and where can I check if it's in continuous mode except from the gui?

xiaoqiangwang commented 3 years ago

It could be that you don't have std module built in your IOC. And doAfterIocInit is then missing. Do the following after EPICS IOC starts,

# continuous acquisition
$ caput 13SIM1:cam1:ImageMode Continuous 
# enable array callback
$ caput 13SIM1:cam1:ArrayCallbacks 1
# enable NDPluginZMQ
$ caput 13SIM1:ZMQ1:EnableCallbacks 1
# acquire
$ caput 13SIM1:cam1:Acquire 1

Now start zmq client script

python zmq_client.py
Tinanuaa commented 3 years ago

Thank you so much! It works! Just one more question, install std module and add doAfterIocInit in st_base.cmd is equivalent to the commands listed above right?

xiaoqiangwang commented 3 years ago

Having doAfterIocInit in st.cmd has the same effect as doing caput after IOC starts, but just more convenient in this case.

I would still suggest you use the control panels to get an overview.

Tinanuaa commented 3 years ago

Sorry for bothering you again, but I was checking whether the zmq_client can receive all the images streamed from ADZMQ and find that the higher the image rate, the more images would get lost. My steps for doing this check is as following: 

# put ImageMode to continuous
caput 13SIM1:cam1:ImageMode Continuous
# set the image rate to full speed, should be around 1500 frame/sec
caput 13SIM1:cam1:AcquirePeriod 0
caput 13SIM1:cam1:AcquireTime 0
# start acquisition
caput 13SIM1:cam1:Acquire 1
# wait for some time and then stop acquisition
caput 13SIM1:cam1:Acquire 0
# get the number of images acquired 
caget 13SIM1:cam1:NumImagesCounter_RBV

then with the zmq_client, the number of images saved is always less than the number from 'caget 13SIM1:cam1:NumImagesCounter_RBV'.

But when I changed the image rate to a lower rate with following command

caput 13SIM1:cam1:AcquirePeriod 1
caput 13SIM1:cam1:AcquireTime 1

the images saved by zmq_client and the number from 'caget 13SIM1:cam1:NumImagesCounter_RBV' is the same.

Am I missing something or the steps for checking the 'images sent = image received' wrong?

xiaoqiangwang commented 3 years ago

On my (not powerful) PC, simDetector cannot reach 1500 fps creating 1024 x 1024 x 8bits Mono. But at its limit, there is no frame loss.

Did you comment out the print functions in _zmqclient.py?

Tinanuaa commented 3 years ago

I suppose image_rate(frames/sec) = 1/(AcquirePeriod+AcquireTime), I got the max image rate from the following link, which shows if AcquirePeriod=AcquireTime=0 then it's around 1500 frames/sec (don't quite understand why).
I didn't comment out the print function and I just commented it out but still lose data. Are you checking frame loss the same way as I did?

So suppose I set AcquireTime=0.01, AcquirePeriod =0 , the image rate should be 100 frames/sec, and I start acquisition for some time, around 10 sec, then stop acquire. The "caget 13SIM1:cam1:NumImagesCounter_RBV" shows 1211 frames acquired, but I only received 1194 frames.

"Simulation detector with a single small peak in Peaks mode. The driver was running in Continuous mode with AcquireTime=0, and AcquirePeriod=0, to achieve maximum frame rate. Both Windows and Linux were able to generate about 1500 frames/sec, which was faster than any of the file plugins could write, so they were always dropping frames." -- https://cars9.uchicago.edu/software/epics/NDPluginFile.html

xiaoqiangwang commented 3 years ago

I have ~400fps for the test. And the actual fps can be read by

$ caget 13SIM1:cam1: ArrayRate_RBV

In my test, I commented out the print functions and use a counter to count the images received. Probably you should reset the array counter on startup.

# reset the array counter
$ caput 13SIM1:cam1:ArrayCounter 0
# start acquisition
caput 13SIM1:cam1:Acquire 1
Tinanuaa commented 3 years ago

Hi, what's the 13SIM1:cam1:NumImages did you set? Have you tried the Multiple ImageMode, I found that when the NumImages is 1000, then the zmq_client sometimes can receive all the images but sometimes lose images. And I checked the source code in ADZMQ, I found that NDPluginZMQ::processCallbacks actually send the header and message into 0MQ, is there a way to check that ADZMQ did send everything to 0MQ?

And if I set NumImages to 10000, then it loses images almost every time.

I commented out writing into files, print functions and just use a counter as you did.

xiaoqiangwang commented 3 years ago

PUB/SUB pattern does not provide the feedback whether messages are delivered. I am adding this information for PUSH/PULL pattern.

I created another topic #4 to track performance related issues.