MEDM not responsive after acquiring image in "continuous" mode and Mono12Packed Pixel format

[LeeYangLBLBCS]

issue moved from: https://github.com/epics-extensions/medm/issues/6

[LeeYangLBLBCS]

I don't know how to setup the status update with 20fps, but here is the screen of top -H

[MarkRivers]

I don't know how to setup the status update with 20fps, but here is the screen of top -H

You misunderstood what I meant. I meant that you set the FrameRate 20 fps and then you set that Status update to non-Passive, e.g. "1 second". I want to set if there are any FailedPacket at 20 fps.

However, I just remember that they changed the name of that statistic between 1.20 and 2.0. It used to be called BufferUnderrun I think. If you use the medm screen and database from 1.20 they should match the driver OK.

[MarkRivers]

When you made that screen shot of "top" what were the camera parameters (bits, FrameRate, PixelFormat, ConvertPixelFormat)?

[LeeYangLBLBCS]

the "top" screen shot was based on: SDK 2.0 framerate: free run (at 85 fps) exposure time 0.001 s ADC 10 bit pixel format; mono16 convert format: none

I tried lower frame rate but it didn't work (20fps, 1fps). As you said I will need to switch back to V1.20 to see it.

On Tue, Sep 1, 2020 at 12:23 PM Mark Rivers notifications@github.com wrote:

When you made that screen shot of "top" what were the camera parameters (bits, FrameRate, PixelFormat, ConvertPixelFormat)?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/areaDetector/ADSpinnaker/issues/4#issuecomment-685082457, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADSYGNEC4HPQHXTXB6ISL43SDVC43ANCNFSM4PJ2ADJQ .

-- Lee Yang Lawrence Berkeley National Lab 1 Cyclotron Road, MS 7H210 Berkeley, California 97320 office:(510)486-7320 fax:(510) 486-4633

[MarkRivers]

I tried lower frame rate but it didn't work (20fps, 1fps).

I don't understand what you mean by "it didn't work". What didn't work at lower frame rate?

[LeeYangLBLBCS]

I've changed back to V1.20, with frame rate at 20 fps. status update at 1 second. All values in status update are 0, shown in screen shot. The acquisition failed to complete 5000 images when it reached 4982. There are 12 error message in IOC, which is less than the difference between images. acquired and number of images need to be acquired. The screen shot of "top" is also attached. epics> 2020/09/01 12:34:51.115 ADSpinnaker::grabImage error GetImageStatus returned 9 2020/09/01 12:34:51.129 ADSpinnaker::grabImage error GetImageStatus returned 9 2020/09/01 12:34:53.115 ADSpinnaker::grabImage error GetImageStatus returned 9 2020/09/01 12:34:53.131 ADSpinnaker::grabImage error GetImageStatus returned 9 2020/09/01 12:35:23.260 ADSpinnaker::grabImage error GetImageStatus returned 9 2020/09/01 12:35:32.515 ADSpinnaker::grabImage error GetImageStatus returned 9 2020/09/01 12:37:15.911 ADSpinnaker::grabImage error GetImageStatus returned 9 2020/09/01 12:38:01.560 ADSpinnaker::grabImage error GetImageStatus returned 9 2020/09/01 12:38:05.959 ADSpinnaker::grabImage error GetImageStatus returned 9 2020/09/01 12:38:45.008 ADSpinnaker::grabImage error GetImageStatus returned 9 2020/09/01 12:38:46.659 ADSpinnaker::grabImage error GetImageStatus returned 9 2020/09/01 12:38:51.608 ADSpinnaker::grabImage error GetImageStatus returned 9

[LeeYangLBLBCS]

I got an email from FLIR tech support a while ago. Here is what they told me about how to optimizing the camera performance: If that does not help, I suggest to increase the StreamDefaultBufferCount to 100 if it set to lower value. If this is too large then please try to reduce to 100. You cannot change this value in SpinView directly so I suggest to run Bufferhandling.cpp example with pCamPtr->TLStream.StreamDefaultBufferCount.SetValue(100).

[MarkRivers]

I just made an interesting observation. A couple of days ago I learned that arvavis 0.8.0 has been released, so I updated ADAravis to use that new release. I then tested it using the Oryx camera. To my surprise it works perfectly, running at the theoretical frame rates published by FLIR without dropping a single frame. It does not yet support Mono12Packed (because it does not provide an 12 to 16 bit converter) but I was able to test Mono8 and Mono8 with the 8, 10, and 12-bit ADC settings. I always got the theoretical frame rate, and I was always able to collect 5000 frames in Multiple mode.

This tells me that the camera hardware and the operating system are working fine. Whatever problem we are having with ADSpinnaker must either be a problem with Spinnaker itself or with the ADSpinnaker driver.

[LeeYangLBLBCS]

Did you also test mono16 with arvavis?

I acquired 5000 images at 80 Hz using spinview 2.0. The BufferUnderRun (which is the lost frame count) is 0. Does that mean spinview can acquire at 80Hz without a problem? I'm not sure if spinview is reading all the image though. Their caption under the image window says "Camera FPS 80.00Hz, Processed FPS: 30.12Hz). Maybe spinview only "process" images at 30 Hz, even it acqires at 80Hz?

Also, does epics interface have a way to set the "StreamDefaultBufferCount"? Their tech support suggested to set it to 100. pCamPtr->TLStream.StreamDefaultBufferCount.SetValue(100)

[MarkRivers]

Did you also test mono16 with arvavis?

Yes, I just did all of the tests in the following table using ADAravis. It never dropped any frames and it always achieved the published frame rate except for 8-bit/Mono16 where it gets 79 rather than 81 and 10-bit/Mono16 where it gets 84 rather than 85. But I think the published values are probably wrong.

Pixel format	ConvertPixelFormat	ADC bits	FLIR specified frame rate	Actual frame rate	# Images collected	Can stop?
Mono8	None	8	162	162	1000	Yes
Mono16	None	8	81	79	1000	Yes
Mono8	None	10	144	144	1000	Yes
Mono16	None	10	85	84	1000	Yes
Mono8	None	12	89	89	1000	Yes
Mono16	None	12	89	89	1000	Yes

The BufferUnderRun (which is the lost frame count) is 0. Does that mean spinview can acquire at 80Hz without a problem? I'm not sure if spinview is reading all the image though. Their caption under the image window says "Camera FPS 80.00Hz, Processed FPS: 30.12Hz). Maybe spinview only "process" images at 30 Hz, even it acqires at 80Hz?

I think it is probably working. Processed may mean "displayed" and that involves the speed of the viewer.

Also, does epics interface have a way to set the "StreamDefaultBufferCount"? Their tech support suggested to set it to 100. pCamPtr->TLStream.StreamDefaultBufferCount.SetValue(100)

No it does not have that, but it could probably be added.

[decarlof]

I don't see drop frame count on the top right corner. Maybe I don't know where to look? The screen I attached is showing no dropped frames in "status" area, while the acquiring is stuck at frame 4784 out of 5000 frames.

we had dropped frames and hanging when the camera run above 50C. Since then we had an external fan that keeps the temperature reading below 45C and this solved the issue for us.

[LeeYangLBLBCS]

Awesome! that could be the reason. I'll find a way to cool it.

On Thu, Sep 10, 2020 at 10:05 AM Francesco De Carlo < notifications@github.com> wrote:

I don't see drop frame count on the top right corner. Maybe I don't know where to look? The screen I attached is showing no dropped frames in "status" area, while the acquiring is stuck at frame 4784 out of 5000 frames. [image: image] https://user-images.githubusercontent.com/15041332/91768125-06cbe600-eb92-11ea-8ecd-ea4091372649.png

we had dropped frames and hanging when the camera run above 50C. Since then we had an external fan that keeps the temperature reading below 45C and this solved the issue for us.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/areaDetector/ADSpinnaker/issues/4#issuecomment-690522069, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADSYGNCQCG72FMQPWZH7LULSFEBPHANCNFSM4PJ2ADJQ .

-- Lee Yang Lawrence Berkeley National Lab 1 Cyclotron Road, MS 7H210 Berkeley, California 97320 office:(510)486-7320 fax:(510) 486-4633

[MarkRivers]

The title of this issue was related to the fact that the task that grabs the images from the camera had the port mutex nearly continuously when acquiring fast in Mono12Packed mode. This meant that the Acquire Stop command was not being given time to process. I believe this problem is fixed, because the mutex is now released during the time-consuming conversion operation from Mono12Packed to Mono16.

The other issue that was discussed here was the fact that the driver was dropping frames when collecting in Mono12Packed mode. I have now found that problem and fixed it. The problem was that the conversion operation was taking enough time that that it could slow down the thread that reads the images from a message queue. If the number of unprocessed images was more than 10 then Spinnaker ran out of Transport Layer buffers, and frames were dropped.

I have changed the constructor and the ADSpinnakerConfigure() command to replace the traceMask argument (third argument) with numSPBuffers, which is the number of Transport Layer buffers to allocate. If this argument is 0 or omitted it defaults to 100. With the value set to 100 I no longer see any dropped frames in any of the 9 possible combinations of AdcBitDepth and PixelFormat. I have tested collecting 5000 frames at least 3 times in each mode.

I have also enhanced the ADAravis driver so that it can handle Mono12Packed and Mono12p formats, and convert them to Mono16. It also works perfectly, never dropping any frames.

Pixel format	ADC bits	FLIR specified frame rate	Actual frame rate	# Images collected ADAravis	# Images collected ADSpinnaker
Mono8	8	162	162	5000	5000
Mono12Packed	8	122	122	5000	5000
Mono16	8	81	79	5000	5000
Mono8	10	144	144	5000	5000
Mono12Packed	10	130	130	5000	5000
Mono16	10	85	84	5000	5000
Mono8	12	89	89	5000	5000
Mono12Packed	12	89	89	5000,	5000
Mono16	12	89	89	5000	5000

The master branch of ADAravis also requires the master branch of ADGenICam, which is where the Mono12 decompression is implemented.

The master branch of ADSpinnaker should resolve this issue. NOTE HOWEVER that startup scripts may need to be changed to replace the asynTrace argument to ADSpinnakerConfig with numSPBuffers. The argument can be deleted completely to use the default of 100 buffers.

In the course of this work I have found an issue with the Oryx firmware which was confirmed by FLIR.

The following table shows the values of 3 frame rate related values as a function of the PixelFormat and AdcBitDepth GenICam features.

• The column labeled “FLIR specified frame rate” is from this table: http://softwareservices.flir.com/ORX-10G-51S5/latest/Model/spec.html
• The column labeled “GenICam AcquisitionFrameRate feature maximum” is the value returned by the Spinnaker function pNode->GetMax() for this CFloatPtr feature.
• The column labeled “Measured maximum frame rate” is the maximum frame rate I was able to use and not drop frames.

PixelFormat	AdcBitDepth	FLIR specified frame rate	GenICam AcquisitionFrameRate feature maximum	Measured maximum frame rate
Mono8	8	162	163.12	162
Mono12Packed	8	122	163.12	122
Mono16	8	81	123.97	79
Mono8	10	144	144.55	144
Mono12Packed	10	130	144.55	130
Mono16	10	85	123.97	84
Mono8	12	89	89.80	89
Mono12Packed	12	89	89.80	89
Mono16	12	89	89.80	89

I have highlighted in boldface the values where the AcquisitionFrameRate maximum is significantly different from the FLIR specified frame rate, and also from the measured maximum frame rate. This is a problem because the user has no way to know that they have selected a frame rate that is too large for these modes, except by seeing that the camera does not achieve the requested rate.

FLIR has confirmed that this is a problem with the current FLIR firmware.

They told me this:

I noticed that I was able to update the firmware to a newer version to resolve this issue. This is a beta firmware that hasn't gone through our entire set of tests, so it is possible that there are other issues with it. You can download the firmware here: https://flir.boxcn.net/s/vv6fkmkzpd24ou226zn4nz8b8gwuf17z

I have also found what appears to be a bug in the Spinnaker code to convert Mono12 data to Mono16. It appears to work fine for Mono12Packed. However, it gives the wrong data for Mono12p, which is a slightly different packing algorithm. I implemented the unpacking myself in ADGenICam and I use that for ADAravis. It works correctly for both Mono12Packed and Mono12p, so I am quite sure the problem is in Spinnaker. I am waiting to get a confirmation of this bug from FLIR.

[decarlof]

I did not use Mono12Packed so I did not see the problem. Now the computer controlling my FLIR Oryx has been just updated from Ubuntu to RH8 and I get the following:

Pixel format	ADC bits	Actual frame rate
Mono8	8	161
Mono12Packed	8	error (*)
Mono16	8	64
Mono8	10	144
Mono12Packed	10	error (*)
Mono16	10	70
Mono8	12	90
Mono12Packed	12	error (*)
Mono16	12	84

(*) 2020/09/28 13:37:33.806 ADSpinnaker:grabImage: unsupported pixel format 0xb

The ADSpinnaker installed version is ~ a week old and does not contain the recent fix, I will have that installed as soon as possible, still the actual frame rate in Mono16 is lower than the specified. Is something else affecting the Mono16 frame rate?

Below are software/skd/driver versions:

[LeeYangLBLBCS]

the error you are seeing: (*) 2020/09/28 13:37:33.806 ADSpinnaker:grabImage: unsupported pixel format 0xb is becaused the Convert Format is not set to "mono16" when using mono12packed.Pixel format.

On Mon, Sep 28, 2020 at 12:21 PM Francesco De Carlo < notifications@github.com> wrote:

I did not use Mono12Packed so I did not see the problem. Now the computer controlling my FLIR Oryx has been just updated from Ubuntu to RH8 and I get the following: Pixel format ADC bits Actual frame rate Mono8 8 161 Mono12Packed 8 error () Mono16 8 64 Mono8 10 144 Mono12Packed 10 error () Mono16 10 70 Mono8 12 90 Mono12Packed 12 error (*) Mono16 12 84

(*) 2020/09/28 13:37:33.806 ADSpinnaker:grabImage: unsupported pixel format 0xb

The ADSpinnaker installed version is ~ a week old and does not contain the recent fix, I will have that installed as soon as possible, still the actual frame rate in Mono16 is lower than the specified. Is something else affecting the Mono16 frame rate?

Below are software/skd/driver versions: [image: Screen Shot 2020-09-28 at 1 42 56 PM] https://user-images.githubusercontent.com/5884150/94474626-877f0180-0193-11eb-8b02-04f0f609d1bb.png

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/areaDetector/ADSpinnaker/issues/4#issuecomment-700231528, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADSYGNDRCBH3JSCRNLEC74DSIDOZXANCNFSM4PJ2ADJQ .

-- Lee Yang Lawrence Berkeley National Lab 1 Cyclotron Road, MS 7H210 Berkeley, California 97320 office:(510)486-7320 fax:(510) 486-4633

[MarkRivers]

@decarlof you have FrameRateEnable=No. In that case camera will try to use the maximum frame rate that the camera firmware thinks is possible, which can be a little too high. Try setting FrameRateEnable=Yes and then set the FrameRate to the value from the "Measured maximum frame rate" from my table above. See if you can reproduce my table. If not I can give some other suggestions.

[decarlof]

with FrameRateEnable=Yes I get:

Pixel format	ADC bits	Actual frame rate	Set frame rate
Mono16	8	61	79
Mono16	10	68	84
Mono16	12	83	89

[MarkRivers]

@decarlof how are you getting the Actual frame rate? From FrameRate_RBV, or from ArrayRate_RBV, i.e. is it accepting your requested FrameRate and just not achieving it, or it is rejecting your requested FrameRate?

Did you try the Mono12Packed with ConvertFormat=Mono16? What frame rate do you get with those?

Can you send a screen shot of the features-2 screen? This is what mine looks like:

[decarlof]

The actual frame rate in my table is from ArrayRate_RBV. Only the Monoo16/12 bit frame rate is not accepting my requested FrameRate. Here is the same table with more details:

Pixel format	ADC bits	Actual frame rate (ArrayRate_RBV)	Set frame rate (FrameRate)	FrameRate_RBV
Mono16	8	61	79	79
Mono16	10	68	84	84
Mono16	12	83	89	84

Here is my features-2 screen:

There are differences in the DeviceMaxThroughput, DeviceLinkThroughputLimit, DeviceLinkCurrentThroughput and PayloadSize.

I did not try Mono12Packed with ConvertFormat=Mono16 yet since my current version does not accept Mono12Packed (keeps Data type unchanged). I will try that as soon as I get the update installed.

[MarkRivers]

You can try increasing the DeviceLinkThroughputLimit up to the same value as DeviceLinkSpeed. That is what I did.

Also, when you updated from Ubuntu to RH8 did you increase the system settings described here: https://areadetector.github.io/master/ADGenICam/ADGenICam.html#linux-usb-and-gige-system-settings

I did not try Mono12Packed with ConvertFormat=Mono16 yet since my current version does not accept Mono12Packed (keeps Data type unchanged). I will try that as soon as I get the update installed.

You are using R2-2 which can do the conversion from Mono12Packed to Mono16. The main screen should look like this. I have AdcBitDepth=10, and it does 130 frames/s.

[MarkRivers]

I forgot to paste the screen shot in the previous comment.

[decarlof]

By increasing the DeviceLinkThroughputLimit up to the same value as DeviceLinkSpeed, I can now get the same results you have:

Pixel format	ADC bits	Actual frame rate (ArrayRate_RBV)	Set frame rate (FrameRate)	FrameRate_RBV
Mono16	8	79	79	79
Mono16	10	84	84	84
Mono16	12	89	89	89

and for the Mono12Packed to Mono16 and AdcBitDepth=10, I only get up 79 fps:

and once it hangs at after collecting 3012 frames:

The the system settings are:

net.core.rmem_default=26214400 net.core.rmem_max=268435456

[MarkRivers]

OK, we are making progress.

The Mono12Packed mode can have another issue. The conversion from Mono12Packed to Mono16 is done in the ADSpinnaker thread that reads the frames from the camera. The conversion takes some time, 6-7 ms on my system. If the thread cannot keep up with the frame rate then the Spinnaker library will begin to drop frames. In the master branch I have exposed the setting for the number of buffers to use in the Spinnaker Transport Layer. By setting that to 100 (rather than the default of 10) I no longer drop any frames at the full 130 frames/s in Mono12Packed 10-bit mode.

That does require that your computer have sufficient fast cores to do the conversion at the full 130 frames/s.

This is what I see with the command more /proc/cpuinfo

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 85
model name      : Intel(R) Xeon(R) W-2145 CPU @ 3.70GHz
stepping        : 4
microcode       : 0x2006906
cpu MHz         : 1200.623
cache size      : 11264 KB
physical id     : 0
siblings        : 16
core id         : 0
cpu cores       : 8
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 22
wp              : yes

So I have 8 cores, 16 siblings (threads) at 3.70 GHz. What do you see?

You should try decreasing the requested frame rate when running Mono12Packed 10-bit from 130->125->120 and see when it can give you the requested rate.

[decarlof]

I am still running last week version, once updated my computer should be able to keep up more /proc/cpuinfo :

processor   : 0
vendor_id   : GenuineIntel
cpu family  : 6
model       : 85
model name  : Intel(R) Xeon(R) Silver 4116 CPU @ 2.10GHz
stepping    : 4
microcode   : 0x2006906
cpu MHz     : 2520.836
cache size  : 16896 KB
physical id : 0
siblings    : 24
core id     : 0
cpu cores   : 12
apicid      : 0
initial apicid  : 0
fpu     : yes
fpu_exception   : yes
cpuid level : 22
wp      : yes

[MarkRivers]

You have more cores but they are slower, 2.1 GHz vs 3.7 GHz.

You should try decreasing the requested frame rate and see when it can keep up.

[decarlof]

ok I can go up to 75 fps (2.1/3.7 * 130 = 73!). At 80 the camera hangs at ~4938/5000 frames.

[decarlof]

Also, when you updated from Ubuntu to RH8 did you increase the system settings described here: https://areadetector.github.io/master/ADGenICam/ADGenICam.html#linux-usb-and-gige-system-settings

@MarkRivers IT had to set other parameters to get the FLIR to work correctly. I am not sure if these are FLIR specific but if not probably should be added to the ADGenICam docs. Here they are:

1. Prerequisites: 64GB memory Cat 6A cable Intel X550T2 ETHERNET CONVERGED Network Adapter X550-T2
2. Enable jumbo packet
3. Disable DHCP and set a fixed IP address on the Ethernet port connecting to the FLIR
4. Increase the receive buffer size
5. Set the NIC tx queue length

(2. 3.). are documented at: FLIR doc: https://www.flir.com/support-center/iis/machine-vision/knowledge-base/lost-ethernet-data-packets-on-linux-systems/

(4. already documented) edit /etc/sysctl.conf and add:

net.core.rmem_default=26214400
net.core.rmem_max=268435456

(5.) edit /etc/rc.local and add:

#NIC camera settings and  10GB nic settings  In this example the camera is attached to  ens1f1    
/usr/sbin/ifconfig ens1f1 txqueuelen 3000

[MarkRivers]

I have released ADSpinnaker R3-1 and also ADAravis R2-1.

The new version of ADSpinnaker allows increasing the number of buffers used in the Spinnaker library. It also has much improved statistics reporting on the transport layer to aid in diagnosing problems. This is a screen shot when running the new version with the ORX-10G-51S5M in 10-bit mode, Mono12Packed, 130 frames/s.

Note that the input buffers is 77. I specified a total of 100 buffers, whereas in the previous release the value was fixed at the Spinnaker default of 10. This means that 23 buffers are currently in use, because the thread that reads the camera frames and does the Mono12Packed to Mono16 conversion is barely keeping up. That can also be seen with top -H

Note that the thread called ADSpinnakerImag is using 99.9% of a core. That is the thread the reads the images and does the conversion. On my 3.7 GHz machine that thread just barely keeps up, and I don't drop any frames as long I have 100 buffers.

This is a screen shot of ADAravis running the Oryx cameras with the same settings as above.

This is top -H when adAravis is running:

In this case aravisPoll is the thread that is reading the images and doing the Mono12Packed to Mono16 conversion, using functions that I added in ADGenICam R1-6. It is only using 65.5% of a core, compared to 99.9 in ADSpinnaker.

@decarlof this means that ADAravis may be able to keep up with the Oryx 10-bit, Mono12Packed on your machine. An advantage of ADAravis over ADSpinnaker is that it runs on RHEL7, it does not require RHEL8 or Ubuntu 18. Now that I have added Mono12Packed and Mono12p support to ADAravis it really has all the functions needed for mono cameras and ADSpinnaker provides no addtional benefit that I know of. However, for color cameras ADSpinnaker does Bayer to RGB conversion that ADAravis currently lacks.

[decarlof]

@MarkRivers your FLIR/Spinnaker screen shows you are using Driver version 3.0.0. I am not sure if this is related but on the 3.1.0 version installed at 2-BM the max number I get for the input buffer is still 10 (see here).

[MarkRivers]

You need to change your startup script as documented in the R3-1 notes in RELEASE.md.

The arguments in the startup script are documented here: https://areadetector.github.io/master/ADSpinnaker/ADSpinnaker.html#ioc-startup-script

[LeeYangLBLBCS]

I updated ADSpinnaker to 3.10 but I still see quite a few lost frames even at at conservative frame rate of 60fps and ADC=10 bits. (see attached screen shot that shows 4990 of 5000 frames acquired). Also the improved statistics is not available. Maybe I need to compile ADAravis to get that?

tttt.txt

[MarkRivers]

I updated ADSpinnaker to 3.10 but I still see quite a few lost frames even at at conservative frame rate of 60fps and ADC=10 bits. (see attached screen shot that shows 4990 of 5000 frames acquired). Also the improved statistics is not available. Maybe I need to compile ADAravis to get that?

Your screen shot above shows that you are running ADSpinnaker 3.0.0. You need to update to ADSpinnaker 3.1.0. You are running ADCore 3.10, which is fine.

The second problem is that you are not using the latest medm screen from ADSpinnaker 3.1.0, so you are not seeing all of the statistics.

Be sure to also change your startup script, as I noted to @decarlof above.

This is what the screen should look like. Note that the Driver version is 3.1.0 and there are additional statistics values.

[MarkRivers]

In the above screen I am collecting 5000 frames at 129 frames/s, Mono12Packed, 10-bit, converting to Mono16.

I see the input buffers start at 95 and gradually decrease during the acquisition. When it reaches 5000 frames input buffers is down to ~20. This means I likely would have run out of buffers if I had collected 10000 frames. I could avoid this by increasing the maximum number of buffers in the startup script from the default of 100.

[decarlof]

@LeeYangLBL my detector is in use at the beamline so I could not complete the test, @MarkRivers was correct after installation the old start-up script was copied over to the new version. I will update next week.

[LeeYangLBLBCS]

@MarkRivers v3.10 still drops frames. FLIR released two more versions, 2.1.0.95/2.2.0.48. I wonder if that'll work with this driver? You mentioned ADAravis driver works better compared to ADSpinnaker. Do you recommend ADArvis instead?

[MarkRivers]

Please post a screen shot of the new version.

[LeeYangLBLBCS]

screen shot with 6 lost frames: all plugin settings: Feature #2 screen shot:

[MarkRivers]

Please do it again. When it gets to about 4500 frames complete take the screen shot of just the main ADSpinnaker screen. What value does the Input buffers have at that point?

[LeeYangLBLBCS]

here is at 4567 frame:

[MarkRivers]

At the end of that acquisition how many images were complete? It looks like InputBuffers never drops below 100?

You are only trying to do 80 frames/s. As my screen shot above shows with 10-bit Mono12Packed I can do 130 frames/s and not drop any frames.

[LeeYangLBLBCS]

it lost 16 frames at the end. this is the end of acquisition: ![Uploading image.png…]()

[LeeYangLBLBCS]

last message seems to be stuck at "Uploading image". Uploading again:

[MarkRivers]

What happens if you try to run at 130 frames/s?

It looks like you are getting a lot of FailedPackets. That indicates a problem at the lower-level Ethernet interface. I don't have any failed packets in my screen shot.

[LeeYangLBLBCS]

maybe there is something with the brand of the computer we are using (supermicro): CPU is 6 core, 3.5GHz. What computer are you using? I wonder if we should get the same one. cpuinfo.txt

[LeeYangLBLBCS]

at 112 fps, lost more frames, ~220 frames.

[MarkRivers]

Note that in your first screen shot above InputBuffers is only 2, while it started at 100. The was when it was acquiring because Image rate=108. This strongly suggests that you are dropping frames because you are running out of input buffers. Here are things to do:

• Run camonitor on InputBuffers while you are acquiring. See if the value is dropping
• Run "top -H" while you are acquiring. When you get to 4000 frames complete capture the top -H screen and post it. We can see if any threads are using 100% CPU.
• Increase input buffers in the st.cmd file to 1000 and see if it can complete 5000 frames OK.

[decarlof]

I set the numSPBuffers to 100 with mono12Pack, 10bit and I run out of input buffer and loose frames if I set frame rate above 80.

I increased the buffer to 400 and get:

and at the end

I never see the failed packet counter to increase, just at the end I get 4178/5000 frames.

If I set buffer > 400 the IOC crashes with:

epicsThreadSleep(1.0)
epics> /net/s2dserv/xorApps/PreBuilts/areaDetector-R3-10/ADSpinnaker-R3-1/iocs/spinnakerIOC/iocBoot/iocSpinnaker/softioc/2bmbSpinnaker.sh: line 185: 2932845 Segmentation fault      (core dumped) ${IOC_CMD}

I think is because my MemTotal: 64460268 kB

[MarkRivers]

Hi Francesco,

Your screen shot above shows that InputBufferCount has decreased from 400 down to 2. Can you repeat the test with 400 input buffers, but this time run this command in another window:

camonitor 2bmbSP1:cam1:NumImagesCounter_RBV 2bmbSP1:cam1:InputBufferCount

Then we can see how the number of free buffers changes as it is acquiring.

Can you also run "top -H" in another window and capture that screen when it acquiring?

[decarlof]

Hi Mark, I restarted the camera with 400 input buffer and start the monitoring, first in 10-bit /Mono12Pack/80fps and I could see the InputBufferCount stabilizing around 200.

2bmbSP1:cam1:InputBufferCount  2020-11-03 16:54:59.135727 215  
2bmbSP1:cam1:NumImagesCounter_RBV 2020-11-03 16:54:59.147795 982  
2bmbSP1:cam1:InputBufferCount  2020-11-03 16:54:59.147804 216  
2bmbSP1:cam1:NumImagesCounter_RBV 2020-11-03 16:54:59.182009 983  
2bmbSP1:cam1:NumImagesCounter_RBV 2020-11-03 16:54:59.217220 984  
....
2bmbSP1:cam1:NumImagesCounter_RBV 2020-11-03 16:56:01.897469 2769  
2bmbSP1:cam1:NumImagesCounter_RBV 2020-11-03 16:56:01.932332 2770  
2bmbSP1:cam1:NumImagesCounter_RBV 2020-11-03 16:56:01.967699 2771  
2bmbSP1:cam1:NumImagesCounter_RBV 2020-11-03 16:56:02.003755 2772  
2bmbSP1:cam1:NumImagesCounter_RBV 2020-11-03 16:56:02.037956 2773  
2bmbSP1:cam1:NumImagesCounter_RBV 2020-11-03 16:56:02.072963 2774  
2bmbSP1:cam1:NumImagesCounter_RBV 2020-11-03 16:56:02.107975 2775  
2bmbSP1:cam1:NumImagesCounter_RBV 2020-11-03 16:56:02.143357 2776  
2bmbSP1:cam1:NumImagesCounter_RBV 2020-11-03 16:56:02.178097 2777 

I then, before increasing the fps, I wanted to see the same parameters when running in 8-bit/160fps but I end up with the camera in strange state. I can now only run it a very slow speed (~30fps). I have seen this in the past and solved it by adjusting ISP Enable parameter or increasing the DeviceLinkThroughputLimit. (see warning for more details) but now nothing seems to work ... so I stock at 30fps even if I set 100fps and I know the camera can do 160. Here are the current settings:

[MarkRivers]

You are only getting 30 fps because your AcquireTime is 0.035. You need to change it to 0.001 for testing.