Corrupted image from the detector

[hinxx]

I'm trying to figure out what is happening to the image data from the FLIR blackfly S BFS PGE 70S7M detector. Every now and then there seems to appear a variable amount of rows of corrupt pixels that can be observed in the captured images. Detector is producing 3208x2200 image in Mono16 format. For these tests I've used the built-in test pattern generator of the detector. Similar results were observed without using test pattern, with the actual images taken.

Corrupted:

Fine:

These were produced by a BOB OPI that shows the StdArray ArrayData in the image widget. I can reproduce bad captures quite easily ; takes a couple of acquisitions to get a bad one. That is with single image mode. If running in continuous mode, 1 sec acquire period the I can acquire 100+ images and all would be OK (the bad ones are more rare, but they do appear).

The color pattern and image ticks are kind of screwed up ; that is probably phoebus issue.

In order to eliminate the widget / BOB OPI / phoebus, I created a file saving plugin (FileRaw) that dumps the raw pixel data into a file (top right OPI on the above screenshots). The result of these dumps was then loaded into gimp2.10 that can handle raw image data with 16bit pixels.

Here is the result:

This makes me think the problem is not in the phoebus/OPI.

Next, I used arv-camera-test test CLI tool from aravis-0.8.1 to see if I can capture the same faulty rows. I added the pixels data dump to that utility as well, to create raw binary files (same as with FileRaw plugin).

I ran 10 acquisitions, 1 second apart, 5x times (50 images in total). Here is the CLI output of one acquisition for reference:

Looking for the first available camera
vendor name           = FLIR
model name            = Blackfly S BFS-PGE-70S7M
device id             = 20177339
image width           = 3208
image height          = 2200
horizontal binning    = 1
vertical binning      = 1
payload               = 14115200 bytes
exposure              = 2003 µs
gain                  = 0 dB
gv n_stream channels  = 1
gv current channel    = 0
gv packet delay       = 1000 ns
gv packet size        = 8985 bytes
image 14115200 bytes, pixel format 01100007
pixel format ARV_PIXEL_FORMAT_MONO_16
wrote 14115200 bytes to file 1600947290_90019.data
  1 frame/s  -    14.1 MiB/s
Completed buffers = 1
Failures          = 0
Underruns         = 0

I loaded the files from these acquisitions into gimp as well. None of them showed any faulty rows.

I'm not sure what to conclude on where the issue is. I've never had EPICS / AD behave like this with other detectors. What is different for me is this particular detector itself and the latest ADGenICam based ADAravis, aravis 0.8 software stack I'm using these days. Sadly, I have not used this software stack with any other detector so far hence I'm not sure if this is at all an AD issue.

It seems that the aravis transport is fine from what I've tested so far. That being said, I'm not sure how EPICS / AD could have messed up the image pixel data since the raw data plugin is not even moving data out of IOC, no PVs involved, not network.

Any pointers on how to narrow the origin of the faulty rows is greatly appreciated.

[MarkRivers]

Are there any error messages on the IOC console?

You did not show the upper-right corner of the ADAravis OPI screen. That will show statistics on packet and frame errors. Are any of those non-zero? What is the StatusRate.SCAN field shown above the statistics?

[hinxx]

No errors in the IOC.

No errors in the OPI :

Number of complete frames is the same in the driver and in the plugin, nothing is dropped in IOC.

A little gif about the rate of bad images in relation to the acquisition rate:

Might not be obvious due to gif frame rate, but it seems that higher the acquisition rate more bad images arrive.

The NIC is using jumbo frames, as is the detector:

[hinxx]

I went and tried to set the Jumbo packet to 9000 (again!) and this time the detector accepted 9000; previously it responded with 8985 as seen in the previous comment.

Running the acquisition now I do not see any bad frames at 5 Hz. So far so good..

Not sure why the device wanted 8985 in the first place, I know I have not requested that weird value..

[MarkRivers]

Try going back to 8985 and see if the problem re-appears.

[hinxx]

Yes, it is instantly worse; back to bad frames.

[MarkRivers]

OK, that's good that you can reproduce the problem.

[MarkRivers]

@hinxx can we close this issue?

[hinxx]

Lets close it. My user reports major improvement after a day.

[hinxx]

BTW, thank you for your time!