search

areaDetector / ADGenICam

areaDetector base class for GenICam cameras.
https://areadetector.github.io/master/ADGenICam/ADGenICam.html
7 stars 16 forks source link

PVs take too long to become responsive #37

Closed henriquesimoes closed 8 months ago

henriquesimoes commented 8 months ago

Hi, I've been working with updating IOCs based on the deprecated aravisGigE to the new ADGenICam and ADAravis support modules. During initial tests with the new modules, I noticed a significant delay time to get PVs responsive (regarding their set-point and read-back values) at IOC boot. Have you noticed this already?

To get an insight on what was going on, I run the IOC under Linux Perf profiling tool. From about 106 seconds the driver thread (the one named after the port name) takes to start up, about 57 secs (~53%) is collectively taken by readAll calls, called both by writeInt32 and writeFloat64 functions.

The side-effect of this is having the parameter library frozen (due to the global lock being taken?) and PVs being unresponsive for about a minute. I'm thinking these write calls are being triggered during the database initialization, leading to processCallbackOutput calls. However, I'm not certain of that. Does it make sense?

As far as I understand, the rationale here to readAll after these writes is to synchronize parameter library with the camera registers, since its firmware may update one register based on the other, right? Could we delay them until all these sequential writes are performed or coalesce readAll requests to minimize these synchronizations and improve performance?

I may provide more information about the setup I tested it and the profiling data I gathered, if needed.

I'd like to work on this to have it improved. Some insights on the best path to go are pretty much appreciated.

MarkRivers commented 8 months ago

I have not observed that behavior recently. I did observe it with an old version of aravis before it implemented register caching. What version of aravis are you using, and on what camera?

henriquesimoes commented 8 months ago

I'm using Aravis 0.8.6 (packaged for Debian 11). It does include register caching, but I now find I'm not enabling it in my iocsh script in aravisConfig. Thanks for the note.

I'm going to test it with this enabled to see the effect. The camera I'm testing with is a Basler acA1300-75gm (version 106755-13). I see the note in the arv_device_set_register_cache_policy has a warning about register caching metadata not always being reliable. I'm also going to check it for this camera.

Thanks!

MarkRivers commented 8 months ago

Have you been able to test this?

henriquesimoes commented 8 months ago

Yes, but not so thoroughly as I'd like yet. Caching does reduce the start-up time significantly.

Basic functionality seems fine with a camera I could test with. Thanks for the help.

MarkRivers commented 8 months ago

I'm going to close this issue, since I think feature caching was the problem.