Open mhier opened 3 years ago
A theory how this bug happens:
When using the Logical Name Mapper, it may happen that the same PcieBackend instance is used in two LNM devices. In ApplicationCore, this connection is not known. Hence the two DeviceModules might attempt to recover the same PcieBackend concurrently (indirectly through the logical device).
Since open() is not considered to be thread safe, this is not allowed. On the other hand, the application has no way of knowing about this entanglement. Not sure how to best solve this problem. Either ApplicationCore (and basically any application) has to make sure, no device is concurrently opened/recovered with any other device, or we have to change the requirement and expect open() to be thread safe.
Note: The logical name mapping backend cannot fix this. One of the usages could be direct, without a LNM backend in between, so it is impossible to know for the LNM backend if a concurrent open() is currently in progress.
As the problem you describe only can happen when using the LNM, we could require that you have to use LNM for all devices in your application. Then we could build it into the LNM.
Or we put this task to each application and put the same mechanism into ApplicationCore. The call to open() in the DeviceModule could be surrounded by a global mutex, which makes all recoveries/initialisations sequential.
I don't like the first option (force using LNM for all devices), since this can easily be forgotten and somehow contradicts our principle of abstraction.
Due to another bug, I had a server in a "recovery loop": The devices were switched to error state via setException() and then recovered again via open() in an endless loop. Occasionally (quite rare actually), the a crash "double free or corruption (fasttop)" happened with the following backtrace:
The server was the llrfctrl server at A0M, and it was stuck in the recovery loop because one ADC board was powered down through the MCH.