FairRootGroup / DDS

The Dynamic Deployment System
http://dds.gsi.de
GNU Lesser General Public License v3.0
14 stars 14 forks source link

ToolsAPI: add SHM clean up on session end #404

Open AnarManafov opened 2 years ago

AnarManafov commented 2 years ago

Extend Tools API with a possibility to register shared memory names. Those SHM segments must be removed, if exists, on DDS session shutdown, i.e. on the agent's clean up.

rbx commented 2 years ago

In case of FairMQ it is not as simple as giving a list of names. We use a number of different boost shared memory objects of different types.

Current list is:

name type
fmq_<shmId>_m_<segmentId> boost::interprocess::shared_memory_object
fmq_<shmId>_mng boost::interprocess::shared_memory_object
fmq_<shmId>_mtx boost::interprocess::named_mutex
fmq_<shmId>_cv boost::interprocess::named_condition
fmq_<shmId>_rg_<index> boost::interprocess::shared_memory_object OR boost::interprocess::file_mapping (user configured)
fmq_<shmId>_rgq_<index> boost::interprocess::message_queue
fmq_<shmId>_ms boost::interprocess::named_mutex

(The shmId is derived from the current FairMQ session id.)

Each of these types have their own removal methods according to boost docs, for example shared_memory_object is removed via shared_memory_object::remove(), but named_mutex is removed named_mutex::remove(). It may well be that the underlying implementation is the same for all of them, but I would prefer to use correct methods, to be robust against changes in future boost versions.

I tested if all of the above can be removed simply via boost::interprocess::shared_memory_object::remove(). And it works for all except the mutexes, possibly simply because boost prefixes the mutex files with sem. - sem.fmq_<shmId>_mtx.

Alternatively, do not call any boost API to remove these objects at all (unless you know which type is which somehow...), and simply rm -f them. However, here special knowledge would be necessary to know that boost prefixes some of these files, like in mutex case.

Another point is - where exactly should this registration happen? FairMQ core has no dependency on DDS Tools API, and this registration certainly should not always be done, but only in cases where it is desired. One option would be to make it a configurable setting in the ODC plugin. FairMQ core shmem could provide a static list of names, ODC plugin would read this list and give it to DDS and DDS would then call rm for each.

In my opinion the cleanest way would be to call fairmq-shmmonitor --cleanup --session <sessionId>, which knows what methods to call for which objects, instead of providing a list of names or unknown type. To remain robust, you can treat it as a user task and force kill it if it misbehaves.

Whatever the solution will be, it must be flexible enough that a new FairMQ version can change the names, types and the number of these files if it needs to, without requiring update anywhere else.

rbx commented 2 years ago

Alternatively, do not call any boost API to remove these objects at all (unless you know which type is which somehow...), and simply rm -f them. However, here special knowledge would be necessary to know that boost prefixes some of these files, like in mutex case.

Actually a static list will not work, because the sessionid/shmid us dynamic, but also the number of segments and regions and their IDs will vary depending on runtime configuration.

A regex/wildcards would do the trick, something like rm -f /dev/shm/*fmq_<shmid>*.