IPC::ShareLite store() error: Operation not permitted with mysql_ on fedora33

[dwreski]

Hi, I have a fedora33 install with mariadb-10.4.17 and munin-node-2.0.63 and trying to use the mysql_ plugin.

munin-run mysql_connections
IPC::ShareLite store() error: Operation not permitted at /usr/share/perl5/vendor_perl/Cache/SharedMemoryBackend.pm line 156.

This is effectively the same outcome for all plugins I've linked.

# pwd
/etc/munin/plugins
# ls -l mysql*
lrwxrwxrwx 1 root root 31 Dec  6 10:37 mysql_connections -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root 31 Dec  6 10:37 mysql_innodb_log -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root 31 Dec  6 10:37 mysql_innodb_queries -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root 31 Dec  6 10:38 mysql_max_mem -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root 31 Dec  6 10:37 mysql_network_traffic -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root 31 Dec  6 10:37 mysql_performance -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root 31 Dec  6 10:37 mysql_slow -> /usr/share/munin/plugins/mysql_

The following are my /etc/munin/plugin-conf.d/00-default settings for mysql:

[mysql*]
   env.mysql /usr/bin/mysql
   env.mysqlopts -u munin --password=muninpass --host=localhost
   env.warning 2147483648
   env.critical 1073741824
   env.mysqluser munin
   env.mysqlpassword muninpass
   env.slave_io_running_warning 0.5
   env.slave_sql_running_warning 0.5
   env.seconds_behind_master_warning 3600
   env.seconds_behind_master_critical 86400
   env.wsrep_flow_control_paused_ns_warning 200000000
   env.wsrep_flow_control_paused_ns_critical 10000000000
   env.cachenamespace munin_mysql_pri
   env.mysqlconnection DBI:mysql:information_schema;host=127.0.0.1;port=3306

What more do I need to do to make this plugin work?

[sumpfralle]

Thanks for your report!

Did you try running the plugin via munin-run mysql_connections? Did you also try munin-run --ignore-systemd-properties mysql_connections?

In case the latter works (and the former fails): maybe the execution environment of your munin-node service restricts shared memory? (e.g. RemoveIPC=yes)

[dwreski]

Thanks so much for your help.

It seems like it produces the IPC problem every third of fourth time I run "munin-run mysqlconnections" (or any of the other mysql plugins). It apparently doesn't happen every time.

It also still occurs with the --ignore-systemd-properties. It requires ctrl-C to quit or it hangs indefinately.

# munin-run --ignore-systemd-properties mysql_connections
IPC::ShareLite store() error: Operation not permitted at /usr/share/perl5/vendor_perl/Cache/SharedMemoryBackend.pm line 156.

Here is the contents of the munin-node systemd service file.

# cat ./system/multi-user.target.wants/munin-node.service
[Unit]
Description=Munin Node
Documentation=man:munin-node(1) http://guide.munin-monitoring.org/en/latest/node/index.html
After=network.target network-online.target
PartOf=munin-asyncd.service

[Service]
Type=notify
ExecStart=/usr/sbin/munin-node --foreground
PrivateDevices=no
PrivateTmp=yes
ProtectHome=read-only
ProtectSystem=full
TimeoutStopSec=30s

[Install]
WantedBy=multi-user.target

[sumpfralle]

It seems like it produces the IPC problem every third of fourth time I run "munin-run mysqlconnections" (or any of the other mysql plugins). It apparently doesn't happen every time.

that sounds interesting (in the weird way) :(

Thus we can rule out a configuration issue. Instead it seems to be some kind of resource usage collision.

Sadly I have no good understanding of the SharedMemoryBackend module. Thus I need to guess ...

Maybe multiple instances of this plugin are competing for a single resource or lock?

Maybe you could take a look at this discussion at serverfault? Does the recommended switch to the Cache::FileCache module fix your problem?

[dwreski]

Hi, thanks for your continued support. I also saw this serverfault post, but wanted to be sure it wasn't otherwise fixed with an upstream version, etc. I've made the change to use FileCache instead of SharedMemoryBackend and it fixed it. That was from five years ago - I really thought there were more people using these plugins, particularly the ones that are included with the munin-node distro RPM.

There's also the other mysql_* plugins that are also included, but don't seem as capable. Makes it confusing for the end-user, though.

The only problem with the mysql_ plugin now is with mysql_innodb_queries producing no value, but I suspect that's another problem. I still have to troubleshoot that.

[github-actions[bot]]

Stale issue message

[kimheino]

It seems like it produces the IPC problem every third of fourth time I run "munin-run mysqlconnections" (or any of the other mysql plugins). It apparently doesn't happen every time.

This sounds like SElinux problem. Can you check your audit.log file, or try to run it permissive mode?

You can also run "ipcs --shmems" and "ipcs --semaphores". Does it work when those lists are empty, and doesn't work when there is something owned by munin?