max_user_watches cannot be changed in Linux LX zone

[jcroix]

I discovered this when attempting to install CrashPlan in a Ubuntu LX zone in SmartOS. I received a message saying that the default value of 8192 should be increased, and the web page associated with this message indicated that a substantial increase might be called for (see http://support.code42.com/CrashPlan/Latest/Troubleshooting/Linux_Real-Time_File_Watching_Errors). Unfortunately, as I discovered, this value cannot be changed in an individual zone, only at the system level.

[simmel]

@jcroix do you remember what key, or rather the namespace, was to set this in the global zone?

# echo "set inotify_maxwatches = 1048576" | mdb -kw
mdb: failed to dereference symbol: unknown symbol name

[jjelinek]

"inotify_maxwatches" is the correct variable name. It is likely that your mdb command is failing because the "inotify" module is not currently loaded into the kernel. That module will dynamically load as-needed but could have also been unloaded if there was nothing using inotify.

[simmel]

Wow, thanks for the speedy response @jjelinek! Like @jcroix I'm trying to run CrashPlan which tries to watch every file that it backups via inotify and since I have more files than the default 8k it's failing:

WARN  Thread-24    ode42.jna.inotify.InotifyManager] Unable to add watch for path /data/backup/some-file, errno: , 28

28 seems to be ENOSPC which according to inotify_add_watch(3c) means The number of watches on the specified instance would exceed the maximum number of watches per inotify(5) instance.. So it looks like it wants to use inotify but I can't see the module loaded or load it (if that's the right syntax) when starting mdb:

# mdb -k
Loading modules: [ unix genunix specfs dtrace mac cpu.generic uppc pcplusmp scsi_vhci ufs ip hook neti sockfs arp usba stmf_sbd stmf zfs mm sd lofs idm sata crypto random cpc logindmux ptm kvm sppp nsmb smbsrv nfs ]
>>
> ::load inotify  
mdb: no module 'inotify' could be found
# uname -a
SunOS nadesico.waza.se 5.11 joyent_20161208T003707Z i86pc i386 i86pc

I've never used mdb before so I have no idea what I'm doing really = )

[jjelinek]

You can check if the module is loaded into the kernel by running "modinfo | grep inotify". If you see nothing then the module is not loaded. You can load the module into the kernel by running "modload /usr/kernel/drv/amd64/inotify".

[rmustacc]

Please note that if the module loads and unloads, then the value will be reset.

[simmel]

# modinfo | grep inotify
228 fffffffff8306000   3078 288   1  inotify (inotify support)
# echo "set inotify_maxwatches = 1048576" | mdb -kw
mdb: failed to dereference symbol: unknown symbol name

= / Any more ideas?

[rmustacc]

The mdb syntax isn't correct, which is why you're getting that. The syntax there is for the /etc/system file. You would want the following:

mdb -kwe 'inotify_maxwatches/W 1048576'

[simmel]

Sorry for the late reply, that was the correct syntax @rmustacc!

Though, according to mdb(1) the default value is hex, so to get it set to 1048576 I had to use the 0T prefix:

# mdb -kwe 'inotify_maxwatches/W 0T1000000'
inotify_maxwatches:             0x2000          =       0xf4240
# mdb -kwe 'inotify_maxwatches/D'
inotify_maxwatches:
inotify_maxwatches:             1000000
# zlogin c120b18d-cd01-ec8c-9164-caec5719c7cc
lx-machine:~# cat /proc/sys/fs/inotify/max_user_watches
1000000

I agree with @jcroix that it would be nice to be able to set per zone.