Read permission problems cause false `inotify` warning upon startup

[Tek4]

I'm suddenly getting inotify warnings upon reboot/startup: For the past year I have experienced no issues after increasing inotify to 1048576: > echo 1048576 > /proc/sys/fs/inotify/max_user_watches > cat /etc.defaults/sysctl.conf ... fs.inotify.max_user_watches = 1048576

I have now increased both to 16777216 (a 16x increase) and I'm still getting warnings!

Any ideas?

Synology DS918+

[jlesage]

Are you sure is properly applied? You can verify with cat /proc/sys/fs/inotify/max_user_watches.

[excalibr18]

If you figure out how to make it stick and what is an appropriate number, please report back. As I'd be interested in your solution. I have set mine to 1048576 and it persists through a reboot of my Synology 1515+ but I still get the iNotify watch limit warning at the beginning. Grated I have a 4.5 TB backup set, and dismissing the warning doesn't seem to affect the nightly scan and backup to CrashPlan.

[Tek4]

jlesage Yes, it is applied and persists. I figured that part out a year ago, and have had to reapply since after one or two firmware updates.

Yesterday, I took my system down for spring cleaning (compressed air dust removal) and got the warning upon reboot. I increased 16x and I'm still getting the warning.

Like excaliber18 reports, CrashPlan appears to be working, but my confidence in real-time file watching is now lacking. I could increase further, but 16M seems quite high as it was working previously with 1M.

Thanks.

Kevin

[Tek4]

No resolution so far. I changed inotify to an absurdly large number (2^28) and I still get the inotify warning.

Despite the warning, CrashPlan logs-in and appears to be backing up, but it's unclear if there is a file watching problem.

[jlesage]

You can look at log/service.log.0. You should see a log related to the inotify limit. Maybe you could find more details?

[bigtfromaz]

A 16x increase seems large. Try dropping it to 2x. Perhaps the 16x number is being rejected leaving you with a default. You might want to count the number of files in the folders that are being backed up to get a good feel for what the minimum count should be.

[Tek4]

The only thing obvious in log/service.log.0 are several lines similar to this one:

[03.21.19 06:21:44.687 WARN Thread-28 ode42.jna.inotify.InotifyManager] Unable to add watch for path /volume1/.../0, errno: , 13

I'll dig further as I get time...

[jlesage]

Errno 13 is a "permission denied" error. Are you sure the container has read access to this file?

[Tek4]

My particular issue is related to Synology Active Backup for Business, which creates a folder /@ActiveBackup/@data.

The @data folder is the cause of the problem. Despite granting read access to the top-level folder, CrashPlan's service.log.0 reflects permission errors such as:

[03.30.19 02:20:26.173 WARN Thread-28 ode42.jna.inotify.InotifyManager] Unable to add watch for path /volume1/ActiveBackupforBusiness/@ActiveBackup/@data/Pool/0, errno: , 13 [03.30.19 02:20:26.174 WARN Thread-28 ode42.jna.inotify.InotifyManager] Unable to add watch for path /volume1/ActiveBackupforBusiness/@ActiveBackup/@data/Composition/0/0.com, errno: , 13

WHY DO THESE FILE PERMISSION ERRORS RESULT IN INOTIFY WARNINGS? I HAVE NO IDEA.

Solution 1: You can exclude the @data folder in your backup selection:

Solution 2: Perhaps a bit more sexy... You can add a file exclusion to your backup set Settings:

Hope this helps someone in the future.

[Tek4]

@excalibr18 Since it appears file permission errors trigger inotify warnings, I highly suggest you open up /docker/appdata/crashplan-pro/log/service.log.0 in a text editor and search for " WARN " (spaces on each side). See if you are getting the Unable to add watch for path ... warnings. If so, resolving those warnings was the fix for me.

It's helpful to Stop the CrashPlan container, give it a minute to shut down, the restart it. This will append a fresh spate of startup messages to the log file. Then you can open the log file, put the cursor at the bottom (Ctrl-End), and search "Previous" to find the most recent " WARN " entries.

[excalibr18]

@Tek4 Thanks for the tip. I'm still at a loss. I don't have Synology Active Backup for Business, so that part doesn't apply to me, and I'm not including Synology system folders in my back-up set - only shared folders.

I followed your directions and stopped the container and restarted it then searched through /docker/appdata/crashplan-pro/log/service.log.0. I found that the paths that are showing up with the Unable to add watch for path ... warnings are paths with non-system files like PDFs and word files, and the paths were innocuous like a backup of My Documents, etc.

I confirmed that my entire back-up set only has 707779 files in it. And I confirmed that I set the fs.inotify.max_user_watches = 1048576 in /etc.defaults/sysctl.conf and that the output of cat /proc/sys/fs/inotify/max_user_watches is 1048576.

@jlesage Since Errno 13 is a "permission denied" error, I confirmed that the UID and GID associated with the USER_ID and GROUP_ID environment variables when creating the container is an admin user that has read and write access to those folders. When I check folder ownership, I see that those folders are owned by users other than the user tied to the container, but again I confirmed that the user for the container has read and write access to the folder. I should note that there are other folders owned by that same user that are NOT triggering the warning.

[Tek4]

@excalibr18 Irrespective of the Synology software, I think you found the reason for the problem, and it is essentially the same as my issue. I probably should rename the thread to "Permissions problems causing inotify warnings".

On my system as well, the user/group has permission for the rejected folders. Some sort of file locking or other mechanism must be in play here. I didn't dig beyond excluding the problematic folders, but I'm starting to have some concerns about that "metadata" (I'm presuming) being essential to restoring backups.

Hence, I'm very interested in furthering the investigation here.

Please do post back if you discover anything.

[Tek4]

Information only: I performed a file count and discovered I was previously running with no inotify warnings despite having over 1M files. I'm not convinced a 1:1 ratio is required. Nevertheless, I raised the max to 2M.

Could this be a memory issue? I noted memory usage spikes upon startup to over 6GB. I've only dedicated 4GB to CrashPlan since my system is maxed at 8GB installed RAM. Perhaps we should try increasing the RAM allotment?

[excalibr18]

@tek4 my Synology has 16GB of RAM and I have 8GB allocated to the container. I've only seen the container max out around 6GB for me, so I don't believe it's a memory issue (at least in my case). I'll try creating the container under root to see if I still get the inotify warnings and will report back.

[jlesage]

I found that the paths that are showing up with the Unable to add watch for path ... warnings are paths with non-system files like PDFs and word files, and the paths were innocuous like a backup of My Documents, etc.

@excalibr18, did you see that these warning messages end with the errno: 13 ?

[excalibr18]

@jlesage: Yes, that is correct.

[jlesage]

So that's definitely a permission issue. Note that have permission to read/write a folder doesn't guarantee to have the same permissions on the files within this folder. If you need help, you can provide an example of permission on a problematic file:

ls -l <path to the file>

[excalibr18]

Update: I confirmed that creating the container with the USER_ID and GROUP_ID environment variables set to 0 (root) doesn't trigger any inotify watch warnings.

I also changed the owner of the problem folders, sub-folders, and files to match the user_id of other folders that don't have inotify warnings. This seemed to work for all the problem folders except for one /volume1/documents_1/#recycle folder which is owned by root. Mind you I have /#recycle folders for each shared folder and those other folders are also owned by rootand aren't triggering the inotify watch warning.

I compared the ownership and access rights using ls -l /volume1/documents_1/#recycle to another /#recycle folder that doesn't trigger the inotify watch warning and couldn't find any difference. All are owned by rootand have drwxrwxrwx+ as permissions.

Thankfully this is just a /#recycle folder and isn't really integral to the back-up set at all. Still I wonder why it's just this one folder when the others like it aren't a problem.

[Tek4]

@excalibr18 I think we've confirmed one problem:

Permissions problems cause "false" inotify warnings

Whether or not these "false" inotify warnings kill file watching is unknown to me. Normally, when CrashPlan emits an inotify warning, it's because max_user_watches is set too low and CrashPlan cannot create enough watches for all the files/folders in your backup sets. I believe, in this case, CrashPlan reverts to file polling (continually checking files for changes), which is much less efficient than a "watch notification" from the operating system when a file is changed. When "false" inotify warnings result from a permissions error (and not from truly exceeding the max_user_watches limit), it's unclear if CrashPlan reverts to polling or not.

I'm experimenting to see if I can get to the bottom of the permissions problem: Why is my CrashPlan user account getting denied permission to sub-folders for which the account has read permission? FYI—My CrashPlan user account has no explicit permission, but is a member of a backup agents group that has permission.

Choosing the right value for max_user_watches

I'm am also experimenting with this setting. The errors I found in the log lead me to believe CrashPlan watches directories, not files. Thus, I suspect the proper metric is to do a folder count instead of a file count. I believe I have max_user_watches set to a value which is far higher than necessary.

Hence, I performed a file and folder count and set the value to about half the number of folders in my backup sets. I am expecting this to cause a warning, but—so far—it has not! Interesting.

[EDIT: After a few minutes, CrashPlan did emit an inotify warning. I have now increased the value to be about 20% higher than the total number of folders in my backup sets. This number is about one-third of the total number of files, so if CrashPlan is watching folders I should have no errors; but, if CrashPlan is watching individual files I should get another inotify warning.]

I'm also trying to notice if less memory is used by the CrashPlan container with the lower value, but the memory usage appears about the same.

Therefore, it appears the value is merely a maximum limit and actual memory usage seems to occur when a watch is created. It would seem an arbitrarily high value does not increase memory usage.

Thus, I haven't discovered anything which leads me to believe setting max_user_watches to an unnecessarily high (or even ridiculously high) maximum value has any negative effects. It's unclear to me why the operating system imposes any limit to begin with.

[Tek4]

Follow-up and Recommendation regarding max_user_watches:

After several hours running CrashPlan, I am strongly confident max_user_watches should be set based on the total number of folders (not files) in your backup set.

I currently have ≈240,000 folders and over 1 million files. I ran CrashPlan with max_user_watches set at 131072 (2^17) and got an inotify warning after a few minutes. I then increased max_user_watches to 300000 and have been running CrashPlan for several hours with no warnings.

My recommendation, then is to set max_user_watches for a value that leaves more than ample room for expansion. In my case, I'm going with 1048576 (2^20 or 1 Mi (mebi)), which allows for 4x growth. I found no reason to be stingy with this value as I noticed no increase in memory usage with even an unreasonable setting such as 256 Mi (in my case).

I do lament CrashPlan's general memory appetite. In my case, CrashPlan hogs a good 4 to 6 GiB of RAM and >25% of CPU time, which I find wholly unreasonable.

I'd still like to get to the bottom of the user permissions (folder access) problem. Hoping for some additional help and/or ideas.

[jlesage]

@Tek4, is /volume1/ActiveBackupforBusiness/@ActiveBackup/@data/Pool/0 a file or a folder?

The pop-up can certainly be fixed: when inotify fails to be setup because of a permission issue, a different message should be presented to the user. Currently the message is confiusing since increasing the limit won't help to solve the issue.

[Tek4]

@jlesage, it's a folder. All of the similar log entries pertain only to folders.

The mystery is, what is causing permission denied errors? Everything I have checked indicates my CrashPlan user/group should have access.

Second, is CrashPlan reverting to file polling instead of watches when these "fake" warnings occur?

[Tek4]

FAILED experiments (attempting to find cause of permission denied error):

• Reduced CrashPlan access from rw to ro
• Stopped Synology ActiveBackupforBusiness service (thinking there might be some sort of open file or lock)

Looking closer at directory permissions... here is ls -leah on the directory causing errors:

Here is ls -leah on a directory that causes no errors:

I'm still at a loss. I'm thinking maybe I should take a snapshot/shadow copy of this folder and see if I can get CrashPlan to backup the snapshot copy instead of the original.

[asucher]

Any progress / updates on this? I think I am seeing something similar.

[jlesage]

What are you seeing exactly? The only action item on this issue is the fix for the incorrect pop-up message.

[asucher]

What are you seeing exactly? The only action item on this issue is the fix for the incorrect pop-up message.

Not sure if this was directed to me...but...I also have run into iNotify warnings seemingly no matter what I set my max watches too. Based on this thread I reviewed the logs and found that in fact there are numerous errno13 errors even though I have the recommended USER_ID and GROUP_ID environment variables configured.

[jlesage]

Even if you configure USER_ID and GROUP_ID, you can have files with ownership/permissions that make them unreadable by CrashPlan. You likely need to be review and fix them.

[asucher]

Even if you configure USER_ID and GROUP_ID, you can have files with ownership/permissions that make them unreadable by CrashPlan. You likely need to be review and fix them.

That's what is odd about it. Similar to what Tek4 described the account that is used shows as having read rights to the folders that are showing the error. If I run the container as root the errors go away.

[Tek4]

@jlesage Just as @asucher described, there are certain folders (no files in my case) that cause error 13 warnings (Unable to add watch for path...).

My investigation into setting max_user_watches did not resolve the issue.

I have been unsuccessful in trying to ascertain why the error 13 occurs (see above posts).

[jlesage]

Are you sure that the user has also read permission on all the parent folders?