Sync fails: "too many open files" (original matter addressed but bulk upload introduced another w/ similar symptoms)

[gcala]

Hi, sync seems broken on my laptop. It fails with a series of "Too many open files" messages in the client. It stops for a moment and then restarts the whole process from beginning.

I don't know if this error is related to desktop client or machine where it runs.

Any idea? Thanks

Client configuration

Client version: 2.5.1 git (rev. b37cbe)

Operating system: Arch Linux

OS language: Italian

Qt version used by client package (Linux only, see also Settings dialog): 5.12.0

Client package (From Nextcloud or distro) (Linux only): community/nextcloud-client 2.5.1-2

Server configuration

Operating system: Arch Linux

Web server: apache 2.4.37

Database: mariadb 10.1.37

PHP version: 7.3.0

Nextcloud version: 15.0.2

Logs

1. Client logfile: https://cuteworks.it/nextcloud/s/KzWbigWWwNjRk2n

2. Web server error log: no errors

3. Server logfile: no errors

[benjamin-leiding]

I have the same issue on my Linux system and it seems to be only triggered when I process larger numbers of very small files, e.g., small pictures or git-repositories. Client 2.3.3 works, so it has to be a problem that was introduced with version 2.5.x. I also tried the dev-branch for the nextcloud-client -> same issue.

Client configuration

Client version: 2.5.1 Operating system: Xubuntu 18.04 LTS OS language: English

Server configuration Managed Nextcloud (Owncube.com) - Version: 15.0.0

Logs Web server error log: to my knowledge -> no errors Server logfile: to my knowledge -> no errors Client logfile: 20190109_1148_owncloud.log.0.gz 20190109_1200_owncloud.log.0.gz 20190109_1200_owncloud.log.1.gz 20190109_1212_owncloud.log.0.gz 20190109_1213_owncloud.log.0.gz

Let me know whether you need more info for debugging purposes!

[GieltjE]

Could it be this crashes on Windows machines? When we try to sync large numbers of new files (2k+) it just crashes after a while.

[benjamin-leiding]

Yes, I was also able to reproduce the same results on a virtual machine with Win 7 and the 2.5.1 client for Windows. Unfortunately, I cannot provide logs at the moment. But the Linux logs should already help the dev team once they take care of this issue.

@GieltjE Could you maybe also upload logfiles to make it easier for the dev team to fix our issue?

[GieltjE]

The client log just logs sync complete, a friend of mine will hopefully post the windows logs soon.

We also noticed that it happens on lots of small files, when the files get bigger it doesn't crash every time.

@bleidingGOE, the windows logbook should provide the most information

[dennmuel]

Confirming the "too many files" error on Linux Mint. However the client does not restart but instead either dies without notification or causes the operating system to freeze. As a workaround one can disable sync for the causing folders, so that at least all others get synced.

[benjamin-leiding]

It has been 2 month since @gcala opened the issue - the dev team did not even respond despite us providing several logs and precise descriptions. Is nextcloud still actively developed?

[GieltjE]

According to github there are enough commits in the server code, but the desktop client just seems to be getting updated translations.... We can't even force the clients to update to new versions resulting in major parts of the userbase running outdated crap....

[camilasan]

Hi All,

Note that Github isn't a support portal. We provide support to customers only. Github is our developer platform. If you're not a customer you can ask the community for help on help.nextcloud.com.

We of course try to deal with bugs as quick as we can, but it can take time as we have lots of other things on our plate. You are welcome to contribute to this free and open source product, to submit patches that fix bugs for example. We try to respond quickly to pull request from contributors.

Have a great day! Camila

[lbdroid]

I ran into this issue today after a distro upgrade. The version I was running prior to the upgrade was https://github.com/nextcloud/desktop/commit/57bc7918d7b0650c116f3512787f7677d4e5ab17 . I am running that version again, and it is not affected by this REGRESSION.

If anybody cares enough about this problem to work on it, I'd suggest bisecting with that commit as a starting point.

@camilasan : your response comes off as hostile and rude. Not a particularly good way to encourage people to help in this project. I mean after all, how dare people want to ASSIST the project by reporting and discussing bugs in lieu of supporting it financially.

[jmjos]

Hello!

I also encountered the problem. It's not a bug within nextcloud, rather a system feature limiting the number of open files. E.g. see https://easyengine.io/tutorials/linux/increase-open-files-limit/ for a fix.

The issue can be closed.

[GieltjE]

It's not a Linux only problem, we only use Windows clients.

[benjamin-leiding]

I already tried this on my ubuntu 18.04 LTS - did not work. Moreover, the problem also occurs on Windows.

On 27 June 2019 10:00:14 CEST, JM Joseph notifications@github.com wrote:

Hello!

I also encountered the problem. It's not a bug within nextcloud, rather a system feature limiting the number of open files. E.g. see https://easyengine.io/tutorials/linux/increase-open-files-limit/ for a fix.

The issue can be closed.

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/nextcloud/desktop/issues/1035#issuecomment-506237981

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

[wioxjk]

Experiencing the same problem with 2.6.1 on the following distros:

Ubuntu 18.04, 19.10 Ubuntu Budgie 18.04, 19.10 Trisquel 8, 9 Debian 9, 10 Parabola Windows 8, 8.1, 10 Linux Mint 18, 19 OpenSuSE Tumbleweed

sadly, changing ulimit does not work.

The problem does not occur on 2.3.1

Photo and videofolder is 900 GB

[joenepraat]

+1 on Ubuntu 19.10.

I'm trying to sync a 25GB photo folder. 'Too many open files', 100% CPU and application freeze. I tried it with or without hidden files.

Seems like a huge issue and I'm perplexed that this is not taken seriously and still not fixed.

[wioxjk]

Seems like a huge issue and I'm perplexed that this is not taken seriously and still not fixed.

I agree, I've asked in the official Nextcloud channels but I have not even gotten a reply there.

[wioxjk]

Still no attention from the developers? I am strongly considering stop using Nextcloud both for private use and corporate, this issue is starting to become a plague for both usecases.

[claell]

I added some appropriate labels. Hopefully some developer can take a look at this, soon.

Just to get the impact correctly: This is completely preventing the concerning files to be ever synced and does not slowly disappear after more and more files got synced, correct?

[benjamin-leiding]

Thanks for adding the labels! Regarding your question: No, it does not slowly disappear - I left it running for several days without any progress.

[claell]

Thanks for clarifying @bleidingGOE. That make the issue rather severe in cases where it appears.

[SolarDesalination]

It's a bit embarrassing that Nextcloud is more focused on developing sub-standard apps that there already exist viable open source solutions for instead of making sure that foundational pieces such as folder synchronization is working properly. This bug has been open for over a year. I won't be recommending this software in the future to corporate clients.

[lbdroid]

@mpass3 : in all fairness (and I have no ties to nextcloud from a development perspective, just as a user), desktop synchronization really isn't a primary feature of nextcloud, and in many/most cases, the concept of synchronization is too confusing to actually use safely/reliably. I.e., change a file in both local and remote filesystem... Which version wins? The fact that there is no INTUITIVE answer to this means that the concept is flawed. In fact, this very problem is the reason why tools like git exist, and you would never hand that tool to a corporate paper pusher.

For corporate use, what you really want is to mount the nextcloud data into the local filesystem. You can use WebDAV for that. Let the end user decide on what files to overwrite.

[benjamin-leiding]

@lbdroid : I absolutely disagree. Even for large corporations, a client synchronization feature is essential. The issue of "which file should I keep" has been solved by Dropbox, GDrive, Syncthing, and so many other projects.

Moreover, based on your suggestion I tried to use my Nextcloud on Ubuntu 18.04 LTS with different WebDAV approaches -> They are horribly slow. No way to use this productively. Once you start googling the "slow WebDAV" issue they recommend using the client, which leads us back to this issue here.

[claell]

Can we please keep the discussion here on topic? Feel free to continue in the forum, you may post a link to a that here, so the others know where to continue. The topic you are talking about is definately worth disputing. But this is the issue tracker of nextcloud and this is a specific issue, so your comments are rather unrelated.

[benjamin-leiding]

Again, I disagree. The relevance of an issue is determined by how many users it affects and whether it can easily be solved by using a different approach. Not clarifying the gravity of the issue is why it has not been addressed in more than a year.

[claell]

Users affected may either use the emojis on the top to vote or if relevant add additional information in the comments. Also workarounds can of course be mentioned.

However, the last comments were discussing the relevance of having a working client. This is a meta discussion and could be applied to just any bug that is open here. So it is not specific for this issue.

Not clarifying the gravity of the issue is why it has not been addressed in more than a year.

I guess lacking development power for the client is the main reason. If you have the skills and would like to contribute that would be very much appreciated.

[nerdling]

Workaround for me (both changes required):

/etc/systemd/user.conf
DefaultLimitNOFILE=65536

/etc/security/limits.conf
*       soft    nofile  65536
*       hard    nofile  65536

And reboot.

Seems like the sync client isn't closing files as it's done with them. Normally fine, but after 1024 items it's this error. Is some code closing the FDs outside of a loop rather than inside?

[GieltjE]

/etc/systemd/user.conf DefaultLimitNOFILE=65536 /etc/security/limits.conf

• soft nofile 65536
• hard nofile 65536

Awesome for *nix variants but Windows doesn't have these settings.

[benjamin-leiding]

Already tried that on Ubuntu 18.04 LTS - did not work for me.

On 23 April 2020 16:40:07 CEST, Michiel Hazelhof notifications@github.com wrote:

/etc/systemd/user.conf DefaultLimitNOFILE=65536 /etc/security/limits.conf

• soft nofile 65536
• hard nofile 65536

Awesome for *nix variants but Windows doesn't have these settings.

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/nextcloud/desktop/issues/1035#issuecomment-618433696

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

[wioxjk]

Workaround for me (both changes required):

/etc/systemd/user.conf
DefaultLimitNOFILE=65536

/etc/security/limits.conf
*       soft    nofile  65536
*       hard    nofile  65536

And reboot.

Seems like the sync client isn't closing files as it's done with them. Normally fine, but after 1024 items it's this error. Is some code closing the FDs outside of a loop rather than inside?

Does not work as I wrote here: https://github.com/nextcloud/desktop/issues/1035#issuecomment-546673468

And I agree with the previous poster - bleidingGOE - that the sync feature is a essential part of Nextcloud and has always been. Defending the developers in this case does not make any sense at all, it is a bug and should be treated as such.

[warnerbryce]

I got this bugs too, I get back to v.2.3.3 and no more issue. @camilasan don’t want to open her eyes and see that since v.2.5 things went wrong with the sync engine on Linux / Windows and OS X. The v.2.6.4 get better results in many cases i use it on my linux but not on my windows. Impossible to get logs because the software hangs and is irresponsible.

One year waiting bug

[claell]

Impossible to get logs because the software hangs and is irresponsible.

You should be able to write a log directly to a file, that will probably circumvent this problem.

[claell]

This might be the same as #1769. It should have been fixed in #1899. Please read through #1769 whether it is the same issue and there is also a link to a daily AppImage build where that issue seems to be fixed. Maybe you also want to test that out, whether it fixes your problems as well.

[nerdling]

Removing the increased fd limits and then using the AppImage to fetch a fresh Nextcloud mount succeeds. The problem appears to be fixed in the AppImage.

[claell]

Nice, thanks for the feedback! I will leave this open until it lands in a stable release.

[danjpgriffin]

PR #1899 to fix this issue only made it into the 2.6-stable branch. I downloaded v3.0.1 yesterday and unfortunately the issue has reappeared. I'm just checking if reapplying the patch to 3.0.1 works, with then reopen PR against master

[github-actions[bot]]

This bug report did not receive an update in the last 4 weeks. Please take a look again and update the issue with new details, otherwise the issue will be automatically closed in 2 weeks. Thank you!

[wioxjk]

Commenting because this is not fixed.

[github-actions[bot]]

This bug report did not receive an update in the last 4 weeks. Please take a look again and update the issue with new details, otherwise the issue will be automatically closed in 2 weeks. Thank you!

[mgallien]

Please provide the desktop client logs:

• Since 3.1: Under the "General" settings, you can click on "Create Debug Archive ..." to pick the location of where the desktop client will export the logs and the database to a zip file.
• On previous releases: Via the command line: nextcloud --logdebug --logwindow or nextcloud --logdebug --logfile log.txt (See also https://docs.nextcloud.com/desktop/3.1/troubleshooting.html#log-files) Thanks!

[ffuentese]

Please at least explain how to work it around. I think this is an issue for everyone with enough files no matter the OS.

This is what I get in OpenBSD when I start the GUI client:

QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-ffuentes'
2021-06-20 17:47:44:789 [ critical  ]:  QNetworkInterface: could not create IPv6 socket (Too many open files)
2021-06-20 17:47:44:789 [ critical  ]:  QNetworkInterface: could not create IPv6 socket (Too many open files)
2021-06-20 17:47:44:789 [ critical  ]:  QNetworkInterface: could not create IPv6 socket (Too many open files)
2021-06-20 17:47:44:790 [ critical  ]:  QNetworkInterface: could not create IPv6 socket (Too many open files)

(nextcloud:95531): GLib-ERROR **: 17:47:46.792: Creating pipes for GWakeup: Too many open files
[1]    95531 trace trap (core dumped)  nextcloud

The application doesn't even stay open because it's gonna crash almost immediately so no one will be able to provide a log from the GUI.

This is the whole log file I get https://ttm.sh/F7z.txt

[github-actions[bot]]

This bug report did not receive an update in the last 4 weeks. Please take a look again and update the issue with new details, otherwise the issue will be automatically closed in 2 weeks. Thank you!

[ffuentese]

I wrote this is an update on how to deal with the issue at least in OpenBSD but it might be useful for other OSs

https://ffuentes.sdf.org/unix/2021/06/23/nextcloud-client-in-openbsd.html

[github-actions[bot]]

This bug report did not receive an update in the last 4 weeks. Please take a look again and update the issue with new details, otherwise the issue will be automatically closed in 2 weeks. Thank you!

[mgallien]

I wrote this is an update on how to deal with the issue at least in OpenBSD but it might be useful for other OSs

https://ffuentes.sdf.org/unix/2021/06/23/nextcloud-client-in-openbsd.html

Thanks a lot for this. We should really try to make good use of your blog post. Do you mind if I try to share it ?

[azcn2503]

Workaround for me (both changes required):

/etc/systemd/user.conf
DefaultLimitNOFILE=65536

/etc/security/limits.conf
*       soft    nofile  65536
*       hard    nofile  65536

And reboot.

Seems like the sync client isn't closing files as it's done with them. Normally fine, but after 1024 items it's this error. Is some code closing the FDs outside of a loop rather than inside?

This workaround worked for me today on desktop client version 3.4.0 on Manjaro (manjaro-5.10.84-1-MANJARO).

[richardpeng]

Increasing the process max open files on Mac doesn't fully work around this issue. While the sync no longer fails early due to "too many open files", the sync slows to a crawl exponentially. By the time the client syncs files 1000+, it's taking several minutes for each 100-file chunk. The client also becomes sluggish and unresponsive. Please fix this bug and release file descriptors as files are done syncing.

Here are more details about my attempted workaround:

The default MacOS max process file open limit is 256. I increased this to 4096 using the configuration below:

/Library/LaunchDaemons/limit.maxfiles.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
  <plist version="1.0">
    <dict>
      <key>Label</key>
      <string>limit.maxfiles</string>
      <key>ProgramArguments</key>
      <array>
        <string>launchctl</string>
        <string>limit</string>
        <string>maxfiles</string>
        <string>4096</string>
        <string>2147483647</string>
      </array>
      <key>RunAtLoad</key>
      <true/>
      <key>ServiceIPC</key>
      <false/>
    </dict>
  </plist>

# Secure file permissions
$ sudo chown root:wheel /Library/LaunchDaemons/limit.maxfiles.plist
# Load new settings
$ sudo launchctl load -w /Library/LaunchDaemons/limit.maxfiles.plist
# Verify new settings
$ launchctl limit maxfiles
    maxfiles    4096           2147483647

You may need to restart the system for all processes to launch using the new settings.

You can monitor the open files using the Activity Monitor and see that files are never removed from the list during a sync. You can Pause/Restart sync and the open files will drop back down to the number open when you first launch the client, which is around 62.

This is a screenshot of the client after syncing 1400 files:

References: https://neo4j.com/developer/kb/setting-max-open-file-limits-on-osx/ https://apple.stackexchange.com/questions/366187/why-does-setting-the-hard-limit-for-maxfiles-to-unlimited-using-launchctl-lim

[mazen-mardini]

I can reproduce this error on Linux and client version 3.4.1. The client is syncing against Nextcloud 23.

[l-ucky]

Workaround for me (both changes required):

/etc/systemd/user.conf
DefaultLimitNOFILE=65536

/etc/security/limits.conf
*       soft    nofile  65536
*       hard    nofile  65536

And reboot. Seems like the sync client isn't closing files as it's done with them. Normally fine, but after 1024 items it's this error. Is some code closing the FDs outside of a loop rather than inside?

This workaround worked for me today on desktop client version 3.4.0 on Manjaro (manjaro-5.10.84-1-MANJARO).

I had issues moving 2.5Gb of Zotero files (research papers, pdfs htmls) becaue of the 'too many open files' error. I did the workaround above on my desktop, not on my server. All I have changed on my server are the user.ini and PHP.ini for upload size. I learned that from DBTech's videos.

[Jip-Hop]

Got excited because of the Nextcloud Sync 2.0 brings 10x faster syncing announcement. Syncing many small files has been the reason I can't use Nextcloud since I first evaluated it in 2016. So I decided to give it another try and benchmark the speed by syncing the extracted contents of the Firefox source code.

Quite disappointed to be running into the "too many open files" issue. Seems like this issue should have been fixed before encouraging users to sync many small files: "Users will benefit from instant availability of their files, even if they need to have large numbers of small files made available locally."

For me it happens on macOS Mojave 10.14.6. Initially syncing starts over after the "too many open files" error. But eventually syncing halts at 0B of 1,4 GB remaining with 100% CPU usage.

Nextcloud Desktop Client
Version 3.4.1 (macOS). For more information please click here.
Using virtual files plugin: suffix

osx-18.7.0

Update:

Tried syncing the same folder again on macOs 10.15.7 (osx-19.6.0) after applying the workaround suggested by @richardpeng.

launchctl limit maxfiles
    maxfiles    400000         400000  

Even with the raised limits, syncing gets stuck at Syncing file 33056 of 305067.

Why does the Nextcloud Desktop client keep so many files open?

[edraft]

no fix yet???