[Bug]: desktop client DoS-ing nextcloud server

mrAceT commented 1 year ago

⚠️ Before submitting, please verify the following: ⚠️

[X] This is a bug, not a question or a configuration issue.
[X] This issue is not already reported on Github (I've searched it).
[X] Nextcloud Server and Desktop Client are up to date. See Server Maintenance and Release Schedule and Desktop Releases for supported versions.
[X] I agree to follow Nextcloud's Code of Conduct

Bug description

the anti-abuse-software of my server had blocked me!

Ik checked for the cause and I discovered that the syncing was the cause.

The point that triggered the "abuse detection" was this: "PROPFIND /remote.php/dav/files/[user]/[folder] HTTP/1.1" 207 547 "-" "Mozilla/5.0 (Windows) mirall/3.10.0stable-Win64 (build 20230915) (Nextcloud, windows-10.0.19045 ClientArchitecture: x86_64 OsArchitecture: x86_64)"

Every line an other folder, at a speed of about 50 request per second!

That is a bit excessive is my opinion! This acts a bit like a DoS attack.. and I could see a (serious) drop in performance of the server..

Steps to reproduce

I could not reproduce the huge amount of requests per second, but checking the access-log when a (manual) sync command is given dit result in about 15 to 30 requests per second..

Expected behavior

I have programmed an exception for this behavior of Nextcloud, to prevent me (and customers) to get firewalled, but I expected a bit of "throttling"..

I could see a serious drop in performance for just me syncing! And the server I'm using isn't a "light weight one"! I will admit that the account synced has hundreds of folders, but I'dd rather have the sync take a bit longer by design so that the server wouldn't run the rick of dying of (or answering slower and slower..) due to huge amounts of sync-requests..

In other words, I think the Nextcloud server would operate much more safely and stable when these "propfind requests" would be throttled to a maximum of something like 15 requests per second..

Which files are affected by this bug

PROPFIND /remote.php/dav/files/

Operating system

Windows

Which version of the operating system you are running.

Windows 10

Package

Appimage

Nextcloud Server version

25.0.10

Nextcloud Desktop Client version

3.10.0

Is this bug present after an update or on a fresh install?

Updated from a minor version (ex. 3.4.2 to 3.4.4)

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

Are you using an external user-backend?

[X] Default internal user-backend
[ ] LDAP/ Active Directory
[ ] SSO - SAML
[ ] Other

Nextcloud Server logs

No response

Additional info

No response

mgallien commented 1 year ago

do you really suggest us slowing down the sync of a single client over concern of server being overloaded ? I cannot really understand your rationale there Can you explain more ?

mrAceT commented 1 year ago

I'll try as best as I can to explain my thoughts:

I think the whole concept would include the creation of a stable and working setup for data management. As for this part, when the desktop client is "over enthusiastic" in it's requests for data synchronization it could (and can) ask so much of the server that it realizes the exact opposite: when one over-requests, one creates a form of "Denial of Service" attack.. that the server willing to answer all those requests gets so busy in (preparing) to answer that it actually answers slower..

So, back to my original premise: one wants a stable and working setup: So that is why I created this "bug report". Throttling the maximum number of requests per second might at first glance seem counter productive, but this actually isn't that strange.. asking for something is more often then not easier than coming up with the answer.. And since the server and the client should be a team, I thought it wouldn't be that odd to ask (somehow) for the "asking party" (the desktop client) to be "nice" to the "answering party" (the server).

That way the stability of the server can increase and therefor the stability of the whole system..

My first thought would be to put a cap on the maximum number of requests. Mostly because that's the easiest way I see to address this, but you might have a much better idea!?

mgallien commented 1 year ago

are you really sure a single client can send enough parallel requests that it would slow down a server ?

mrAceT commented 1 year ago

This happend at night, so this wasn't 'a disaster'.. (and I was asleep ;)

Here the log of my monitorring system:

Again, this 'peak' isn't a direct problem, but was caused by just one account.. and that was a bit scary to me.. if just one sync action can cause such a peak in load.. what could happen when my project (that intensively uses NextCloud) really kicks off..

isdnfan commented 11 months ago

I don't see from your description if you enabled notify_push aka high performance back-end for files: it's highly recommended to use notify_push as if highly reducing server load resulting from client polling.

for details consult:

joshtrichards commented 1 month ago

the anti-abuse-software of my server had blocked me! [...] The point that triggered the "abuse detection" was this:

What "anti-abuse-software"?

Is this just a theoretical concern based on some arbitrary values placed in this "anti-abuse-software" that set off some alerts or were you seeing a real resource problem on your server? If a real resource problem, what types of resources were being bumped up against?

Just trying to confirm this is actually a real problem caused by an overloaded server, or more an optimization problem where there were plenty of resources available and maybe some re-config is necessary (see below).

I could see a serious drop in performance for just me syncing! And the server I'm using isn't a "light weight one"! I will admit that the account synced has hundreds of folders, but I'dd rather have the sync take a bit longer by design so that the server wouldn't run the rick of dying of (or answering slower and slower..) due to huge amounts of sync-requests..

Rather then speculate, do you know where the drop in performance was arising from? i.e. were you hitting up against web server connection limits, fpm connection pool limits, etc. or something? Or even more generally: memory? cpu? etc.

Unfortunately it sounds like you're asking for a simple solution to a problem that isn't yet well defined. There are a million variables here, and they may or may not even be real problems. Some maybe the client could do something about, but some may be addressable - more appropriately even - in other ways.

There's simply not enough information in this report to judge.

I have programmed an exception for this behavior of Nextcloud, to prevent me (and customers) to get firewalled, but I expected a bit of "throttling"..

And has that allowed you to gather more information to determine what, precisely, are the resource limits that are being bumped up against in your environment?

github-actions[bot] commented 1 week ago

This issue has been marked as "needs info" 4 weeks ago. Please take a look again and try to provide the information requested, otherwise the issue will be automatically closed in 2 weeks. Thank you!

nextcloud / desktop