search

nextcloud / desktop

šŸ’» Desktop sync client for Nextcloud
https://nextcloud.com/install/#install-clients
GNU General Public License v2.0
3.03k stars 794 forks source link

Nextcloud client 3.1 syncs "ignored files" #2728

Closed libeanim closed 2 years ago

libeanim commented 3 years ago

How to use GitHub

Expected behaviour

Nextcloud client should ignore files and folders that have been added to the "Ignored Files" list through the UI.

Actual behaviour

Nextcloud client syncs folders that are in the "ignored files" list since the update to 3.1.
I've experienced this bug on "folder sync-connection"-level (see "Steps to reproduce" if unclear), it is also possible that it happens when using the global ignore file.

Steps to reproduce

  1. Add a new 'Folder Sync Connection'
  2. Click on the three dots on the settings window next to the newly added "folder sync connection" and select "Edit Ignored Files" from the dropdown menu.
  3. Add a folder you want nextcloud to ignore
  4. Start sync and create a folder that is in the ignored list
  5. Watch nextcloud client sync this folder

Client configuration

Client version: 3.1.0-20201214.180315.7f4ffd0f7-1.0~focal1

Using Qt 5.12.8, built against Qt 5.12.8

Using Qt platform plugin 'xcb'

Using 'OpenSSL 1.1.1f 31 Mar 2020'

Running on Ubuntu 20.04.1 LTS, x86_64

Installation path of client: /usr/bin/nextcloud

Logs

Haven't seen any errors in the logs, it seems like a general client problem. If it is only my installation I can add this information.

er-vin commented 3 years ago

@camilasan could be related to your work in that area? rings any bell?

nursoda commented 3 years ago

Not only applies to per-instance ignore lists but also for global ignore list (in Settingsā†’General).

benjamindahl commented 3 years ago

this slows down my client extremely. How can I go back to 3.0.3 under ubuntu? There's no 3.0.3. package in the repository anymore.

expert-geeks commented 3 years ago

You might find this link helpful, https://download.nextcloud.com/desktop/releases/

benjamindahl commented 3 years ago

You might find this link helpful, https://download.nextcloud.com/desktop/releases/

Thanks a lot. It works for the moment.

libeanim commented 3 years ago

Hmm so the global ignore file seems to work for me and it also appears in the log:

2020-12-21 18:02:28:264 [ info nextcloud.sync.configfile ]:
Adding user defined ignore list to csync: "/home/andreas/.config/Nextcloud/sync-exclude.lst"

I'm just guessing, but @camilasan is it possible that the new path building in the directory traversal function has a bug?

roundoutandabout commented 3 years ago

@libeanim I also have this issue. Nextcloud-3.1.0-x86_64.AppImage syncs everything ignoring the ignore list. Nextcloud-3.0.3-x86_64.AppImage works fine. I'm using KDE Neon 5.20

albgus commented 3 years ago

I noticed this with 3.1.0 and 3.1.1 on Windows as well.

chuck-r commented 3 years ago

This isn't just a Windows problem. I noticed the same thing when I upgraded Nextcloud using the Linux AppImage yesterday. Unfortunately, I'm not sure what version I came from, but upgrading to 3.1.1 caused it to try to sync all of my excluded files. I downgraded to 3.0.3 as mentioned here and I still have the same problem.

EDIT: The following workaround/hackery only works for me on 3.0.3, but not 3.1.1

However I noticed that if I move my excluded files out of the sync path and let the folders sync sans the ignored files it syncs fine (of course, since it doesn't have to consider them). Then, if I move the ignored files back into the folder once the initial sync finishes successfully the files are properly ignored -- at least in my very small test case.

In one of my problem folders I had two subfolders, one with 50 or so fairly large source archives within a couple dozen folders. When I moved that folder into place it had no problems and did not synchronize them as the whole folder was on the ignore list. However, I had an issue with moving my second folder into place which had 15,357 folders and 330,359 files -- basically it at up all 119GB of my RAM before I finally had to kill it. But, that's a separate issue I think.

So, there is some disparity between the ignore list when it comes to a full sync (or a force resync, presumably) and an update sync (when it detects a file change).

qu1ck commented 3 years ago

I have this issue with 3.1.1 on windows. Folder specific ignore list doesn't work, global one does.

chuck-r commented 3 years ago

Thanks @qu1ck! I was getting really frustrated by not being able to sync a couple of folders due to this bug. Adding the exception globally did fix the problem, albeit not a perfect solution. So, it seems that the client it's finding/reading/parsing the folder-specific sync excludes. At least that gets us closer to the right direction. I've been investigating this one myself as well, but I'm still trying to get the "lay of the land" as it were with the Nextcloud Client source.

StarSmasher44 commented 3 years ago

If I may provide some small insight into my actions dealing with this, I had to exclude log folders & contents, adding just Logs/ did not work.

However, adding both 'Logs/' and 'Logs/*' made the entire folder plus contents be ignored as it should. Not sure if it is useful information, but that worked well enough for me.

camilasan commented 3 years ago

I will investigate it.

FiC5020 commented 3 years ago

Have the same problem / issue too. Upgraded from 3.0.3 auf 3.1.1. Now the the client would sync 300 GB off Data which are excluded. Downgrade to 3.0.3 the ignore list works.

Problem solved with an older client.

StarSmasher44 commented 3 years ago

Have the same problem / issue too. Upgraded from 3.0.3 auf 3.1.1. Now the the client would sync 300 GB off Data which are excluded. Downgrade to 3.0.3 the ignore list works.

Problem solved with an older client.

Curious, have you tried the method I spoke of a bit above here? it worked out for me in that way, it may also work for you without the need to go back versions :)

RM-0454 commented 3 years ago
  1. Add a folder you want nextcloud to ignore
  2. Start sync and create a folder that is in the ignored list
  3. Watch nextcloud client sync this folder

@libeanim I have been having the same issue, but after the following fix worked, I think I know the bug's source.

However I noticed that if I move my excluded files out of the sync path and let the folders sync sans the ignored files it syncs fine (of course, since it doesn't have to consider them). Then, if I move the ignored files back into the folder once the initial sync finishes successfully the files are properly ignored -- at least in my very small test case.

After trying both, my sync excludes are working and I think I know why.

Let's walk through our process again. Pretend we want to exclude the directory foo from a synced folder.

However, if instead of creating foo, we copy foo into the synced folder, nextcloud ignores it (as it should). What's the difference?

Theory When a new folder is created it is automatically assigned the name "New folder" (in windows). "New folder" isn't an excluded file, so it is immediately synced even as you are renaming it. Then when you do rename it, nexctloud records the change in the directory's name without double checking it against the exclude list. That is my suspicion anyway.

OS: Windows 7 | NC Version 3.1.1

timpur commented 3 years ago

I found the opposite of https://github.com/nextcloud/desktop/issues/2728#issuecomment-751670821

Global Ignore list didn't work but folder ignore list did.

Standard format node_modules/ worked in folder ignore list not root list.

Its like it wasn't reading the list at all, or maybe the working dir is incorrect from root ?

(windows 3.1.2)

dasaweb commented 3 years ago

this slows down my client extremely. How can I go back to 3.0.3 under ubuntu? There's no 3.0.3. package in the repository anymore.

Are you sure that this is a related problem? I see the same 2 problems when I update from 3.0.3 to 3.1.x:

  1. .sync-exclude.lst are ignored.
  2. The client get's extremly slow.

I'm not sure if these two are related. But I hope, that this is the case, so solving one problem would also solve the other ;-)

benjamindahl commented 3 years ago

this slows down my client extremely. How can I go back to 3.0.3 under ubuntu? There's no 3.0.3. package in the repository anymore.

Are you sure that this is a related problem? I see the same 2 problems when I update from 3.0.3 to 3.1.x:

  1. .sync-exclude.lst are ignored.
  2. The client get's extremly slow.

I'm not sure if these two are related. But I hope, that this is the case, so solving one problem would also solve the other ;-)

I don't know exactly what the problem is but I'll stick now with 3.0.3 for a while.

dasaweb commented 3 years ago

Staying with version 3.0.3 is the only option for me at the moment. However, with version 3.0.3 I have this additional problem: https://github.com/nextcloud/desktop/issues/324 So no working client for me right now... :-/

talesam commented 3 years ago

This should be simpler, in the GUI to have an option to deselect what you don't want to synchronize, is that so complex to do?

SMESH109 commented 3 years ago

Has anyone tested the new 3.2.0 client yet? The changelog doesn't seem to address the problems from this thread, but you never know.

dasaweb commented 3 years ago

Has anyone tested the new 3.2.0 client yet? The changelog doesn't seem to address the problems from this thread, but you never know.

I'm not brave enough to test it. My last test with a version > 3.0.3 resulted in hours of manual "synchronization" ;-/

SMESH109 commented 3 years ago

Has anyone tested the new 3.2.0 client yet? The changelog doesn't seem to address the problems from this thread, but you never know.

I'm not brave enough to test it. My last test with a version > 3.0.3 resulted in hours of manual "synchronization" ;-/

Exactly the same herešŸ˜„ I don't have a test server or something and don't want to ruin my currently working 3.0.3, although I'm getting sick of it digging through my HDD endless times (SSDs are no problem at all! but it's also just Ā¼ the amount of data).

filviu commented 3 years ago

I only have a handful of ignored files so I was using the workaround (Adding to "Ignored Files" under settings not under the account)

I removed them from there and redid my list under account - ... - Edit Ignored Files and the initial impression is that it's back working.

My version is 3.2.0 stable.

SMESH109 commented 3 years ago

I only have a handful of ignored files so I was using the workaround (Adding to "Ignored Files" under settings not under the account)

I removed them from there and redid my list under account - ... - Edit Ignored Files and the initial impression is that it's back working.

My version is 3.2.0 stable.

I uninstalled 3.0.3, installed 3.2.0 and did the sameā€”removed the folders to exclude from the global ignore list and added them to the folder sync ignored filesā€”but they get synced. Added them back to the global ignore list and they get ignored as they should

dasaweb commented 3 years ago

My problem is that I have to use different exclude patterns for different sync pairs, so I can't define them in the global ignore list :-/

filviu commented 3 years ago

I take it back, it's still not working.

A simple exclusion like: apps/keypirinha/portable/Profile/User/Keypirinha.history (I have some portable apps under apps in nextcloud)

is ignored (it's synced constantly and get conflicts when changed).

I'm puzzled why this doesn't get more exposure ? Is it some exotic server side setting that causes this ? I feel a lot more people should be reporting if exlude really is broken...

Server: 21.0.1 (docker) Clients: Windows 3.2.1

dasaweb commented 3 years ago

Unfortunately, this case (#1875) was closed this morning due to inactivity. In my opinion, however, the two problems could be related. No idea how we can bring the problem to the attention of someone here who has a clue about it...

bluebirch commented 3 years ago

I accidentally started 3.1 instead of 3.0, all my .git repositories was synked (again!) and i have to clean up the mess (again!). Would really appreciate having this bug fixed. ;-P

dasaweb commented 3 years ago

Version 3.3.0-rc1 has come out, unfortunately there seems to be no progress regarding this issue here :-/ . So I'm stuck on version 3.0.3 for about 1 year now (and that is really annoying because 3.0.3 and 3.3.0 look so similar :-) ). But maybe there is hope after the bug was labeled as "approved" by the team here 8 weeks ago.

LecrisUT commented 3 years ago

Maybe we can get some update from @camilasan for #2723. Would that PR likely fix this issue and what's the status of it?

FlexW commented 3 years ago

Just for the record (not sure if anyone mentioned it already): The bug only occurs on root folders. It does not happen in subfolders.

alexjmccoy commented 3 years ago

Just for the record (not sure if anyone mentioned it already): The bug only occurs on root folders. It does not happen in subfolders.

I'm not sure about this, unless I'm doing my patters incorrectly. My root folder is Nextcloud and I want to ignore Nextcloud/org/notes.org and the directory Nextcloud/org/journal. Accordingly, my ignore patterns are:

notes.org
org/notes.org
org/journal

and yet everything is synced. This is client 3.2.0 on macOS.

patagonaa commented 3 years ago

@alzxjm I think what @FlexW meant, is that the ignore file in the root directory does not work but ignore files in subdirectories work (at least during my tests on Windows, that was the only way I could get ignore files to work at all).

So in your case instead of an ignore file Nextcloud/.sync-exclude.lst, you should be able to use an ignore file Nextcloud/org/.sync-exclude.lst with the patterns

notes.org
journal

and those directories should be correctly ignored.

dasaweb commented 3 years ago

Maybe that's absolute nonsense, but I just try to help (and to keep this problem alive):

I checked all the changes between the still working version 3.0.3 and the no longer working version 3.1.0 (v3.1.0-rc1 v3.1.0-rc2 v3.1.1), and the only change that has to do with "ignore files" or "exclude files" is this pull request.

@er-vin, you were involved in that, can you assess if that change could be related to this problem here?

arrowd commented 3 years ago

Have this problem on FreeBSD with Nextcloud client 3.2.4

solracsf commented 3 years ago

@camilasan as you're the PR author, could this be related to https://github.com/nextcloud/desktop/pull/2540?

jobec commented 3 years ago

It would be great to get this fixed. As mentioned in #1875 this is very very annoying. Been scratching my head again for a few hours trying to figure out why the hell some folders got ignored and others didn't. šŸ¤Æ

arrowd commented 3 years ago

Interestingly, this functionality does work on Windows.

jobec commented 3 years ago

Yes and no... I'm also on Windows. And the ignore function works. But only the global ignore settings. Those per sync setting do not. And I'm in a situation where I need to ignore some files in sync setting A, but include those in sync setting B.

viendocraz commented 3 years ago

Also on Windows and experiencing this issue with v3.3.5. Folder Sync excludes are ignored at the root level but global excludes are working, which is fine as a temporary workaround but I'm hoping for a fix to make it work better!

xdanaux commented 3 years ago

At least on macOS, it looks like the sync-exclude.lst files are ignored at the root level but are correctly observed at any level below.

LecrisUT commented 3 years ago

By now there is a strong consensus on the current behaviour on both MacOS, Windows and Linux, and it appears to be a bug in the code not any specific implementation. So it is no longer necessary to report this behaviour unless there is any changes.

Observed behaviour

Temporary workaround

Potential solutions

2723 Please test it and report if it fixes the issue. I will try it out myself as soon as I have the time.

camilasan commented 3 years ago 
* The root `sync-exlcude.lst` is not recognized. It is synced but it is not used.

it should be fixed by #2723 and #3882.

jokay commented 3 years ago

@camilasan "should" or "is" fixed? :)

SMESH109 commented 3 years ago

I wondered the same.

3882 is already merged into main and set for Release 3.4

2723 has been rejected from automatic merge and has to be reviewed manually before merging.

camilasan commented 3 years ago

I was too eager to close the issue :)

LecrisUT commented 3 years ago

I would like to test both of these PRs, but there are no macos artifacts generated, and I'm having trouble compiling from source due to homebrew qt. If anyone can confirm that the root .sync-exclude.lst is picked up locally, and whether or not it's shared among other clients (preferably it should unless .sync-exclude.lst pattern is included in itself). Probably any OS would suffice.

dasaweb commented 2 years ago

I wondered the same. #3882 is already merged into main and set for Release 3.4 #2723 has been rejected from automatic merge and has to be reviewed manually before merging.

Both pull requests are merged into the master branch now. Does this mean that we can now hope?