[Bug]: Nextcloud client generates >100 GB worth of log files every day (800MB per sync run).

retnag commented 1 year ago

⚠️ Before submitting, please verify the following: ⚠️

[X] This is a bug, not a question or a configuration issue.
[X] This issue is not already reported on Github (I've searched it).
[X] Nextcloud Server and Desktop Client are up to date. See Server Maintenance and Release Schedule and Desktop Releases for supported versions.
[X] I agree to follow Nextcloud's Code of Conduct

Bug description

First of all, thank you for developing this awesome and completely "free" home cloud solution! I use it on a daily basis.

Perhaps I'm alone with this, but my Nextcloud client generates >100 GB (800MB per sync run) worth of log files every day. I use an SSD; therefore this behaviour is seriously and unnecessarily (I never read these logs) reducing the lifespan of my hardware, as you surely know, SSD-s can be written to only a limited amount of times. I am on Windows and using Virtual files support, so every file in the cloud gets a go for sync.

I have worked around this problem temporarily, by removing write permissions for every user for the logs folder, but now I get a pop-up Error from the client every half an hour or so, therefore I'm not satisfied with this solution. I had added the permissions back while writing this review, it generated 2.25 GB worth of logs in just 20 minutes. That's 162 GB of unnecessary data a day, contributing 57 TB a year, seriously bad for your SSD! That alone consumes an average SSD with 100TBW in less than 2 years!

Possible solution suggestions/ideas:

Cheap: I suggest adding an option in the client settings to disable the saving of log files on disk. And/Or, alternatively, add an option to save the logs in a compressed file format. More time consuming, but better alternative: add an option for the verbosity of the logs, with the usual options for the loglevel: "debug" (the current behaviour), "only errors" - only logs unexpected errors,

The logs are VERY verbose, here is an excerpt:

2022-12-31 10:03:41:389 [ debug nextcloud.sync.database.sql C:\Users\sysadmin\AppData\Local\Temp\2\windows-8634\client-building\desktop\src\common\ownsql.cpp:295 ] [ OCC::SqlQuery::exec ]:    SQL exec "SELECT path, inode, modtime, type, md5, fileid, remotePerm, filesize,  ignoredChildrenRemote, contentchecksumtype.name || ':' || contentChecksum, e2eMangledName, isE2eEncrypted  FROM metadata  LEFT JOIN checksumtype as contentchecksumtype ON metadata.contentChecksumTypeId == contentchecksumtype.id WHERE parent_hash(path) = ?1 ORDER BY path||'/' ASC"
2022-12-31 10:03:41:389 [ info sync.discovery C:\Users\sysadmin\AppData\Local\Temp\2\windows-8634\client-building\desktop\src\libsync\discovery.cpp:362 ]:  Processing "Documents/Dokumentumok/Könyvtár/manuals/zwq_5100_user_manual.pdf" | valid: true/true/db | mtime: 1597998770/1597998770/0 | size: 2157020/2157020/0 | etag: "640d0ea5671d237e9363196c4c7f5557"//"" | checksum: ""//"" | perm: "WDNVR"//"" | fileid: "22247373ockyzphxy1jr"//"" | inode: 839869/839869/ | type: CSyncEnums::ItemTypeVirtualFile/CSyncEnums::ItemTypeVirtualFile/CSyncEnums::ItemTypeFile | e2ee: false/false | e2eeMangledName: ""/""
2022-12-31 10:03:41:389 [ info sync.discovery C:\Users\sysadmin\AppData\Local\Temp\2\windows-8634\client-building\desktop\src\libsync\discovery.cpp:1368 ]: Discovered "Documents/Dokumentumok/Könyvtár/manuals/zwq_5100_user_manual.pdf" CSyncEnums::CSYNC_INSTRUCTION_NONE OCC::SyncFileItem::None CSyncEnums::ItemTypeVirtualFile
2022-12-31 10:03:41:389 [ info sync.discovery C:\Users\sysadmin\AppData\Local\Temp\2\windows-8634\client-building\desktop\src\libsync\discovery.cpp:86 ]:   STARTING "Documents/Dokumentumok/Szerződések" OCC::ProcessDirectoryJob::ParentNotChanged "Documents/Dokumentumok/Szerződések" OCC::ProcessDirectoryJob::NormalQuery
2022-12-31 10:03:41:389 [ info sync.discovery C:\Users\sysadmin\AppData\Local\Temp\2\windows-8634\client-building\desktop\src\libsync\discovery.cpp:86 ]:   STARTING "Documents/Dokumentumok/UsersGuides" OCC::ProcessDirectoryJob::ParentNotChanged "Documents/Dokumentumok/UsersGuides" OCC::ProcessDirectoryJob::NormalQuery
2022-12-31 10:03:41:389 [ debug nextcloud.sync.database.sql C:\Users\sysadmin\AppData\Local\Temp\2\windows-8634\client-building\desktop\src\common/ownsql.h:145 ]   [ OCC::SqlQuery::bindValue ]:   SQL bind 1 -1074047111436947454
2022-12-31 10:03:41:389 [ debug nextcloud.sync.database.sql C:\Users\sysadmin\AppData\Local\Temp\2\windows-8634\client-building\desktop\src\common\ownsql.cpp:295 ] [ OCC::SqlQuery::exec ]:    SQL exec "SELECT path, inode, modtime, type, md5, fileid, remotePerm, filesize,  ignoredChildrenRemote, contentchecksumtype.name || ':' || contentChecksum, e2eMangledName, isE2eEncrypted  FROM metadata  LEFT JOIN checksumtype as contentchecksumtype ON metadata.contentChecksumTypeId == contentchecksumtype.id WHERE parent_hash(path) = ?1 ORDER BY path||'/' ASC"
2022-12-31 10:03:41:389 [ info sync.discovery C:\Users\sysadmin\AppData\Local\Temp\2\windows-8634\client-building\desktop\src\libsync\discovery.cpp:362 ]:  Processing "Documents/Dokumentumok/Other/inet számla/26105590_1629834850396710_813499199_n.jpg" | valid: true/true/db | mtime: 1514397316/1514397316/0 | size: 24795/24795/0 | etag: "615712d36cb8429729606950e7d7a49f"//"" | checksum: ""//"" | perm: "WDNVR"//"" | fileid: "20462908ockyzphxy1jr"//"" | inode: 839874/839874/ | type: CSyncEnums::ItemTypeVirtualFile/CSyncEnums::ItemTypeVirtualFile/CSyncEnums::ItemTypeFile | e2ee: false/false | e2eeMangledName: ""/""
2022-12-31 10:03:41:389 [ info sync.discovery C:\Users\sysadmin\AppData\Local\Temp\2\windows-8634\client-building\desktop\src\libsync\discovery.cpp:1368 ]: Discovered "Documents/Dokumentumok/Other/inet számla/26105590_1629834850396710_813499199_n.jpg" CSyncEnums::CSYNC_INSTRUCTION_NONE OCC::SyncFileItem::None CSyncEnums::ItemTypeVirtualFile
2022-12-31 10:03:41:389 [ info sync.discovery C:\Users\sysadmin\AppData\Local\Temp\2\windows-8634\client-building\desktop\src\libsync\discovery.cpp:362 ]:  Processing "Documents/Dokumentumok/Other/inet számla/26105684_1629834407063421_1422797738_n.jpg" | valid: true/true/db | mtime: 1514397324/1514397324/0 | size: 24719/24719/0 | etag: "097b00fd36fc9509755ac3bae1dbada2"//"" | checksum: ""//"" | perm: "WDNVR"//"" | fileid: "20462909ockyzphxy1jr"//"" | inode: 839875/839875/ | type: CSyncEnums::ItemTypeVirtualFile/CSyncEnums::ItemTypeVirtualFile/CSyncEnums::ItemTypeFile | e2ee: false/false | e2eeMangledName: ""/""
2022-12-31 10:03:41:389 [ info sync.discovery C:\Users\sysadmin\AppData\Local\Temp\2\windows-8634\client-building\desktop\src\libsync\discovery.cpp:1368 ]: Discovered "Documents/Dokumentumok/Other/inet számla/26105684_1629834407063421_1422797738_n.jpg" CSyncEnums::CSYNC_INSTRUCTION_NONE OCC::SyncFileItem::None CSyncEnums::ItemTypeVirtualFile
2022-12-31 10:03:41:389 [ info sync.discovery C:\Users\sysadmin\AppData\Local\Temp\2\windows-8634\client-building\desktop\src\libsync\discovery.cpp:362 ]:  Processing "Documents/Dokumentumok/Other/inet számla/26176926_1629834723730056_2018223740_n.jpg" | valid: true/true/db | mtime: 1514397321/1514397321/0 | size: 29817/29817/0 | etag: "cdbad2f8f7b8ce9c4f82583f38a9db20"//"" | checksum: ""//"" | perm: "WDNVR"//"" | fileid: "20462907ockyzphxy1jr"//"" | inode: 839876/839876/ | type: CSyncEnums::ItemTypeVirtualFile/CSyncEnums::ItemTypeVirtualFile/CSyncEnums::ItemTypeFile | e2ee: false/false | e2eeMangledName: ""/""
2022-12-31 10:03:41:389 [ info sync.discovery C:\Users\sysadmin\AppData\Local\Temp\2\windows-8634\client-building\desktop\src\libsync\discovery.cpp:1368 ]: Discovered "Documents/Dokumentumok/Other/inet számla/26176926_1629834723730056_2018223740_n.jpg" CSyncEnums::CSYNC_INSTRUCTION_NONE OCC::SyncFileItem::None CSyncEnums::ItemTypeVirtualFile
2022-12-31 10:03:41:389 [ debug nextcloud.sync.database.sql C:\Users\sysadmin\AppData\Local\Temp\2\windows-8634\client-building\desktop\src\common/ownsql.h:145 ]   [ OCC::SqlQuery::bindValue ]:   SQL bind 1 -5541886882352546247

I would expect the log to only contain information about special cases - being no smaller than a 100KB per sync run - not a detailed explanation of every atomic step during the sync process (this is a debug log, not a production log). For the people stumbling across this issue, and are wondering: you can find the logs under %appdata%\Nextcloud\logs on Windows, or C:\Users\username\AppData\Roaming\Nextcloud\logs.

Steps to reproduce

Have about 300 000 files for your user, and sync them using Windows virtual file support.
Check the size of your logs folder: %appdata%\Nextcloud\logs. It should have the size of multiple Gigabytes.

Expected behavior

I'd expect log files to only contain data about unexpected runtime errors that can help the team fix those errors. Not a detailed log of every atomic step during an otherwise uneventful sync.

Which files are affected by this bug

all of them

Operating system

Windows

Which version of the operating system you are running.

Windows 10 Pro 22H2, OS Build 19045.2364

Package

Appimage

Nextcloud Server version

25.0.2

Nextcloud Desktop Client version

3.4.1

Is this bug present after an update or on a fresh install?

Fresh desktop client install

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

Are you using an external user-backend?

[X] Default internal user-backend
[ ] LDAP/ Active Directory
[ ] SSO - SAML
[ ] Other

Nextcloud Server logs

No response

Additional info

No response

eibex commented 1 year ago

Related: #3312

eibex commented 1 year ago

A temporary fix could be adding a radio button to enable/disable logging until verbosity can be reduced only to errors.

JonathanxD commented 1 year ago

On Linux I've literally set the logs folder to read-only, I've tried everything and nothing helped.

I was working and suddenly my SSD ran out of space and the editor was closed abruptly because it failed to save its data. Nextcloud generated over 200GB of logs and the directory was full of compressed log files and a single text file with more than 150GB+.

I've then traced the problem to Dolphin, but Dolphin was not the culprit, it was just requesting file status for every file in my /home directory (which has 4 million files), this directory is not even set to sync, it's my external drive directory that is. And Nextcloud log every file status request attempt, this also caused Dolphin to use 6GB of RAM, after setting the logs directory to read-only, it's now using 100MB.

I don't know if this will also work for Windows (maybe Nextcloud can just try to change the permission on Windows, on Linux it's not possible without root privileges), but you can try as well. The only downside is that you lose all logs, but it's better than losing an NVMe SSD because of the massive wear.

user23498723452 commented 1 year ago

hmm, I use the appimage (previously a repo as well) on Ubuntu 20 (for years now) and have never had log file issues.

retnag commented 1 year ago

Iteresting. Just out of curiousity, did you check the size of the logs folder? What size is it? What version is the client? Perhaps a later change introduced this behavoiur? (I have also been using the client for years, and noticed this problem only recently, because my disk got close to being full)

hmm, I use the appimage (previously a repo as well) on Ubuntu 20 (for years now) and have never had log file issues.

retnag commented 1 year ago

Oh I just remembered, there is no virtual file support for linux (yet, I hope), so maybe there is the culprit, and this issue only impacts Windows users using the virtual file support.

bgeneto commented 1 year ago

We should have an option to reduce log level in the client. My nextcloud.cfg config file has the following options:

logDebug=false
logExpire=24

But I still have hundreds of megabytes in compressed log files!

gitskot commented 1 year ago

I have this issue also, although I didn't enable virtual files yet, the log folder ballooning to hundreds of gigabytes!

I am on Windows 10, just reinstalled the client to confirm the issue (now 3.8.1) My data is around 25GB, ~50.000 files.

I had to symlink the log folder to a spinning HD to move it, but that is not a solution because its growing too fast.

I really hope this will be fixed soon, I am surprised that you don't have more reports on this.

isdnfan commented 1 year ago

Same here - version 3.7.4 has 239MBs of zipped logs from one day only and I don't have as many files (~1TB / 100k files on server) - and I didn't made any excessive changes in the last days..

In my case there are sets of 4 files every 2 hours.. where archives of the same size are created

30MB archive is ~600MB extracted and it looks like it contains some log records to literally each file stored in my account.. even it didn't change at this time..

on sample of random directory record :

2023-05-15 11:27:15:680 [ info nextcloud.sync.discovery C:\Users\sysadmin\AppData\Local\Temp\2\windows-14239\client-building\desktop\src\libsync\discovery.cpp:423 ]:   Processing "Fotos/2021_Kulinarische_Reise" | (db/local/remote) | valid: true/true/db | mtime: 1673359040/1673359040/0 | size: 0/131072/0 | etag: "63bd6ec0c1975"//"" | checksum: ""//"" | perm: "DNVCKRm"//"" | fileid: "01395102occ5uwmhix0f"//"" | inode: 395154/395154/ | type: CSyncEnums::ItemTypeDirectory/CSyncEnums::ItemTypeDirectory/CSyncEnums::ItemTypeFile | e2ee: false/false | e2eeMangledName: ""/"" | file lock: not locked//
2023-05-15 11:27:15:680 [ debug nextcloud.sync.discovery C:\Users\sysadmin\AppData\Local\Temp\2\windows-14239\client-building\desktop\src\libsync\discovery.cpp:911 ]   [ OCC::ProcessDirectoryJob::processFileAnalyzeLocalInfo ]:  File "Fotos/2021_Kulinarische_Reise" - servermodified: false noServerEntry: false
2023-05-15 11:27:15:680 [ info nextcloud.sync.discovery C:\Users\sysadmin\AppData\Local\Temp\2\windows-14239\client-building\desktop\src\libsync\discovery.cpp:1540 ]:  Discovered "Fotos/2021_Kulinarische_Reise" CSyncEnums::CSYNC_INSTRUCTION_NONE OCC::SyncFileItem::None CSyncEnums::ItemTypeDirectory
2023-05-15 11:27:15:680 [ debug nextcloud.sync.discovery C:\Users\sysadmin\AppData\Local\Temp\2\windows-14239\client-building\desktop\src\libsync\discovery.cpp:59 ]    [ OCC::ProcessDirectoryJob::ProcessDirectoryJob ]:  "Fotos/2021_Kulinarische_Reise" OCC::ProcessDirectoryJob::ParentNotChanged "Fotos/2021_Kulinarische_Reise" OCC::ProcessDirectoryJob::NormalQuery 1684135631
2023-05-15 11:27:15:680 [ warning default C:\Users\sysadmin\AppData\Local\Temp\2\windows-14239\client-building\desktop\src\csync\csync_exclude.cpp:445 ]:   System exclude list file could not be read: "C:/Nextcloud/Fotos/2022_ZuHause/.sync-exclude.lst"

another sample is for a file:

2023-05-15 11:27:32:069 [ info nextcloud.sync.discovery C:\Users\sysadmin\AppData\Local\Temp\2\windows-14239\client-building\desktop\src\libsync\discovery.cpp:423 ]:   Processing "Fotos/#Archiv/2019/2019_wwe_handy/.@__thumb/default20190301_111949.jpg" | (db/local/remote) | valid: true/true/db | mtime: 1551435589/1551435589/0 | size: 43794/43794/0 | etag: "e16afd6cfb80a1f98536bed3939e187e"//"" | checksum: ""//"" | perm: "WDNVRm"//"" | fileid: "01437551occ5uwmhix0f"//"" | inode: 1165765/1165765/ | type: CSyncEnums::ItemTypeVirtualFile/CSyncEnums::ItemTypeVirtualFile/CSyncEnums::ItemTypeFile | e2ee: false/false | e2eeMangledName: ""/"" | file lock: not locked//
2023-05-15 11:27:32:069 [ debug nextcloud.sync.discovery C:\Users\sysadmin\AppData\Local\Temp\2\windows-14239\client-building\desktop\src\libsync\discovery.cpp:911 ]   [ OCC::ProcessDirectoryJob::processFileAnalyzeLocalInfo ]:  File "Fotos/#Archiv/2019/2019_wwe_handy/.@__thumb/default20190301_111949.jpg" - servermodified: false noServerEntry: false

take a look at servermodified: false - this folder was not touched on either on the client nor on the server.. what is the reason to log it? In general constant looping over all known items doesn't sounds very resource effective.

UPDATE: adding

logDebug=false
logExpire=24

doesn't change anything.. still 30MB archive (600MB plain log) is written every 2 hours :(

isdnfan commented 1 year ago

hopefully the problem is improved due to #5634

UPDATE:

updating to 3.8.2 seems no improvement for fresh start. now the client write thousands of 30k archive files each holding 512k of the log (but still around 30MB in total after client restart)..
will keep an eye on how it works while normal operations.

UPDATE 2:

client logs are split into 512k chunks now. so the archive files are smaller now. there many many small archives every 30 to 90 minutes
logs still contain useless lines for each existing file as mentioned above:

2023-05-24 10:15:22:012 [ info nextcloud.sync.discovery C:\Users\sysadmin\AppData\Local\Temp\windows-15401\client-building\desktop\src\libsync\discovery.cpp:1540 ]:    Discovered "113917_VFS_huge_folders/5/4/180_toucan.jpg" CSyncEnums::CSYNC_INSTRUCTION_NONE OCC::SyncFileItem::None CSyncEnums::ItemTypeFile
2023-05-24 10:15:22:012 [ info nextcloud.sync.discovery C:\Users\sysadmin\AppData\Local\Temp\windows-15401\client-building\desktop\src\libsync\discovery.cpp:423 ]: Processing "113917_VFS_huge_folders/5/4/181_toucan.jpg" | (db/local/remote) | valid: true/true/db | mtime: 1631739264/1631739264/0 | size: 167989/167989/0 | etag: "69eefec211d73274356c293ad40cbf2b"//"" | checksum: "SHA1:6846e9d9634e37c8abce44449b42a59d57a5de3f"//"" | perm: "WDNVR"//"" | fileid: "00125140oc52dthqts8g"//"" | inode: 809943/809943/ | type: CSyncEnums::ItemTypeFile/CSyncEnums::ItemTypeFile/CSyncEnums::ItemTypeFile | e2ee: false/false | e2eeMangledName: ""/"" | file lock: not locked//
2023-05-24 10:15:22:012 [ debug nextcloud.sync.discovery C:\Users\sysadmin\AppData\Local\Temp\windows-15401\client-building\desktop\src\libsync\discovery.cpp:911 ] [ OCC::ProcessDirectoryJob::processFileAnalyzeLocalInfo ]:  File "113917_VFS_huge_folders/5/4/181_toucan.jpg" - servermodified: false noServerEntry: false
2023-05-24 10:15:22:012 [ info nextcloud.sync.discovery C:\Users\sysadmin\AppData\Local\Temp\windows-15401\client-building\desktop\src\libsync\discovery.cpp:1540 ]:    Discovered "113917_VFS_huge_folders/5/4/181_toucan.jpg" CSyncEnums::CSYNC_INSTRUCTION_NONE OCC::SyncFileItem::None CSyncEnums::ItemTypeFile
2023-05-24 10:15:22:012 [ info nextcloud.sync.discovery C:\Users\sysadmin\AppData\Local\Temp\windows-15401\client-building\desktop\src\libsync\discovery.cpp:423 ]: Processing "113917_VFS_huge_folders/5/4/182_toucan.jpg" | (db/local/remote) | valid: true/true/db | mtime: 1631739264/1631739264/0 | size: 167989/167989/0 | etag: "1217cdac297e6de6c240b3c9d75ff5c9"//"" | checksum: "SHA1:6846e9d9634e37c8abce44449b42a59d57a5de3f"//"" | perm: "WDNVR"//"" | fileid: "00125141oc52dthqts8g"//"" | inode: 809746/809746/ | type: CSyncEnums::ItemTypeFile/CSyncEnums::ItemTypeFile/CSyncEnums::ItemTypeFile | e2ee: false/false | e2eeMangledName: ""/"" | file lock: not locked//
2023-05-24 10:15:22:012 [ debug nextcloud.sync.discovery C:\Users\sysadmin\AppData\Local\Temp\windows-15401\client-building\desktop\src\libsync\discovery.cpp:911 ] [ OCC::ProcessDirectoryJob::processFileAnalyzeLocalInfo ]:  File "113917_VFS_huge_folders/5/4/182_toucan.jpg" - servermodified: false noServerEntry: false

I think if the log output of ..\client-building\desktop\src\libsync\discovery.cpp lines 423, 911 and 1540 is removed/moved to debug log level the amount of logging will drop significant.

@allexzander as you are working on useless logs already would be great you can look at this issue as well.

quicktrick commented 1 year ago

The number of log files is growing at an alarming rate. We need to do something about it immediately.

isdnfan commented 1 year ago

I'm not very sure about the size of the logs.. I 'm under impression the size is reduced with the last version but there was a huge user experience regression - now an archive is generated for every 512K log - resulting in insane number of of log archives.. my first sync of the day consists of 1291 files (33Mb).. two hours later same files count/size..

I don't think it changes the whole size of logging but makes it harder to follow/understand.

definitely killing useless logs by default is a MUST.

EDIT: another point is definitely personal data included in the logs. current implementation leaks all file names stored in the account through the logs. The warning introduced with #6237 is more like a joke if the user is expected to anonymize gigabytes of logs.

bgeneto commented 1 year ago

I have also seen thousands of gzipped nextcloud log files, lucky for me I have reserved a folder in my ramdisk for these files (my ssd thank me for it).

isdnfan commented 1 year ago

for me update to 3.9.0 reduces logging from 30MB archives every 2h to 20MB (likely due to #5796) not many but a right direction already.

quicktrick commented 1 year ago

I created a task in Task Scheduler that deletes all *.gz every hour.

gitskot commented 1 year ago

I just tried installing the Windows client again, now in version 3.9.0 - in only 1 hour I have 265MB of log files, logging every small action - this is totally weird behaviour and makes Nextcloud unusable for me, also - I can't recommend it to anyone in this state.

godfuture commented 1 year ago

Same here. For me it looks like the client keeps syncing in a loop. Every attempt a new logfile is created. A single attempt takes only a second till the next attempt is started (I can see the green icon in client turns quickly to the blue icon during the second of sync attempt). %appdata%/Nextcloud is being filled with log.gz

2023-09-16 03:27:42:735 [ info nextcloud.gui.socketapi C:\Users\sysadmin\AppData\Local\Temp\2\windows-17553\client-building\desktop\src\gui\socketapi\socketapi.cpp:338 ]: Lost connection QLocalSocket(0x21764436660)

Being a Nextcloud user since many years now, my impression is that the Nextcloud client is tested very badly. Often there have been issues after updates. Problems with data being deleted, corrupted or now endlessly logged. Not good at all!

Win10 x64 NC Client 3.9.4

isdnfan commented 1 year ago

I hoped 3.10 will fix the the issue due to: "Bugfix/adjust log levels by @mgallien in" #5796

but still huge logs - in my case this are around zipped 20MB (~250MB) logs on each client start and awake from sleep..

now the logs lines look like this - including very strange static string '[ info nextcloud.sync.discovery C:\Users\sysadmin\AppData\Local\Temp\2\windows-17670\client-building\desktop\src\libsync\discovery.cpp:464 ]'

${timestamp} [ info nextcloud.sync.discovery C:\Users\sysadmin\AppData\Local\Temp\2\windows-17670\client-building\desktop\src\libsync\discovery.cpp:464 ]: ${filename} | (db/local/remote) | valid: true/true/db | mtime: 1457627598/1457627598/0 | size: 6090/6090/0 | etag: "155c38ffd623e2d576c46464d66b6022"//"" | checksum: ""//"" | perm: "WDNVRm"//"" | fileid: "01683291occ5uwmhix0f"//"" | type: CSyncEnums::ItemTypeVirtualFile/CSyncEnums::ItemTypeVirtualFile/CSyncEnums::ItemTypeFile | e2ee: false/false | e2eeMangledName: ""/"" | file lock: not locked// | metadata missing: /false/

@mgallien - can we please got it sorted? there is no value of logging each existing file.. normal logs should only log when files are changed and require replication to or from the server.

retnag commented 1 year ago

I feel like it is overlooked, that most of the users don't even need the logs... I'm pretty sure, most don't even know they exist. Can't there just be a switch to turn logging on/off? It should be relatively easy to implement, and it would solve the issue. What speaks against it?

eibex commented 1 year ago

Can't there just be a switch to turn logging on/off? It should be relatively easy to implement, and it would solve the issue. What speaks against it?

Didn't look like a priority #3312

eibex commented 1 year ago

I find it infuriating that if I set the log folder to read only Nextcloud keeps popping up a message (and switching the window focus to it) every few seconds saying that the log folder isn't writable.

JP95Git commented 10 months ago

I also have this issue, it even persists in Nextcloud-3.11.0-x64.msi. 2 suggestions:

Either lower the log level to warning or error
Or do not log to much.

I had to delete the log-folder each day using manually or my partition runs out of space.

thvitt commented 5 months ago

BTW at least for Nextcloud 3.11 on Debian setting an environment variable QT_LOGGING_RULES for the Nextcloud process seems to help somewhat. QT_LOGGING_RULES='*=false' completely turns off all log messages, although the Nextcloud client still creates a gz file of compressed nothingness every minute or so.

To do that, you can copy /usr/share/dbus-1/services/com.nextcloudgmbh.Nextcloud.service to ~/.local/share/dbus-1/services/ (keep the filename) and /usr/share/applications/com.nextcloud.desktopclient.nextcloud.desktop to ~/.local/share/applications/ and then, in both copied files, adjust the Exec lines such that they start with Exec=/usr/bin/env QT_LOGGING_RULES='*=false' /usr/bin/nextcloud. Leave options after nextcloud as they are. If you have a .desktop file for NextCloud in ~/.config/autostart, copy/link the modified desktop file there, as well.

Julrob199 commented 2 months ago

I hope this gets fixed soon. For years this habit is wearing off my SSD.

An alternative workaround would also be to sync via rsync or so, but this is a bit fiddly.

LordBaryhobal commented 2 months ago

Currently facing the same issue with v3.13.2 (on Linux) It has become very painful to use Nextcloud because of the gigabytes of useless log. It is still viable on my PC but my laptop cannot handle constant writes without heating up like a barbecue My current workaround is to simply pause sync while I work and modify files, and turn it back on at the end, which feels more like using git than a cloud service

How has still not been solved, or at least been improved ? Maybe I'm missing something

retnag commented 2 months ago

I find it infuriating that if I set the log folder to read only Nextcloud keeps popping up a message (and switching the window focus to it) every few seconds saying that the log folder isn't writable.

Yes, I find it infuriating too. I had once spent some time trying to build a custom fork of client myself, where that error is disabled (for my Windows machine I needed the custom client. I switched back to Linux a while ago and have the logs folder mounted as a nullfs now, which fixes the issue for me with minimal drawbacks, but I doubt that's possible using Windows). I found a somewhat confusing documentation on how to build the Nextcloud client, but I'm also not used to working with the tech-stack of the client, so not getting the docs might just be a "me" problem.

....Anyway, even so, I managed to find the one line that pops up that error message and yeets focus away from the active window every 5 minutes, and commented it out. That's all it took. Just had to search for the error message and I found it pretty quick (It was a while back, I dont remember the file or line number, and dont have the source checked out anymore to check.) Took me some hours actually getting it built, but I was able to, I imagine the Nextcloud team could get this done in about 5 minutes. If you can't push a real fix to this Issue for some years now, please consider just removing that error message so it becomes possible to work around this issue by manually denying writes to the log-folder.

For all the linux users out there, I recommend mounting the log-folder as a filesystem that basically just deletes everything you write onto it... You wont get errors from Nextcloud, it doesn't even notice that something is wrong. I'm on Arch Linux, I installed this package from the AUR fuse-nullfs-git, and then put the following lines in my fstab:

# workaround for the tremenduous amounts of logfiles that  nextcloud creates
none    /var/log/nextcloud-client    fuse.nullfsfuse    nonempty

For different distros installation will ofc be different.

isdnfan commented 2 months ago

the was a pull request #5780 addressing the logs issue and setting the useless "discovered" and "processing" logs to debug level.. for some reason major developer is reluctant to accept it. for now the only workaround remains to write the logs onto nullfs or ramdisk

chainria commented 3 weeks ago

After numerous times Nextcloud filled up my /home, after numerous days of cleaning the log folder, I eventually decided for this step:

cd .config/Nextcloud rm -rf logs && ln -s /tmp logs

My /tmp is a tmpfs with about 16 GB of space (using it to compile Gentoo Ebuilds). It can have it's fun there now. The log size defeats the purpose of the logs anyway. Even IF Nextcloud decides to delete everything again, I won't bother to look up in Gigabytes of files what happened. I'll just restore the backup I maintain since the last times that happened.

Still, I really hope this gets adressed soon.

limes007 commented 2 weeks ago

Setting env variable QT_LOGGING_RULES='*=false' as described by @thvitt helps. Thank you. But empty gz-Files will still be created - every 7 seconds for me!

It would be great to fix this bug.

gheesh commented 1 week ago

Setting env variable QT_LOGGING_RULES='*=false' as described by @thvitt helps. Thank you. But empty gz-Files will still be created - every 7 seconds for me!

It would be great to fix this bug.

If you don't want to completely disable logging, this worked for me: QT_LOGGING_RULES='*.debug=false' Even with info messages it is now down to a few KB per minute, except for the initial run (as expected)

nextcloud / desktop