Closed olizimmermann closed 1 year ago
I tried your patch and unfortunately it doesn't solve the problem:
$ du -sh .../data/user1/files
2.0G data/user1/files
$ php occ user:info user1
[...]
- quota: 4 GB
- storage:
- free: 2263992505
- used: 3643321208
- total: 4294967296
- relative: 84.83
- quota: 4294967296
[...]
$ patch -p1 < ~/35748.patch
[...]
$ php occ files:scan user1
$ php occ user:info user1
[...]
- quota: 4 GB
- storage:
- free: 2263992505
- used: 3643321208
- total: 4294967296
- relative: 84.83
- quota: 4294967296
[...]
@SudipBatabyal I just noticed your comment
problem users not shown in this comment, problem users are different
ok, so if it's not in the database then your issue is a different one you can check oc_filecache for the problematic user and see what value of "size" is on the one where path='' and the one where path='files' of that user's storage, because that's what's used for the web UI.
are you on a 32-bit system ? there are known problems with bigger numbers there due to PHP limitations
System ubuntu 22.04 64bit on Microsoft hyper-V, nextcloud version 25.02 PHP: 8.1.2
@t2d @doobry-systemli please run the query from https://github.com/nextcloud/server/issues/25283#issuecomment-1346825930 to see if the size discrepancies appear for you
for some people the problem is size discrepancies and for others, currently still a mystery. with more clues from people we might be able to find common patterns
I've made a PR with the change: #35748 will need to discuss with @icewind1991 if this is viable once he's back
in the meantime, it would be good if more people here could test that change:
- apply the patch https://patch-diff.githubusercontent.com/raw/nextcloud/server/pull/35748.patch
- run
occ files:scan --all
- rerun the query from Nextcloud displays wrong used storage for one user #25283 (comment) to confirm that discrepancies are gone
I have followed your steps, but the problem still exists
@t2d @doobry-systemli please run the query from #25283 (comment) to see if the size discrepancies appear for you
This query only lists six entries, and not the problematic users where we know about the quota discrepancy. The groupfolders storage entries must be very old, we don't have groupfolders active since at least two years.
MariaDB [nextcloud]> select s.id storage_id, s.numeric_id storage_numeric_id, fc.parent parentfileid, fcp.path, fcp.size, sum(fc.size) "size_of_chidlren" from filecache fc left join filecache fcp on fcp.fileid=fc.parent left join storages s on s.numeric_id=fc.storage where fcp.path not like 'appdata_%' group by storage_id, storage_numeric_id, parentfileid, fcp.path, fcp.size having fcp.size != sum(fc.size) and fcp.size >= 0 order by fcp.storage, parentfileid;
+------------------------------------------+--------------------+--------------+-----------------------------------------------------------------------+----------+------------------+
| storage_id | storage_numeric_id | parentfileid | path | size | size_of_chidlren |
+------------------------------------------+--------------------+--------------+-----------------------------------------------------------------------+----------+------------------+
| local::/var/www/cloud.systemli.org/data/ | 82 | 1273139 | __groupfolders/versions/1/1273136 | 0 | 904144 |
| local::/var/www/cloud.systemli.org/data/ | 82 | 1273145 | __groupfolders/versions/1/1273144 | 0 | 1 |
| local::/var/www/cloud.systemli.org/data/ | 82 | 1565639 | userABC/files/some/path/to/a/folder | 3013968 | 978303 |
| home::userX | 776 | 611750 | | 21394344 | 22371126 |
| home::userY | 862 | 771950 | | 36683760 | 36683758 |
| home::userZ | 874 | 782246 | | 4646273 | 0 |
+------------------------------------------+--------------------+--------------+-----------------------------------------------------------------------+----------+------------------+
6 rows in set (4.417 sec)
for those where the used storage is wrong and occ files:scan doesn't fix it, how do you know that the used storage is wrong ? I'm guessing that the result of running du
on the user's folder is different than what's in oc_filecache ?
are you able to find out which subfolder has a different size in Nextcloud ? I assume no one is using symlinks/hard links or anything (those are not supported by the way)
for those where the used storage is wrong and occ files:scan doesn't fix it, how do you know that the used storage is wrong ? I'm guessing that the result of running
du
on the user's folder is different than what's in oc_filecache ?are you able to find out which subfolder has a different size in Nextcloud ? I assume no one is using symlinks/hard links or anything (those are not supported by the way)
@SudipBatabyal can you also check the two hidden entries, in case they were omitted in the calculation ?
focussing on this user, please run the following: occ user:info $userId
and: select s.id,fc.path,fc.size from oc_storages s left join oc_filecache fc on s.numeric_id=fc.storage and s.id like 'home::$userId' and path in ('', 'files');
and replace "$userId" accordingly above.
I assume that this user does not have access to any external storage, I didn't see such icon in the screenshot.
@SudipBatabyal can you also check the two hidden entries, in case they were omitted in the calculation ?
focussing on this user, please run the following: occ user:info $userId
and:
select s.id,fc.path,fc.size from oc_storages s left join oc_filecache fc on s.numeric_id=fc.storage and s.id like 'home::$userId' and path in ('', 'files');
and replace "$userId" accordingly above.
I assume that this user does not have access to any external storage, I didn't see such icon in the screenshot. #
for those where the used storage is wrong and occ files:scan doesn't fix it, how do you know that the used storage is wrong ? I'm guessing that the result of running
du
on the user's folder is different than what's in oc_filecache ?are you able to find out which subfolder has a different size in Nextcloud ? I assume no one is using symlinks/hard links or anything (those are not supported by the way)
2.9 GB folder actual data is 2.7 GB ...........
for those where the used storage is wrong and occ files:scan doesn't fix it, how do you know that the used storage is wrong ? I'm guessing that the result of running
du
on the user's folder is different than what's in oc_filecache ? are you able to find out which subfolder has a different size in Nextcloud ? I assume no one is using symlinks/hard links or anything (those are not supported by the way)2.9 GB folder actual data is 2.7 GB ...........
for those where the used storage is wrong and occ files:scan doesn't fix it, how do you know that the used storage is wrong ? I'm guessing that the result of running
du
on the user's folder is different than what's in oc_filecache ?are you able to find out which subfolder has a different size in Nextcloud ? I assume no one is using symlinks/hard links or anything (those are not supported by the way)
Yes, I know via du
and because I deleted a lot of files in my user but the space is never freed.
Hi @PVince81
unfortunately I also have to report that while removing the REUSE_SIZE
flag does indeed seem to update the odd entries found by your query, this does not change anything about the problem - so it seems to lay somewhere else.
@simonspa to delete those obsolete storage entries from the database, for you specifically:
Hm, I don't feel comfortable doing that, a select fileid, path from oc_filecache where storage=1
gives me more than 800 entries. Are you positive these are all old and unused?
you can check if manually changing the size value also changes the displayed quota for that user
Done that, it actually changes the value properly only for the first entry with empty path
, when I try to do the same for the filedid=591
, the line still appears in the query afterwards. Also, there is no change in displayed quota.
for those where the used storage is wrong and occ files:scan doesn't fix it, how do you know that the used storage is wrong ?
Yes, du
, but in my case it is obvious for many users, like my own one - I have years of data in there, with videos and pictures, but the displayed quota is only 5.2 MB. (du
tells me it should be around 85GB)
select s.id,fc.path,fc.size from oc_storages s left join oc_filecache fc on s.numeric_id=fc.storage and s.id like 'home::$userId' and path in ('', 'files');
Somehow this query returns a long list of id
s, many not like home::smn
- not sure if the left join is fully correct. However, there are two entries for the user in question:
| home::smn | files | 81520013126 |
| home::smn | | 91045664726 |
The patch did not work for me either. Here are my discrepancies.
- quota: 500 GB
- storage:
- free: 283573556349
- used: 82293399
- total: 283655849748
- relative: 0.03
- quota: 536870912000
[root@nextcloud jarom]# du -sh files
236G files
Quota is set to 500GB, the DB reflects that but the UI shows 264.2GB. There are 236GB of files but the DB shows 82MB and the UI shows 78.5MB.
The patch did not work for me either
assuming you also ran occ files:scan --all
after the patch ?
just checking
also you are on 25.0.2 ?
the occ command is calling https://github.com/nextcloud/server/blob/v25.0.2/core/Command/User/Info.php#L92 the web UI is calling https://github.com/nextcloud/server/blob/v25.0.2/apps/files/lib/Controller/ViewController.php#L155 which is calling https://github.com/nextcloud/server/blob/v25.0.2/lib/private/legacy/OC_Helper.php#L465
for some reason I see that includeMountPoints
is set to true by default, which might make it also count the contents of received shares, but it's not supposed to
Still, including additional data would not explain the used quota two orders of magnitude smaller than what is on disk, right?
Yes I ran sudo -u apache php occ files:scan jarom
and I am on 25.0.2.
{
installed: true,
maintenance: false,
needsDbUpgrade: false,
version: "25.0.2.3",
versionstring: "25.0.2",
edition: "",
productname: "Nextcloud",
extendedSupport: false
}
Still, including additional data would not explain the used quota two orders of magnitude smaller than what is on disk, right?
indeed
I'll need to set aside some time because I have trouble juggling so many different scenarios posted by everyone.
might need to add additional logging in the quota reporting code from https://github.com/nextcloud/server/issues/25283#issuecomment-1348831470 to try and find out where those values are coming from.
apart from stuff that is included additionally, I also don't understand why sometimes it would report less than what's on disk and also be different than the value in oc_filecache's "files" entry.
Thanks a lot @PVince81 for digging into this! Please let me know if I can support you in any way, be it testing patches or gathering more information from my NC instance!
Still, including additional data would not explain the used quota two orders of magnitude smaller than what is on disk, right?
indeed
I'll need to set aside some time because I have trouble juggling so many different scenarios posted by everyone.
might need to add additional logging in the quota reporting code from #25283 (comment) to try and find out where those values are coming from.
apart from stuff that is included additionally, I also don't understand why sometimes it would report less than what's on disk and also be different than the value in oc_filecache's "files" entry.
Definitely, it is a table cache issue not on the file storage label & it is a major bug that comes across with NC nowadays. Install a fresh version of NC latest into our virtual environment but the issue remains the same wrong storage count.
Install a fresh version of NC latest into our virtual environment but the issue remains the same wrong storage count.
With a fresh version ? that surprises me, because I was assuming that the quota issue is related to inconsistencies in the database. I don't see how this could happen in a fresh environment with just a few uploaded files.
For this case, one theory is that maybe PHP isn't reporting the free space correctly: on some partition setups (LVM?) it can happen that PHP's disk_free_space()
function doesn't report the correct value.
You can test it by running this PHP snippet and replace the path with the one from the data directory:
<?php
echo(disk_free_space('/srv/www/htdocs/server/data'));
<?php echo(disk_free_space('/srv/www/htdocs/server/data'));
Returns:
370574577664
vs df -B 1
370574426112
So the two methods differ but not by much. This is data being stored on the root partition of an LVM.
In my case, the errors only seems to appear in the web client. The photo upload from the mobile client still works. Is there some web-specific cache that I can clear?
So I did another round of tests (on Nextcloud 24.0.8). In our case it's pretty obvious that the calculated used quota doesn't get updated after deleting a file. What I did:
Running occ files:scan <user>
didn't change anything. The user doesn't have any shares or external storage.
Yes, This is the same issue I also facing & also have a new issue with this:
Some even more curious situation, even after going through all the cleaning steps discussed above:
I have one user that has a quota set to 200 GB in the User settings:
But checking in the Files app of this user, I get presented with the following:
If I increase the quota to 300GB, the listed available space in the files app changes to 227.3 GB, while the 300 GB are reported in the Users list.
Something is really going awry with the quota calculation and size propagation here and it would be great if you @PVince81 could find the time to look into this deeply!
Dear Nextcloud devs, is there anything we can do to support you with further debugging the issue? It breaks usage of the Nextcloud instance for lots of users and would be really nice to get fixed.
Same issue with a user on my instance (24.0.8 docker official image with a PostgreSQL DB):
id | path | size
------------------------------------------+-------+------------
home::username | files | 1959307369
home::username | | 1959750985
- storage:
- free: 8778110871
- used: 8984372973
- total: 10737418240
- relative: 83.67
- quota: 10737418240
I can perform test if needed (modify file data, patch code, run DB query, etc...)
As far as I understand the quota provided to the end user is from the unencrypted_size value from oc_filecache where path='files'.
I noticed that indeed, the oc_filecache table is not updated consistently. the size column seems to be updated when adding or removing a file, but the unencrypted_size column is only updated when adding and not when removing.
Here the DB evolution for my user during the scenario:
request launched:
select storage, size, unencrypted_size, etag from oc_filecache where storage=1 and path='files';
Scenario:
storage | size | unencrypted_size | etag
---------+--------+------------------+---------------
1 | 672559 | 9855121 | 63c4232aeb871
storage | size | unencrypted_size | etag
---------+--------+------------------+---------------
1 | 919659 | 10102221 | 63c424c62dc13
storage | size | unencrypted_size | etag
---------+--------+------------------+---------------
1 | 672559 | 10102221 | 63c424f722f21
storage | size | unencrypted_size | etag
---------+--------+------------------+---------------
1 | 672559 | 10102221 | 63c424f722f21
I tried to look at the code, but I am not experienced enough to perform a proper analysis.
Thanks a lot for the detailed analysis, @doc75 - that could at least explain the cases reported above in which the displayed quota exceeds the actually existing files.
Do you also have an idea how the opposite could happen, i.e. few MB reported quota but tens of GB actual data?
Do you also have an idea how the opposite could happen, i.e. few MB reported quota but tens of GB actual data?
I have not seen that during my scenario. I am on a standard install with no external storage
I am on a standard install with no external storage
Same here, this is a standard installation, all data on local storage, data directory residing outside webroot.
Hello,
@doc75 Do you know how you ended up with unencrypted_size not being 0? Did you have encryption enabled in the past?
On a usual installation without encryption activated unencrypted_size should always be 0. Can you show the value of the encrypted column for these files?
@doc75 Do you know how you ended up with unencrypted_size not being 0? Did you have encryption enabled in the past?
Encryption is activated in case a user wants to use external storage. The main storage space is not encrypted. Let me some days to provide the information about encrypted size (no easy access for now), but not sure this will be relevant knowing my above statement. But I’ll provide it.
I just checked, I had encryption enabled a long time ago (NC 11 maybe?) but do not have the encryption module active now. What I see is:
MariaDB [nextcloud]> select storage, size, encrypted, unencrypted_size, etag from oc_filecache where storage=1 and path='files';
+---------+-----------+-----------+------------------+---------------+
| storage | size | encrypted | unencrypted_size | etag |
+---------+-----------+-----------+------------------+---------------+
| 1 | 172172644 | 0 | 127431023 | 547351ba45033 |
+---------+-----------+-----------+------------------+---------------+
we managed to somewhat reproduce the issue where unencrypted size for "files" is non-zero on git master
it required to have encrypted enabled and overwriting a file multiple times
then when decrypting all files and disabling encryption, the unencrypted size was still non-zero
workaround:
0) make sure encryption is disabled (if you are using encryption do not use this solution, the issue might be another one!)
1) backup database
2) update oc_filecache set unencrypted_size=0
@doc75 in your case you might want to apply the query only the "home::*" storages from oc_storages but not external storages. I was not able to test with a mix of encrypted and non-encrypted storages, the behavior might be different.
@icewind1991 there seems to be many issues related to that column, for example we noticed that a first upload of a file kept it zero, but overwriting set it to a value. Also we never saw "size" being different than "unencrypted_size" so it looks like we're not storing the encrypted size any more ? would be good to do more testing of size propagation
A TODO: when decrypting a user's storage in occ encryption:decrypt-all
, at the end of the process, run a one-pass query to set unencrypted_size to 0. @come-nc maybe ?
also wondering if we should just add a repair step that resets unencrypted_size if encryption is clearly disabled, this would hopefully cover all cases without need for a workaround
update oc_filecache set unencrypted_size=0
It works for me! Wonderfull !!!!
(Nextcloud 25.0.3, a lot of external storage, 10k users)
Wow, great, thanks a lot for the investigation and the progress @PVince81!
Wouldn't it be better to run
update oc_filecache set unencrypted_size=0 where encrypted=0;
to make sure we only reset this for unencrypted files?
I just did the update with the additional where-clause:
UPDATE oc_filecache SET unencrypted_size=0 WHERE encrypted = 0
worked perfectly for all of my instances!
Exactly the same problem here:
Quota information: 92.4 GB out of 200 GB
du -h on the user directory: 63 GB
select s.id,fc.path,fc.size from oc_storages s left join oc_filecache fc on s.numeric_id=fc.storage and s.id like 'home::user1' and path in ('', 'files');
id | path | size
------------------------------------------+-------+------------
home:: user1 | | 65218908905
home:: user1 | files | 59873031084
occ user:info user1
- storage:
- free: 45267734528
- used: 99379665318
- total: 144647399846
- relative: 68.7
- quota: 214748364800
Already tried to do a files:scan without REUSE_SIZE as proposed in this comment. Noticed this install also was initially an install with encryption enabled, after some problems we disabled encryption a long time ago. After a:
UPDATE oc_filecache SET unencrypted_size=0 WHERE encrypted = 0
It seems to correct itself as well. Still seems weird there are any files with encrypted greater than 0 on an install without encryption enabled? From a quick look it seems these are all files in files_trashbin/versions, is it safe to remove these DB entries?
a big thanks to @doc75 also who thought about looking at unencrypted_size
the query update oc_filecache set unencrypted_size=0 where encrypted=0;
is indeed safer
if encryption is disabled and you still see entries with encrypted>0 outside of files, check if the original files for those still exist and if not you can delete the entries from the database and also from the data directory.
Voila, I can also report success, this was really the culprit. :tada: Thanks @PVince81 @doc75 !
Could this be something that could be performed as part e.g. of occ maintenance:repair
automatically?
Could this be something that could be performed as part e.g. of occ maintenance:repair automatically?
yes, that could be a good idea, at least until the root cause can be analyzed and fixed
As @prismopensource writes:
Noticed this install also was initially an install with encryption enabled, after some problems we disabled encryption a long time ago. After a:
UPDATE oc_filecache SET unencrypted_size=0 WHERE encrypted = 0
It seems to correct itself as well. Still seems weird there are any files with encrypted great than 0 on an install without encryption enabled?
@PVince81 at least for a fraction of the affected databases at some point encryption was enabled and then turned off, in some cases a decade ago :) Maybe it has to do with the disabling of encryption? At least for me it was very likely the root cause.
Maybe it has to do with the disabling of encryption? At least for me it was very likely the root cause.
FWIW encryption was enabled and active when I first had the issue.
more tests will be needed to see if "unencrypted_size" is properly updated when encryption is enabled. someone reported earlier that it's possible that some operations like file deletion do not properly propagate the size change
so what needs to be done now:
occ maintenance:repair
to always run UPDATE oc_filecache SET unencrypted_size=0 WHERE encrypted = 0
. @simonspa => https://github.com/nextcloud/server/pull/36220occ encryption:decrypt-all
to reset unencrypted sizes to 0 when done. Any takers ?doc PR for the workaround: https://github.com/nextcloud/documentation/pull/9566
How to use GitHub
Steps to reproduce
Expected behaviour
Tell us what should happen Should be 1:1 like all my other users.
Actual behaviour
Tell us what happens instead Real used storage: 9.3 Gb Nextcloud shows me: 18.2
Server configuration
Operating system: FreeBsd Web server: Nginx Database:
PHP version: 7.04 Nextcloud version: (see Nextcloud admin page) 20.0.5 Updated from an older Nextcloud/ownCloud or fresh install:
Where did you install Nextcloud from: TrueNas - Plugin - latest Update via Updater Signing status:
Signing status
``` Login as admin user into your Nextcloud and access http://example.com/index.php/settings/integrity/failed paste the results here. ``` No errors have been found.List of activated apps:
App list
Notes, Decks, Tasks ``` If you have access to your command line run e.g.: sudo -u www-data php occ app:list from within your Nextcloud installation folder ``` ---Nextcloud configuration:
Config report
``` If you have access to your command line run e.g.: sudo -u www-data php occ config:list system from within your Nextcloud installation folder or Insert your config.php content here. Make sure to remove all sensitive content such as passwords. (e.g. database password, passwordsalt, secret, smtp password, …) ``` CONFIG = array ( 'apps_paths' => array ( 0 => array ( 'path' => '/usr/local/www/nextcloud/apps', 'url' => '/apps', 'writable' => true, ), 1 => array ( 'path' => '/usr/local/www/nextcloud/apps-pkg', 'url' => '/apps-pkg', 'writable' => true, ), ), 'logfile' => '/var/log/nextcloud/nextcloud.log', 'passwordsalt' => 'XXXXX', 'secret' => 'XXXXX', 'trusted_domains' => array ( 0 => 'localhost', ), 'datadirectory' => '/usr/local/www/nextcloud/data', 'dbtype' => 'mysql', 'version' => '20.0.5.2', 'overwrite.cli.url' => 'http://localhost', 'overwriteprotocol' => 'https', 'dbname' => 'nextcloud', 'dbhost' => 'localhost', 'dbport' => '', 'dbtableprefix' => 'oc_', 'mysql.utf8mb4' => true, 'dbuser' => 'XXX', 'dbpassword' => 'XXX', 'installed' => true, 'instanceid' => 'XXX', 'twofactor_enforced' => 'false', 'twofactor_enforced_groups' => array ( 0 => 'admin', ), 'twofactor_enforced_excluded_groups' => array ( ), 'mail_smtpmode' => 'smtp', 'mail_sendmailmode' => 'smtp', 'mail_domain' => 'gmail.com', 'mail_smtphost' => 'smtp.gmail.com', 'mail_smtpport' => '587', 'mail_smtpsecure' => 'tls', 'mail_smtpauthtype' => 'LOGIN'..Are you using external storage, if yes which one: local/smb/sftp/... No Are you using encryption: yes/no No Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/... No
LDAP configuration (delete this part if not used)
LDAP config
``` With access to your command line run e.g.: sudo -u www-data php occ ldap:show-config from within your Nextcloud installation folder Without access to your command line download the data/owncloud.db to your local computer or access your SQL server remotely and run the select query: SELECT * FROM `oc_appconfig` WHERE `appid` = 'user_ldap'; Eventually replace sensitive data as the name/IP-address of your LDAP server or groups. ```Client configuration
Browser: all / also in app Operating system:
Logs
Web server error log
Web server error log
``` Insert your webserver log here ```Nextcloud log (data/nextcloud.log)
Nextcloud log
``` Insert your Nextcloud log here ```Shell:
Online: