nextcloud / server

☁️ Nextcloud server, a safe home for all your data
https://nextcloud.com
GNU Affero General Public License v3.0
27.17k stars 4.04k forks source link

Nextcloud displays wrong used storage #25283

Closed olizimmermann closed 1 year ago

olizimmermann commented 3 years ago

How to use GitHub

Steps to reproduce

  1. Install Nextcloud
  2. Create some users
  3. Upload some data
  4. check at the users overview the used storage
  5. compare it with the real used storage within the users folder via shell

Expected behaviour

Tell us what should happen Should be 1:1 like all my other users.

Actual behaviour

Tell us what happens instead Real used storage: 9.3 Gb Nextcloud shows me: 18.2

Server configuration

Operating system: FreeBsd Web server: Nginx Database:

PHP version: 7.04 Nextcloud version: (see Nextcloud admin page) 20.0.5 Updated from an older Nextcloud/ownCloud or fresh install:

Where did you install Nextcloud from: TrueNas - Plugin - latest Update via Updater Signing status:

Signing status ``` Login as admin user into your Nextcloud and access http://example.com/index.php/settings/integrity/failed paste the results here. ``` No errors have been found.

List of activated apps:

App list Notes, Decks, Tasks ``` If you have access to your command line run e.g.: sudo -u www-data php occ app:list from within your Nextcloud installation folder ``` ---

Nextcloud configuration:

Config report ``` If you have access to your command line run e.g.: sudo -u www-data php occ config:list system from within your Nextcloud installation folder or Insert your config.php content here. Make sure to remove all sensitive content such as passwords. (e.g. database password, passwordsalt, secret, smtp password, …) ``` CONFIG = array ( 'apps_paths' => array ( 0 => array ( 'path' => '/usr/local/www/nextcloud/apps', 'url' => '/apps', 'writable' => true, ), 1 => array ( 'path' => '/usr/local/www/nextcloud/apps-pkg', 'url' => '/apps-pkg', 'writable' => true, ), ), 'logfile' => '/var/log/nextcloud/nextcloud.log', 'passwordsalt' => 'XXXXX', 'secret' => 'XXXXX', 'trusted_domains' => array ( 0 => 'localhost', ), 'datadirectory' => '/usr/local/www/nextcloud/data', 'dbtype' => 'mysql', 'version' => '20.0.5.2', 'overwrite.cli.url' => 'http://localhost', 'overwriteprotocol' => 'https', 'dbname' => 'nextcloud', 'dbhost' => 'localhost', 'dbport' => '', 'dbtableprefix' => 'oc_', 'mysql.utf8mb4' => true, 'dbuser' => 'XXX', 'dbpassword' => 'XXX', 'installed' => true, 'instanceid' => 'XXX', 'twofactor_enforced' => 'false', 'twofactor_enforced_groups' => array ( 0 => 'admin', ), 'twofactor_enforced_excluded_groups' => array ( ), 'mail_smtpmode' => 'smtp', 'mail_sendmailmode' => 'smtp', 'mail_domain' => 'gmail.com', 'mail_smtphost' => 'smtp.gmail.com', 'mail_smtpport' => '587', 'mail_smtpsecure' => 'tls', 'mail_smtpauthtype' => 'LOGIN'..

Are you using external storage, if yes which one: local/smb/sftp/... No Are you using encryption: yes/no No Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/... No

LDAP configuration (delete this part if not used)

LDAP config ``` With access to your command line run e.g.: sudo -u www-data php occ ldap:show-config from within your Nextcloud installation folder Without access to your command line download the data/owncloud.db to your local computer or access your SQL server remotely and run the select query: SELECT * FROM `oc_appconfig` WHERE `appid` = 'user_ldap'; Eventually replace sensitive data as the name/IP-address of your LDAP server or groups. ```

Client configuration

Browser: all / also in app Operating system:

Logs

Web server error log

Web server error log ``` Insert your webserver log here ```

Nextcloud log (data/nextcloud.log)

Nextcloud log ``` Insert your Nextcloud log here ```

Shell:

image

Online: image

doobry-systemli commented 1 year ago

I tried your patch and unfortunately it doesn't solve the problem:

$ du -sh .../data/user1/files
2.0G    data/user1/files
$ php occ user:info user1
[...]
  - quota: 4 GB
  - storage:
    - free: 2263992505
    - used: 3643321208
    - total: 4294967296
    - relative: 84.83
    - quota: 4294967296
[...]
$ patch -p1 < ~/35748.patch
[...]
$ php occ files:scan user1
$ php occ user:info user1
[...]
  - quota: 4 GB
  - storage:
    - free: 2263992505
    - used: 3643321208
    - total: 4294967296
    - relative: 84.83
    - quota: 4294967296
[...]
SudipBatabyal commented 1 year ago

@SudipBatabyal I just noticed your comment

problem users not shown in this comment, problem users are different

ok, so if it's not in the database then your issue is a different one you can check oc_filecache for the problematic user and see what value of "size" is on the one where path='' and the one where path='files' of that user's storage, because that's what's used for the web UI.

are you on a 32-bit system ? there are known problems with bigger numbers there due to PHP limitations

System ubuntu 22.04 64bit on Microsoft hyper-V, nextcloud version 25.02 PHP: 8.1.2

PVince81 commented 1 year ago

@t2d @doobry-systemli please run the query from https://github.com/nextcloud/server/issues/25283#issuecomment-1346825930 to see if the size discrepancies appear for you

for some people the problem is size discrepancies and for others, currently still a mystery. with more clues from people we might be able to find common patterns

janamedick commented 1 year ago

I've made a PR with the change: #35748 will need to discuss with @icewind1991 if this is viable once he's back

in the meantime, it would be good if more people here could test that change:

  1. apply the patch https://patch-diff.githubusercontent.com/raw/nextcloud/server/pull/35748.patch
  2. run occ files:scan --all
  3. rerun the query from Nextcloud displays wrong used storage for one user #25283 (comment) to confirm that discrepancies are gone

I have followed your steps, but the problem still exists

doobry-systemli commented 1 year ago

@t2d @doobry-systemli please run the query from #25283 (comment) to see if the size discrepancies appear for you

This query only lists six entries, and not the problematic users where we know about the quota discrepancy. The groupfolders storage entries must be very old, we don't have groupfolders active since at least two years.

MariaDB [nextcloud]> select s.id storage_id, s.numeric_id storage_numeric_id, fc.parent parentfileid, fcp.path, fcp.size, sum(fc.size) "size_of_chidlren" from filecache fc left join filecache fcp on fcp.fileid=fc.parent left join storages s on s.numeric_id=fc.storage where fcp.path not like 'appdata_%' group by storage_id, storage_numeric_id, parentfileid, fcp.path, fcp.size having fcp.size != sum(fc.size) and fcp.size >= 0 order by fcp.storage, parentfileid;
+------------------------------------------+--------------------+--------------+-----------------------------------------------------------------------+----------+------------------+
| storage_id                               | storage_numeric_id | parentfileid | path                                                                  | size     | size_of_chidlren |
+------------------------------------------+--------------------+--------------+-----------------------------------------------------------------------+----------+------------------+
| local::/var/www/cloud.systemli.org/data/ |                 82 |      1273139 | __groupfolders/versions/1/1273136                                     |        0 |           904144 |
| local::/var/www/cloud.systemli.org/data/ |                 82 |      1273145 | __groupfolders/versions/1/1273144                                     |        0 |                1 |
| local::/var/www/cloud.systemli.org/data/ |                 82 |      1565639 | userABC/files/some/path/to/a/folder                                   |  3013968 |           978303 |
| home::userX                              |                776 |       611750 |                                                                       | 21394344 |         22371126 |
| home::userY                              |                862 |       771950 |                                                                       | 36683760 |         36683758 |
| home::userZ                              |                874 |       782246 |                                                                       |  4646273 |                0 |
+------------------------------------------+--------------------+--------------+-----------------------------------------------------------------------+----------+------------------+
6 rows in set (4.417 sec)
PVince81 commented 1 year ago

for those where the used storage is wrong and occ files:scan doesn't fix it, how do you know that the used storage is wrong ? I'm guessing that the result of running du on the user's folder is different than what's in oc_filecache ?

are you able to find out which subfolder has a different size in Nextcloud ? I assume no one is using symlinks/hard links or anything (those are not supported by the way)

SudipBatabyal commented 1 year ago

for those where the used storage is wrong and occ files:scan doesn't fix it, how do you know that the used storage is wrong ? I'm guessing that the result of running du on the user's folder is different than what's in oc_filecache ?

are you able to find out which subfolder has a different size in Nextcloud ? I assume no one is using symlinks/hard links or anything (those are not supported by the way)

image

PVince81 commented 1 year ago

@SudipBatabyal can you also check the two hidden entries, in case they were omitted in the calculation ?

focussing on this user, please run the following: occ user:info $userId

and: select s.id,fc.path,fc.size from oc_storages s left join oc_filecache fc on s.numeric_id=fc.storage and s.id like 'home::$userId' and path in ('', 'files');

and replace "$userId" accordingly above.

I assume that this user does not have access to any external storage, I didn't see such icon in the screenshot.

SudipBatabyal commented 1 year ago

@SudipBatabyal can you also check the two hidden entries, in case they were omitted in the calculation ?

focussing on this user, please run the following: occ user:info $userId

and: select s.id,fc.path,fc.size from oc_storages s left join oc_filecache fc on s.numeric_id=fc.storage and s.id like 'home::$userId' and path in ('', 'files');

and replace "$userId" accordingly above.

I assume that this user does not have access to any external storage, I didn't see such icon in the screenshot. # image

SudipBatabyal commented 1 year ago

for those where the used storage is wrong and occ files:scan doesn't fix it, how do you know that the used storage is wrong ? I'm guessing that the result of running du on the user's folder is different than what's in oc_filecache ?

are you able to find out which subfolder has a different size in Nextcloud ? I assume no one is using symlinks/hard links or anything (those are not supported by the way)

2.9 GB folder actual data is 2.7 GB ...........

SudipBatabyal commented 1 year ago

for those where the used storage is wrong and occ files:scan doesn't fix it, how do you know that the used storage is wrong ? I'm guessing that the result of running du on the user's folder is different than what's in oc_filecache ? are you able to find out which subfolder has a different size in Nextcloud ? I assume no one is using symlinks/hard links or anything (those are not supported by the way)

2.9 GB folder actual data is 2.7 GB ...........

@PVince81 This is another user screenshot of the folder where the folder size is 3.2 GB but inside the folder 2.4 GB data is showing.

image

t2d commented 1 year ago

for those where the used storage is wrong and occ files:scan doesn't fix it, how do you know that the used storage is wrong ? I'm guessing that the result of running du on the user's folder is different than what's in oc_filecache ?

are you able to find out which subfolder has a different size in Nextcloud ? I assume no one is using symlinks/hard links or anything (those are not supported by the way)

Yes, I know via du and because I deleted a lot of files in my user but the space is never freed.

simonspa commented 1 year ago

Hi @PVince81

unfortunately I also have to report that while removing the REUSE_SIZE flag does indeed seem to update the odd entries found by your query, this does not change anything about the problem - so it seems to lay somewhere else.

@simonspa to delete those obsolete storage entries from the database, for you specifically:

Hm, I don't feel comfortable doing that, a select fileid, path from oc_filecache where storage=1 gives me more than 800 entries. Are you positive these are all old and unused?

you can check if manually changing the size value also changes the displayed quota for that user

Done that, it actually changes the value properly only for the first entry with empty path, when I try to do the same for the filedid=591, the line still appears in the query afterwards. Also, there is no change in displayed quota.

for those where the used storage is wrong and occ files:scan doesn't fix it, how do you know that the used storage is wrong ?

Yes, du, but in my case it is obvious for many users, like my own one - I have years of data in there, with videos and pictures, but the displayed quota is only 5.2 MB. (du tells me it should be around 85GB)

select s.id,fc.path,fc.size from oc_storages s left join oc_filecache fc on s.numeric_id=fc.storage and s.id like 'home::$userId' and path in ('', 'files');

Somehow this query returns a long list of ids, many not like home::smn - not sure if the left join is fully correct. However, there are two entries for the user in question:

| home::smn                                          | files | 81520013126 |
| home::smn                                          |       | 91045664726 |
jshatch commented 1 year ago

The patch did not work for me either. Here are my discrepancies.

  - quota: 500 GB
  - storage:
    - free: 283573556349
    - used: 82293399
    - total: 283655849748
    - relative: 0.03
    - quota: 536870912000
image
[root@nextcloud jarom]# du -sh files
236G    files

Quota is set to 500GB, the DB reflects that but the UI shows 264.2GB. There are 236GB of files but the DB shows 82MB and the UI shows 78.5MB.

PVince81 commented 1 year ago

The patch did not work for me either

assuming you also ran occ files:scan --all after the patch ? just checking

also you are on 25.0.2 ?

PVince81 commented 1 year ago

the occ command is calling https://github.com/nextcloud/server/blob/v25.0.2/core/Command/User/Info.php#L92 the web UI is calling https://github.com/nextcloud/server/blob/v25.0.2/apps/files/lib/Controller/ViewController.php#L155 which is calling https://github.com/nextcloud/server/blob/v25.0.2/lib/private/legacy/OC_Helper.php#L465

for some reason I see that includeMountPoints is set to true by default, which might make it also count the contents of received shares, but it's not supposed to

simonspa commented 1 year ago

Still, including additional data would not explain the used quota two orders of magnitude smaller than what is on disk, right?

jshatch commented 1 year ago

Yes I ran sudo -u apache php occ files:scan jarom and I am on 25.0.2.

{
installed: true,
maintenance: false,
needsDbUpgrade: false,
version: "25.0.2.3",
versionstring: "25.0.2",
edition: "",
productname: "Nextcloud",
extendedSupport: false
}
PVince81 commented 1 year ago

Still, including additional data would not explain the used quota two orders of magnitude smaller than what is on disk, right?

indeed

I'll need to set aside some time because I have trouble juggling so many different scenarios posted by everyone.

might need to add additional logging in the quota reporting code from https://github.com/nextcloud/server/issues/25283#issuecomment-1348831470 to try and find out where those values are coming from.

apart from stuff that is included additionally, I also don't understand why sometimes it would report less than what's on disk and also be different than the value in oc_filecache's "files" entry.

simonspa commented 1 year ago

Thanks a lot @PVince81 for digging into this! Please let me know if I can support you in any way, be it testing patches or gathering more information from my NC instance!

SudipBatabyal commented 1 year ago

Still, including additional data would not explain the used quota two orders of magnitude smaller than what is on disk, right?

indeed

I'll need to set aside some time because I have trouble juggling so many different scenarios posted by everyone.

might need to add additional logging in the quota reporting code from #25283 (comment) to try and find out where those values are coming from.

apart from stuff that is included additionally, I also don't understand why sometimes it would report less than what's on disk and also be different than the value in oc_filecache's "files" entry.

Definitely, it is a table cache issue not on the file storage label & it is a major bug that comes across with NC nowadays. Install a fresh version of NC latest into our virtual environment but the issue remains the same wrong storage count.

PVince81 commented 1 year ago

Install a fresh version of NC latest into our virtual environment but the issue remains the same wrong storage count.

With a fresh version ? that surprises me, because I was assuming that the quota issue is related to inconsistencies in the database. I don't see how this could happen in a fresh environment with just a few uploaded files.

For this case, one theory is that maybe PHP isn't reporting the free space correctly: on some partition setups (LVM?) it can happen that PHP's disk_free_space() function doesn't report the correct value. You can test it by running this PHP snippet and replace the path with the one from the data directory:

<?php
echo(disk_free_space('/srv/www/htdocs/server/data'));
simonspa commented 1 year ago
<?php
echo(disk_free_space('/srv/www/htdocs/server/data'));

Returns:

370574577664

vs df -B 1

370574426112

So the two methods differ but not by much. This is data being stored on the root partition of an LVM.

t2d commented 1 year ago

In my case, the errors only seems to appear in the web client. The photo upload from the mobile client still works. Is there some web-specific cache that I can clear?

doobry-systemli commented 1 year ago

So I did another round of tests (on Nextcloud 24.0.8). In our case it's pretty obvious that the calculated used quota doesn't get updated after deleting a file. What I did:

  1. Create a new user with 5GB quota
  2. Login as the user and upload two big files (2.4GB and 1.2GB).
  3. Both the absolute folder size and used quota correctly show 3.7GB.
  4. Delete the 1.2GB file and remove it from trash.
  5. The absolute folder size in Files app correctly shows 2.4GB, used quota still shows 3.7/5.0 GB:

nextcloud_wrong_quota

Running occ files:scan <user> didn't change anything. The user doesn't have any shares or external storage.

SudipBatabyal commented 1 year ago

Yes, This is the same issue I also facing & also have a new issue with this: image

simonspa commented 1 year ago

Some even more curious situation, even after going through all the cleaning steps discussed above:

I have one user that has a quota set to 200 GB in the User settings:

image

But checking in the Files app of this user, I get presented with the following:

image

If I increase the quota to 300GB, the listed available space in the files app changes to 227.3 GB, while the 300 GB are reported in the Users list.

Something is really going awry with the quota calculation and size propagation here and it would be great if you @PVince81 could find the time to look into this deeply!

SudipBatabyal commented 1 year ago

image

doobry-systemli commented 1 year ago

Dear Nextcloud devs, is there anything we can do to support you with further debugging the issue? It breaks usage of the Nextcloud instance for lots of users and would be really nice to get fixed.

doc75 commented 1 year ago

Same issue with a user on my instance (24.0.8 docker official image with a PostgreSQL DB):

I can perform test if needed (modify file data, patch code, run DB query, etc...)

doc75 commented 1 year ago

As far as I understand the quota provided to the end user is from the unencrypted_size value from oc_filecache where path='files'.

I noticed that indeed, the oc_filecache table is not updated consistently. the size column seems to be updated when adding or removing a file, but the unencrypted_size column is only updated when adding and not when removing.

Here the DB evolution for my user during the scenario:

request launched:

select storage, size, unencrypted_size, etag from oc_filecache where storage=1 and path='files';

Scenario:

I tried to look at the code, but I am not experienced enough to perform a proper analysis.

simonspa commented 1 year ago

Thanks a lot for the detailed analysis, @doc75 - that could at least explain the cases reported above in which the displayed quota exceeds the actually existing files.

Do you also have an idea how the opposite could happen, i.e. few MB reported quota but tens of GB actual data?

doc75 commented 1 year ago

Do you also have an idea how the opposite could happen, i.e. few MB reported quota but tens of GB actual data?

I have not seen that during my scenario. I am on a standard install with no external storage

simonspa commented 1 year ago

I am on a standard install with no external storage

Same here, this is a standard installation, all data on local storage, data directory residing outside webroot.

come-nc commented 1 year ago

Hello,

@doc75 Do you know how you ended up with unencrypted_size not being 0? Did you have encryption enabled in the past?

On a usual installation without encryption activated unencrypted_size should always be 0. Can you show the value of the encrypted column for these files?

doc75 commented 1 year ago

@doc75 Do you know how you ended up with unencrypted_size not being 0? Did you have encryption enabled in the past?

Encryption is activated in case a user wants to use external storage. The main storage space is not encrypted. Let me some days to provide the information about encrypted size (no easy access for now), but not sure this will be relevant knowing my above statement. But I’ll provide it.

simonspa commented 1 year ago

I just checked, I had encryption enabled a long time ago (NC 11 maybe?) but do not have the encryption module active now. What I see is:

MariaDB [nextcloud]> select storage, size, encrypted, unencrypted_size, etag from oc_filecache where storage=1 and path='files';
+---------+-----------+-----------+------------------+---------------+
| storage | size      | encrypted | unencrypted_size | etag          |
+---------+-----------+-----------+------------------+---------------+
|       1 | 172172644 |         0 |        127431023 | 547351ba45033 |
+---------+-----------+-----------+------------------+---------------+
PVince81 commented 1 year ago

we managed to somewhat reproduce the issue where unencrypted size for "files" is non-zero on git master

it required to have encrypted enabled and overwriting a file multiple times

then when decrypting all files and disabling encryption, the unencrypted size was still non-zero

workaround: 0) make sure encryption is disabled (if you are using encryption do not use this solution, the issue might be another one!) 1) backup database 2) update oc_filecache set unencrypted_size=0

@doc75 in your case you might want to apply the query only the "home::*" storages from oc_storages but not external storages. I was not able to test with a mix of encrypted and non-encrypted storages, the behavior might be different.

@icewind1991 there seems to be many issues related to that column, for example we noticed that a first upload of a file kept it zero, but overwriting set it to a value. Also we never saw "size" being different than "unencrypted_size" so it looks like we're not storing the encrypted size any more ? would be good to do more testing of size propagation

A TODO: when decrypting a user's storage in occ encryption:decrypt-all, at the end of the process, run a one-pass query to set unencrypted_size to 0. @come-nc maybe ?

PVince81 commented 1 year ago

also wondering if we should just add a repair step that resets unencrypted_size if encryption is clearly disabled, this would hopefully cover all cases without need for a workaround

suaudeau commented 1 year ago

update oc_filecache set unencrypted_size=0

It works for me! Wonderfull !!!!

(Nextcloud 25.0.3, a lot of external storage, 10k users)

simonspa commented 1 year ago

Wow, great, thanks a lot for the investigation and the progress @PVince81!

Wouldn't it be better to run

update oc_filecache set unencrypted_size=0 where encrypted=0;

to make sure we only reset this for unencrypted files?

janamedick commented 1 year ago

I just did the update with the additional where-clause:

UPDATE oc_filecache SET unencrypted_size=0 WHERE encrypted = 0

worked perfectly for all of my instances!

prismopensource commented 1 year ago

Exactly the same problem here:

Already tried to do a files:scan without REUSE_SIZE as proposed in this comment. Noticed this install also was initially an install with encryption enabled, after some problems we disabled encryption a long time ago. After a: UPDATE oc_filecache SET unencrypted_size=0 WHERE encrypted = 0 It seems to correct itself as well. Still seems weird there are any files with encrypted greater than 0 on an install without encryption enabled? From a quick look it seems these are all files in files_trashbin/versions, is it safe to remove these DB entries?

PVince81 commented 1 year ago

a big thanks to @doc75 also who thought about looking at unencrypted_size

the query update oc_filecache set unencrypted_size=0 where encrypted=0; is indeed safer

if encryption is disabled and you still see entries with encrypted>0 outside of files, check if the original files for those still exist and if not you can delete the entries from the database and also from the data directory.

simonspa commented 1 year ago

Voila, I can also report success, this was really the culprit. :tada: Thanks @PVince81 @doc75 !

Could this be something that could be performed as part e.g. of occ maintenance:repair automatically?

PVince81 commented 1 year ago

Could this be something that could be performed as part e.g. of occ maintenance:repair automatically?

yes, that could be a good idea, at least until the root cause can be analyzed and fixed

simonspa commented 1 year ago

As @prismopensource writes:

Noticed this install also was initially an install with encryption enabled, after some problems we disabled encryption a long time ago. After a: UPDATE oc_filecache SET unencrypted_size=0 WHERE encrypted = 0 It seems to correct itself as well. Still seems weird there are any files with encrypted great than 0 on an install without encryption enabled?

@PVince81 at least for a fraction of the affected databases at some point encryption was enabled and then turned off, in some cases a decade ago :) Maybe it has to do with the disabling of encryption? At least for me it was very likely the root cause.

alars commented 1 year ago

Maybe it has to do with the disabling of encryption? At least for me it was very likely the root cause.

FWIW encryption was enabled and active when I first had the issue.

PVince81 commented 1 year ago

more tests will be needed to see if "unencrypted_size" is properly updated when encryption is enabled. someone reported earlier that it's possible that some operations like file deletion do not properly propagate the size change

so what needs to be done now:

PVince81 commented 1 year ago

doc PR for the workaround: https://github.com/nextcloud/documentation/pull/9566