check quota against the default remote local folder (instead of /)

[moscicki]

Currently the quota check (https://github.com/cernbox/smashbox/blob/master/protocol/protocol.md#quota-check-call) is done against / on the server.

In the case of branded client which defines the remote home folder to something different than / (for us: /home) -- the quota check should be done against /home folder on the server.

That would only affect branded clients which redefine the default remote sync folder.

[ogoffart]

But that folder is only a default folder, users can change that. The quota-used should be querried on "/" no matter what, I think, because the quota is for the whole account.

[danimo]

@jancborchardt can you chime in here? We designed the quota to be a property of the account. But if we only sync a sub part as outlined by @moscicki, this number will probably deviate from the number shown on the web interface which gives you the full view. This is because the quota is not uniform, but can be different on certain folders.

[moscicki]

Hello,

I agree with you: quota attribute is a property of the account to which the folder belongs and not of the individual subfolders inside. In other words, no matter which folder you query, if it "belongs" to the same account (or quota node) there should be no difference and the result should always be the same (used space vs total space). Currently it seems there is a problem with implementation on the PHP server which does not respect this assumption.

Please refer to this: https://github.com/owncloud/client/issues/4459

Currently owncloud PHP server seems wrong as it gives the total space occupied by that subfolder only (and it was actually abused by the client instead of oc:size as explained in the ticket above). If this is fixed then there is no problem for you with this behaviour also for branded clients which override the remote directory.

The reason for my request is that in our case, we map URIs to different paths on the backend storage namespace (and these folders do NOT belong to one account). The user home directory is mapped as /home (which is actually a shortcut in the context of authenticated user "bob" to /eos/user/b/bob). We transparently expose the full namespace behind so a user can also define a remote sync folder pair against any other path on the backend (if they have access and not necessarily in their home directory). It is not the same as in the case of opaque storage in the PHP owncloud server, where you assume "/" is the top level home directory. In our case it is not, "/home" is. Yes, user may remove that "/home" sync folder pair and redefine any other. So as such this one number to represent quota information does not make sense but needs to be quantified that it applies to "/home" directory. Then we give a clear and useful piece of information to the user about the quota on his personal storage: /home. Maybe you need another branding option internally for that, I do not know.

In any case, even for model followed by the PHP owncloud server, you should also consider that the current approach to the available quota is a bit simplistic. Consider you have 10G on primary storage in the owncloud server and 100G on a secondary storage (say, Amazon S3) which is mounted as a subfolder e.g. /s3. What are you going to show as available storage in the sync client? 10G, 100G or 110G? None of these numbers is correct because two different quotas apply: 100G for /s3 and 10G everywhere else in /.

So again, I think it makes sense to quantify that "10G" applies to "/" directory (excluding remote storages etc).

In both cases there is the same UX design problem: one number does not represent everything.

[guruz]

@PVince81 Your input here? What quota would the server return if we attempted to query a different folder?

[PVince81]

We should probably be more precise when talking about quota. Here I understand it as being the value of "quota-available-bytes".

Here's an example:

1. user1 has a quota of 10 GB and 125MB free space
2. user2 has a quota of 15 GB and 200 MB free space
3. user1 receives a share from user2 as the folder "/shared_by_user2"
4. As user1, PROPFIND on "/" you'll receive quota-available-bytes=125MB
5. As user1, PROPFIND on "/shared_by_user2" you'll receive quota-available-bytes=200MB because that's the free space when you upload into the shared folder, where the data is counted in user2's quota.

@guruz does that answer your question ?

[ckamm]

@moscicki @ogoffart I currently think it would be a first step to change the quota behavior when only a single sync folder is configured. For that case we could display the quota the server reports for the folder, instead of /.

It'd be slightly more correct - of course subfolders could have different quotas.

When there are multiple sync folders we don't know what to do yet and report for / - that's unchanged for now.

[ogoffart]

Imagine you have this structure:

+- someStuf/ (300M)
+- home/  (100M)

someStuff is not shared, and home is the only shared folder. Then the quota will read: "using 100 M out of 700 M" But will user not complain that the quota is supposed to be 1G and we only display 700M ? (because someStuff is not included)

[ckamm]

@ogoffart I don't understand your example, could you rephrase it? Is it a generally open question or a problem with the change I did?

[ogoffart]

It is a problem with the change. What the server gives us, for a folder, is the quota-used by this folder, and the space available. We do quota-used + quota-available to know the total, and we print out of XXX where XXX is the total. but this XXX is really meaningless if it does not include the full quota from root.

[moscicki]

@ogoffart: I don't think what you describe is correct.

To know the space occupied by the folder you should use oc:size property (as we discussed here: https://github.com/owncloud/client/issues/4459)

Consider a simple scenario: just primary storage (one way of accounting for quota, like in a plian local filesystem without any mount points etc.). The quota-used + quota-available should return the same value for any folder you query. These quota-* properties are not a property of a particular folder but the underlying storage space.

Things get hairy when you start having different storage spaces in the same directory tree (your external mounts for example). The concept of "xxx out of XXX" is pretty confusing at this point (and as such maybe not so meaningful) because it does not take into account different storage spaces mounted inside the tree.

Our particular case is different from owncloud php server because we map URIs directly onto underlying, general namespace. So / maps to the real root of the global namespace, where as /home maps to the user home directory (/eos/user/j/john in the underlying namespace for example). We need to find convergence in longer term between these two models but for now it would just make our life easier if we could query "/home" for quota instead of "/".

[moscicki]

@ckamm: if by "/" we mean quota of the home directory ("primary storage") then in our case it makes always sense to query "/home" (as explained above). Would it be a problem for other owncloud users who have their branded clients setting a different default folder?

[ogoffart]

@moscicki We use "oc:size" to get the size of a folder. But to know what part of the quota is used, we use "quota-used-bytes".

The quota-used [...] should return the same value for any folder you query.

It does not. quota-used only returns the value used by that folder. Not the globaly used quota.

The concept of "xxx out of XXX" is pretty confusing at this point (and as such maybe not so meaningful)

Yes, so maybe it should perhaps be a branding setting. Or an extension to @ckamm patch to only do the quota query on one folder if there is only one folder AND the folder's path is the one from the branding's default folder.

[moscicki]

Should not oc:size and quota-used-bytes be the same value if you follow this per-folder logic for quota-used-bytes?

I think "xxx out of XXX" is confusing in general (external owncloud mounts, multiple sync folders) and this has nothing to do with branding options. If you quantify it with "xxx out of XXX in your home directory" or something like that it is then clear.

Anyway, what is important for me is to be able to query for quota properties on "/home" rather than "/".

Thank you.

[ogoffart]

Should not oc:size and quota-used-bytes be the same value if you follow this per-folder logic for quota-used-bytes?

I think they are, altough maybe if some sub directories are in another storage they may not be included.

[ogoffart]

In other words, if we do a PROPFIND not on the root folder, the total value would be wrong if the user has a subfolder configured (in the normal case). That's why I think we should not merge @ckamm's patch as it.

[moscicki]

Okey, then please propose another solution for my problem. Thanks!

On Mon, Apr 18, 2016 at 3:43 PM, Olivier Goffart notifications@github.com wrote:

In other words, if we do a PROPFIND not on the root folder, the total value would be wrong if the user has a subfolder configured (in the normal case). That's why I think we should not merge @ckamm https://github.com/ckamm's patch as it.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/owncloud/client/issues/4460#issuecomment-211385071

---

Best regards, Kuba

[ogoffart]

A suggestion discussed with @dragotin is to have a new branding option on where the quota should be queried

[moscicki]

Okey. Than for maximum flexibility, could I suggest we put this as a server capability (actually I would also prefer to have the remotebhome folder location set in this way). It is much easier to roll out changes to our clients if it is defined centrally.

Thanks!

On Mon, Apr 18, 2016 at 4:24 PM, Olivier Goffart notifications@github.com wrote:

A suggestion discussed with @dragotin https://github.com/dragotin is to have a new branding option on where the quota should be queried

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/owncloud/client/issues/4460#issuecomment-211402573

---

Best regards, Kuba

[dragotin]

@moscicki well, yes, I think in general that makes sense, but please let us go for now with that new branding option. This is not really a server capability but a client setting that is pulled from the server, and we should handle that in a separate and thought through general approach.

[moscicki]

Ok, get your point but I am afraid that we will be stuck with these branding options forever, which is a very inflexible solution for client updates and which I think the sooner changed the better. So, since a "capability call" mechanism is known and understood I am not sure there is a need to invent something else on the server for that (the simpler the better). The very easy thing to do would be to add a section there: "sync_client_settings" or alike. In the future, one could also fold in there a default list of ignored file patterns etc.

To keep things simple to start with, I would just consider to use the values retrieved from the server instead of any hardcoded values in code or branding... That would be a good start. If in a future there is another call to implement that on the server the transition would be easy. And since this would not initially affect anyone except our installation, we are free to change it later on via the usual collaboration.

On Mon, Apr 18, 2016 at 4:58 PM, Klaas Freitag notifications@github.com wrote:

@moscicki https://github.com/moscicki well, yes, I think in general that makes sense, but please let us go for now with that new branding option. This is not really a server capability but a client setting that is pulled from the server, and we should handle that in a separate and thought through general approach.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/owncloud/client/issues/4460#issuecomment-211416249

---

Best regards, Kuba

[ckamm]

@ogoffart Understood! Notice that my patch was already merged, feel free to revert.

Here's my understanding of how this works, is it correct?

root (quota-available: 100, quota-used: 75)
- folder1 (available: 100, used: 25)
- folder2 (available: 100, used: 50)
- share1 (available: 500, used: 100)
-- subfolder1 (available: 500, used: 40)
-- subfolder2 (available: 500, used: 60)

So while quota-available is the same for all folders that belong to the same user/storage, quota-used only gives you the space used for the given folder and its subfolders.

That means my patch improves the quota-available value but leads us to using an incorrect quota-used value in many cases.

See also https://www.ietf.org/rfc/rfc4331.txt

[moscicki]

Notice that this RFC explicitly refers to the behaviour I also describe (user-based quotas):

1. DAV:quota-used-bytes

[...]

Note that there may be a number of distinct but overlapping sets of resources for which a DAV:quota-used-bytes is maintained (e.g., "all files with a given owner", "all files with a given group owner", etc.). The server is at liberty to choose any of those sets but SHOULD do so in a repeatable way. The rule may be configured per repository.

[...]

1. Notes

[...]

Additionally, because there may be a number of distinct but overlapping sets of resources for which a DAV:quota-used-bytes is maintained (Section 4), there may be no correlation between the size of the resources in a collection and DAV:quota-used-bytes. For example, for a server that implements user-based quotas, DAV:quota-used-bytes usually will be the same for a collection and its members.

On Tue, Apr 19, 2016 at 7:13 AM, ckamm notifications@github.com wrote:

@ogoffart https://github.com/ogoffart Understood! Notice that my patch was already merged, feel free to revert.

Here's my understanding of how this works, is it correct?

root (quota-available: 100, quota-used: 75)

• folder1 (available: 100, used: 25)
• folder2 (available: 100, used: 50)
• share1 (available: 500, used: 100) -- subfolder1 (available: 500, used: 40) -- subfolder2 (available: 500, used: 60)

So while quota-available is the same for all folders that belong to the same user/storage, quota-used only gives you the space used for the given folder and its subfolders.

That means my patch improves the quota-available value but leads us to using an incorrect quota-used value in many cases.

See also https://www.ietf.org/rfc/rfc4331.txt

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/owncloud/client/issues/4460#issuecomment-211731274

---

Best regards, Kuba

[ogoffart]

New pull request: https://github.com/owncloud/client/pull/4714

[dragotin]

Handled by branding option now. ReadyToTest.

[mcastroSG]

Hi hi:

Test executed

1. Set /home value to quotaBaseFolder variable
2. Install proper client
3. Create home folder at Shared sync folder
4. Include some files at home folder

Results

Used space is calculated using files includes at home folder and does not take into account other files included at Shared sync folder.

Client Version: Version 2.2.0-nightly20160429 (build 3319) Client OS: OS X El Capitán 10.11.4

Server Version: 9.1.0 Alpha Server OS : Ubuntu