search

owncloud / ocis

:atom_symbol: ownCloud Infinite Scale Stack
https://doc.owncloud.com/ocis/next/
Apache License 2.0
1.42k stars 183 forks source link

[Question] How to edit/adjust config of archiver? #9709

Open the-hotmann opened 4 months ago

the-hotmann commented 4 months ago

Quick storytime:

I used oCIS since version 3 and waited untill it really is "usable" for productive work. Anyway since I also take photos in my freetime I still use the good old "OwnCloud" for hosting pictures, as oCIS currently just isn't good enough:

But since oCIS v5 things started to iron-out and I thing as a file-cloud it does a pretty good job - still very bad for pictures, but I even introduced it at the company I work to replace the old Cloud.

Anyway the main topic of this "issue" is, that someone thought it is a clever idea to limit archives in general, and I fundamentaly disagree with this: https://github.com/owncloud/ocis/issues/2537

Some limit should be there, but 1.1GB and 1000files .. really? Even my super weak Intel NUC can 100% create bigger archives! Also I would not mind to wait, but simply refuse to do the job is a big NoGo for me. I think some things went wrong here!

Anyway I understand that there needs to be some type of limitation, but 1.1GB really is absolutely normal, even for personal use and should have NEVER been a limit in the first place. I found the limitation here:

  1. https://doc.owncloud.com/ocis/next/deployment/container/orchestration/tab-pages/values-tab-2.html#:~:text=archiver%20can%20create.-,maxSize%3A%201073741824,-%23%20%2D%2D%20Max%20number%20of
  2. https://doc.owncloud.com/ocis/next/deployment/container/orchestration/tab-pages/val-desc-tab-1.html#:~:text=features.archiver.maxSize

But yet I do not know how to adjust this in my docker-compose.yml file and why this ever made it in any release? I can not even thing of any machine, that would be so slow, that it could not create an archive bigger than 1.1GB...

That aside I would be happy to know, what to do, to remove or edit this (in my opinion stupid) limit so a reasonable setting.

Thanks for this great app! :)

micbar commented 4 months ago

We have the honorable duty to offer SaaS solutions for up to 3 million users.

So please do not blame us for some protective default values.

You can change them very easily to accommodate your needs.

the-hotmann commented 4 months ago

So please do not blame us for some protective default values.

One things does not have anything to do with the other.

You can change them very easily to accommodate your needs.

Thanks for basically nothing.. this repeats my question:

But yet I do not know how to adjust this in my docker-compose.yml file

but does not answer it. Just like the docs do not: https://owncloud.dev/ocis/config/#default-config-values-in-yaml

micbar commented 4 months ago

Re opening, not answered.

micbar commented 4 months ago

I am on mobile, I can look it up later.

seems that you have not found the docs yet 🤣

doc.owncloud.com is the official docs.

There is the deployment section where every possible config value is listed and documented.

I need to look it up for myself.

the-hotmann commented 4 months ago

Ok I have found it: https://doc.owncloud.com/ocis/next/deployment/services/s-list/frontend.html#archiver

Name IV Type Default Value Description
FRONTEND_ARCHIVER_MAX_NUM_FILES pre5.0 int64 10000 Max number of files that can be packed into an archive.
FRONTEND_ARCHIVER_MAX_SIZE pre5.0 int64 1073741824 Max size in bytes of the zip archive the archiver can create.

These are the env-vars that are needed to adjust the archiver. Since I now tested it, I have more questions now, than before and understand the limitation of the 1.1GB default. The implementation is so bad, that this limit is ment to conseal the problem.

My .ARW files are about 80MB and when I downloaded three of them oCIS consumed a lot of CPU, but after the download I inspected the downloaded ZIP and saw, that no compression was applied.

Here some pics of the claims:

image

image

bdac606d7704   owncloud        24.45%    181.8MiB / 4GiB     4.44%     334MB / 651MB     0B / 238kB       23

Also: 10min after the download, the CPU is still "high" with about 20%, as if it still would do something:

bdac606d7704   owncloud        19.27%    172.8MiB / 4GiB     4.22%     2.55GB / 4.97GB   0B / 2.62MB      23

Notice: this is a demo installation, with NOTHING, but this demo-download folder and no other users.

The isue goes away, after a restart of the container:

bdac606d7704   owncloud        0.23%     73.23MiB / 4GiB     1.79%     30.6kB / 17.3kB   0B / 164kB       25
prohtex commented 2 months ago

Ok I have found it: https://doc.owncloud.com/ocis/next/deployment/services/s-list/frontend.html#archiver

Name IV Type Default Value Description FRONTEND_ARCHIVER_MAX_NUM_FILES pre5.0 int64 10000 Max number of files that can be packed into an archive. FRONTEND_ARCHIVER_MAX_SIZE pre5.0 int64 1073741824 Max size in bytes of the zip archive the archiver can create. These are the env-vars that are needed to adjust the archiver. Since I now tested it, I have more questions now, than before and understand the limitation of the 1.1GB default. The implementation is so bad, that this limit is ment to conseal the problem.

My .ARW files are about 80MB and when I downloaded three of them oCIS consumed a lot of CPU, but after the download I inspected the downloaded ZIP and saw, that no compression was applied.

Here some pics of the claims:

image

image

bdac606d7704   owncloud        24.45%    181.8MiB / 4GiB     4.44%     334MB / 651MB     0B / 238kB       23

Also: 10min after the download, the CPU is still "high" with about 20%, as if it still would do something:

bdac606d7704   owncloud        19.27%    172.8MiB / 4GiB     4.22%     2.55GB / 4.97GB   0B / 2.62MB      23

Notice: this is a demo installation, with NOTHING, but this demo-download folder and no other users.

The isue goes away, after a restart of the container:

bdac606d7704   owncloud        0.23%     73.23MiB / 4GiB     1.79%     30.6kB / 17.3kB   0B / 164kB       25

Hi @micbar While I don't agree with the anger and entitlement expressed by @the-hotmann, I do agree that the state of Web based downloads in OCIS is lacking. As @the-hotmann points out, the expectation (that I shared) is that the limitation is a server config that can be easily adjusted to allow for larger downloads.

Let's take the example of PHP. On a fresh LAMP deployment one might adjust the php.ini configuration to allow for large downloads. UPLOAD_MAX_FILESIZE, MAX_INPUT_TIME, MEMORY_LIMIT, MAX_EXECUTION_TIME, POST_MAX_SIZE etc are all set to safe values by default, but they can be adjusted to allow an admin to permit the user to execute scripts to download large files. This is how OC10 worked.

This is not the case with OCIS. Adjusting the values above does not allow downloading large files, it simply reveals that the mechanism OCIS Web uses to create archives is resource intensive and inadequate for large files. I am hoping this can be rewritten in future versions of OCIS Web because I share the use case with @the-hotmann - we run file shares for photographers and videographers, and these files are all very large.

@the-hotmann I understand this is not ideal, but we've had luck using OCIS desktop and mobile clients to get around this issue.

As always, it is such an exciting thing to see OCIS being developed so actively, and I am extraordinarily grateful to all the individuals who have been contributing code and working towards its betterment.

Thank you!

micbar commented 2 months ago

This is not the case with OCIS. Adjusting the values above does not allow downloading large files, it simply reveals that the mechanism OCIS Web uses to create archives is resource intensive and inadequate for large files. I am hoping this can be rewritten in future versions of OCIS Web because I share the use case with @the-hotmann - we run file shares for photographers and videographers, and these files are all very large.

Thanks for the objective approach.

I must correct some statements here to give more understanding.

1) Compression

I think the docs are wrong, there has been no requirement to compress files so far. Archive downloads are just "convenience" to have one file instead of many. Sorry for the misleading information.

2) Where is the archive created

You said its the web client, that is wrong. Ocis is a microservice architecture. The archive is created on the archiver service. The archiver service has no access to the files on disk (separation of concern). So it needs to download the file from the storage-users service, hold it in a buffer and create the archive on the fly.

There are possible improvements listed in https://github.com/owncloud/web/issues/10501

3) Native Clients

All our native clients can download and upload very large files. We are using the very robust TUS protocol (tus.io) so that has always been the preferred way to handle large files.

prohtex commented 2 months ago

This is not the case with OCIS. Adjusting the values above does not allow downloading large files, it simply reveals that the mechanism OCIS Web uses to create archives is resource intensive and inadequate for large files. I am hoping this can be rewritten in future versions of OCIS Web because I share the use case with @the-hotmann - we run file shares for photographers and videographers, and these files are all very large.

Thanks for the objective approach.

I must correct some statements here to give more understanding.

  1. Compression

I think the docs are wrong, there has been no requirement to compress files so far. Archive downloads are just "convenience" to have one file instead of many. Sorry for the misleading information.

  1. Where is the archive created

You said its the web client, that is wrong. Ocis is a microservice architecture. The archive is created on the archiver service. The archiver service has no access to the files on disk (separation of concern). So it needs to download the file from the storage-users service, hold it in a buffer and create the archive on the fly.

There are possible improvements listed in owncloud/web#10501

  1. Native Clients

All our native clients can download and upload very large files. We are using the very robust TUS protocol (tus.io) so that has always been the preferred way to handle large files.

Hi @micbar thanks so much for the thoughtful reply and I am excited to see brainstorming for a possible solution. Sounds like the archiver service needs to allocate disk buffer rather than RAM.

I understand archives are not being made client-side but I'm still curious about this process-I'm doing my tests in Mac Safari and it is interesting to see a download workflow that bypasses the typical browser behavior. What happens when I download a large archive is that the AJAX loader plays while the browser keeps eating RAM. Eventually the file is just "downloaded" without the native DL UI.

In any case, here's to hoping for a good solution. Thanks everyone! Viva OCIS!