Admin option to disable creation of ZIP archives

h-2 commented 2 years ago

How to use GitHub

Please use the 👍 reaction to show that you are interested into the same feature.
Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
Subscribe to receive notifications on status change and new comments.

Is your feature request related to a problem? Please describe.

If a user of my instance requests the download of a folder or multiple files, the server will attempt to create a ZIP file. If the folder or summed up file size is large, creating the ZIP file can create a significant computational burden on the server, in the worst case making the instance unusable for everyone. Since there is no visual indication of what is happening, the user doesn't even know that the server is actively zipping and will refresh and/or try to download again, which will create even more strain on the server. (The latter is a different problem, but it certainly makes this one worse).

Describe the solution you'd like

Add a check box for the admin:

[ ] Disable creation of ZIP files from file selections and folders that are larger than ___ MB (0 prevents any ZIP files from being created).

Whenever a user clicks Download on a folder for which ZIP file creation is disabled, they should be shown a dialogue that suggests using a WebDAV client instead. This can conveniently link to: https://docs.nextcloud.com/server/latest/user_manual/en/files/access_webdav.html

Whenever a user clicks Download for a selection of files whose combined size is larger than the threshold, multiple individual downloads should be triggered; each after the last one is complete.

Text in italics was added later

Bounty

I am willing to pay a 50€ bounty to whoever develops the proposed solution and gets it merged until 2024-07-19. Payable via SEPA/Paypal/Paydirekt.

Describe alternatives you've considered

none

Additional context

Here is a help thread on the matter: https://help.nextcloud.com/t/disable-zipping/59279/26

Here is an outdated older issue (it could not be re-opened although there is activity on it) : https://github.com/nextcloud/server/issues/16971

rebirth90 commented 1 year ago

I would like to have this option because i've downloaded ~7 gb of data but i don't have space to actually un-compress it.

sfonteneau commented 1 year ago

My problem is that if the download is canceled (on a 20gb folder) the stream will continue.

On a light processor it is very consumer

A temporary solution I did is to block "addDirRecursive" function in Streamer.php. Waiting for a real patch

For different reasons, having the possibility to deactivate would be interesting

IPFR33LY commented 1 year ago

I'd love the ability to do this. When transferring large datasets between outside collaborators and us, it can be very time consuming having to decompress an already compressed binary dataset. We're talking 100GB+. A flag or toggle switch of some kind would be glorious. Please seriously look at this issue. Thank you.

digitalrevisor commented 1 year ago

We have issues with signatures in .pdf files. When they get zipped by Nextcloud some files are missing signatures. If we download them one by one the files are OK. It's a lot of work to explain to users that if they need to download several pdf files they have to do it one by one. And if we zip the files in windows, there are no issues.

joshtrichards commented 1 year ago

Perhaps you'd all find this helpful:

https://github.com/nextcloud/files_zip

Create zip archives from one or multiple files from within Nextcloud. The archive will be created in the background during cron job execution, so make sure that you have setup a regular cron job. Once the file has been created, the user will be notified about that.

It even supports configuring the maximum size of zip files.

fred0r commented 1 year ago

srsly - its still not possible to disable zip-creation of (already compressed) mp4-family movies to download in batch mode?

h-2 commented 1 year ago

Perhaps you'd all find this helpful:

But this doesn't prevent the normal way ZIP-files are created, correct?

srsly - its still not possible to disable zip-creation of (already compressed) mp4-family movies to download in batch mode?

To be honest, I am also a bit disappointed that this is in "Needs Triage" since almost a year although it has gotten quite a few thumbs-up. I guess this happens when Open Source projects become "too big" 😢

joshtrichards commented 1 year ago

@h-2 wrote:

But this doesn't prevent the normal way ZIP-files are created, correct?

If you mean it leaves the Download button in-place, correct.

srsly - its still not possible to disable zip-creation of (already compressed) mp4-family movies to download in batch mode?

To be honest, I am also a bit disappointed that this is in "Needs Triage" since almost a year although it has gotten quite a few thumbs-up. I guess this happens when Open Source projects become "too big" cry

Nah, it comes up when 20+ people give it a thumbs up, but not one bothers to attempt a PR to explore implementation approaches. :-)

This is a community project. Anyone is welcome to try their hand at using fileMultiSelectMenu.toggleItemVisibility() to turn off the download item if the files_zip app (or item) is active. Then to extend that to, perhaps, an app setting in files_zip that makes it configurable behavior via getAppValue(). If you can't do that then you be patient and cross your fingers - or you learn how - or you pay someone to do it for you. Certainly no worse than non-open-source software, and even better in terms of being empowered to do something about it - if you so choose.

Or if someone just wants to get back to the core request of this particular Issue's enhancement request - it'd probably be even easier to add a config value to disable the Download action outright (with or without the files_zip app). It's not hard to imagine that could then be extended to supporting "disabling if over X MB/GB in size" etc.

Of course to get this accepted someone needs to think though any blow back that arises from making each of these seemingly tiny adjustments. e.g. Where might the Download action be a necessity? Where might fallback behavior be necessary? etc. etc.

Implementation hints

Toggling items:

https://github.com/nextcloud/server/blob/aa5dcb095256d6352de758361798de86c3e2d0d0/apps/files/js/filelist.js#L3490-L3501

https://github.com/nextcloud/files_zip/blob/8e97984803ba8b38d4046157b84ea1afa42a8011/src/main.js#L67-L73

Configuration:

https://github.com/nextcloud/files_zip/blob/8e97984803ba8b38d4046157b84ea1afa42a8011/lib/Service/ZipService.php#L162

https://docs.nextcloud.com/server/latest/developer_manual/basics/storage/configuration.html?highlight=getappvalue#app-values

h-2 commented 1 year ago

Certainly no worse than non-open-source software, and even better in terms of being empowered to do something about it - if you so choose.

Yeah, I am not bashing open source. Closed source could certainly be worse. I was saying that it is hard to scale responsiveness, user interaction etc that are often really good for small open source projects that only receive one issue per week or per month.

This is a community project. Anyone is welcome to try their hand at using

I think "Needs triage" is entirely different from "We have seen this issue, think the proposed solution is viable, but don't have resources to implement it. Please open a pull-request. The following links might be helpful: …"

If someone had said the latter, maybe there would have been a PR. But you can't expect someone who doesn't know anything about the project structure, the workflows, or even the involved programming languages to start reading up on all that and implement a solution without even knowing whether the project wants one?

Anyway. This is getting Offtopic. I am willing to put a 50€ bounty on the solution I proposed in the first post. I know that accounting for all developer time, it will likely cost more, but that's what it's worth to me.

joshtrichards commented 1 year ago

I think "Needs triage" is entirely different from "We have seen this issue, think the proposed solution is viable, but don't have resources to implement it. Please open a pull-request. The following links might be helpful: …"

Yes, but "we" is "we". And you're putting a lot of importance on an arbitrary tag on an open issue.

If someone had said the latter, maybe there would have been a PR. But you can't expect someone who doesn't know anything about the project structure, the workflows, or even the involved programming languages to start reading up on all that and implement a solution without even knowing whether the project wants one?

People work issues in all sorts of states all the time. People submit PRs all the time without discussion ahead of time. Some get in and some don't. Some quickly... some years later. People experiment with different implementation approaches. Debate things openly. etc. That's how it works.

In any case, I just did what you asked by providing some hints. Let's see what happens. :-)

P.S. There are also loads of other ways to help the project which free up developer time. For example, helping triage Issues... :-)

P.P.S. Don't get me wrong - I understand and empathize with you: it can be frustrating when encountering an annoying bug or missing feature gap in a piece of software used every day.

bretthl commented 11 months ago

Compression does not reduce the size of image music and video files. I would also like to be able to disable md5 or whatever "check sum" NC is using. I really don't need that level of security, it just slows things down and I have seen multiple instances of check sum failure on large file.

nextcloud / server