Errors with big files (over 2GB)

[ObfuscateTheWeb]

I've done nc-format-USB nc-datadir and nc-automount via the web interface.

I have an external 1TB usb storage and I test it by uploading big files to it.

First upload was 25.9 GB then it won't allow me to upload anymore.

It will give me a Request Entity Too Large error.

I then try setting theMAXFILESIZE in nc-limits to 50G

Reboot and still doesn't work.

At this point, I'm pretty sure I didn't set it up correctly because my full 1TB of storage isn't being used. How do I set it up correctly? I did not modify anything in fstab.

EDIT: Don't want to open too many issues, so I'll add this here too. I noticed that videos don't play in the web interface, I could hear audio but video is black. I'm pretty sure it's a bug in the code somewhere, how do I manually fix it? Which line of code to modify and what code to add?

[nachoparker]

There is a limit of 2G filesize, this is because Raspbian is 32 bit and PHP sucks. This has been worked around in Nextcloud 13, and the limitation will no longer exist.

It doesn't matter what you setup, if it's bigger than 2GB it will fail.

You can check df -h to see disk usage

You can check from the terminal that your files are indeed in the external drive ( maybe /media/USBdrive/ncdata/admin/files )

wrt the video issue... must be related to upstream nextcloud or your browser / OS. You will have to look that up

[ObfuscateTheWeb]

Turns out, modsecurity was causing the issue ! Disabled it and it now works.

Only 2G? That doesn't sound right @nachoparker because my first upload was a full 25.9 GB (the entire breaking bad series) The second upload was a windows iso file that was 3.27GB and it uploaded just fine..?

Just checked with df -h indeed, my external drive at /dev/sda1 mounted at /dev/USBdrive is at 26GB usage out of 1TB..

I can't find nextcloud 13 by searching, not even a new branch or something in github to indicate the existence of nextcloud 13, can you link me to it so I could read on the changes, features, etc?

[ObfuscateTheWeb]

Okay.. I was wrong, my windows iso (3.27 GB) didn't upload. And thinking about it, I think the only reason I was able to upload 25.9 GB of breaking bad was because each episode was less than 2GB, the total size doesn't matter, what matters is the size per file.

In the case of the windows iso, it was a single file that was more than 2GBs, that's why it failed.

Am I right on this?

[nachoparker]

100%

8^)

[ObfuscateTheWeb]

Just wondering, when will the upload size/file size limit be fixed? Like, a new PHP version that addresses this issue?

This seems to be a big problem in our digital world, I mean everything is data, and being able to handle files and folders of any size, no matter how large, should be common and doesn't need to be a hassle..

Why hasn't this been fixed? PHP, apache developers should be able to address this, but as of now, nothing has come of it...

[nachoparker]

Yeah, it's pretty damn ridiculous. The PHP developers decided they won't fix it, which is incredible. It only affects 32 bits.

Like I said, there is a workaround for it, but we will have to wait until NC 13, which is under development.

If you don't want to wait and you can code, you can patch it yourself

https://github.com/nextcloud/server/issues/1707#issuecomment-291835214

[nachoparker]

For the time being, you can also upload those files via samba/nfs/ssh

https://ownyourbits.com/2017/04/18/different-ways-to-access-your-nextcloud-files/

[ObfuscateTheWeb]

well, pi has 64 bit processor but 32-bit raspbian, waste of hardward..lol

[ObfuscateTheWeb]

Well, read through the post, well it ain't easy because it needs "deep refactoring of PHP"

The devs got their reasons I guess, can't blame em..

[nachoparker]

well, pi has 64 bit processor but 32-bit raspbian, waste of hardward..lol

Yeah, there are a couple reasons. One being compatibility with RPi2, another being they don't have the resources. HW is not much wasted because it doesn't have a lot of RAM anyways. Things should just work, this is mainly a SW problem.

The devs got their reasons I guess, can't blame em..

If you aim at having a 'web language' that can't cope with stuff >2GB... that's pretty bad. PHP receives a lot of criticism and these things don't help

[ObfuscateTheWeb]

Hmm, can't you just install a 64-bit linux on the pi and then get some 64-bit nextcloudpi and the problem of file sizes would be solved?

EDIT: Might be an interesting read

Until nextcloud 13 comes out, I'll just have to break down my files into smaller chunks and then put them together to make one complete file, haven't tested it, but theoretically it should work

[nachoparker]

thanks for the link. I read it. It won't happen any soon, it seems. Literally, it seems like the Raspbian devs are a 2 men army, so they can't afford supporting both versions.

Probably easier to wait for NC13 or manually change the files and apply the patch for the workaround.

I chose Raspbian because it is debian based and it's popular. If I have a 64 bit OS then I lose support for RPi2, which is 32 bit. If we go for the Rock64 though, we might use some Debian 64 bit OS in the future.

I still recommend that you use samba/nfs/ssh for the big files. NCP is ready to use them

[ObfuscateTheWeb]

Then ditch everyone who is using a pi 2 ! Lol..

force all users to go out and get a pi 3 model b

[ghost]

That lines up with my experience. I have 1 2.2 gig file that I had to convert in order to make it just under the limit and then it loaded fine. Because of the conversion, I decided to do the same with everything else of the filetype for consistency. What was once a total of of a few hundred gigs now totals under 50. It was something like avi, mp4, mov and maybe even 3gp, but the end is mkv. Handbrake did all of the conversions.

-- CONFIDENTIALITY NOTE: Some E-Mail communication may occur using a secure method. The information transmitted, including attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material. Any review, re-transmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender via E-Mail, only, and destroy any copies of this information. IMPORTANT: This E-Mail is intended for the use of the individual addressee(s) named above and may contain information that is confidential, privileged or unsuitable for overly sensitive persons with low self-esteem, no sense of humor or irrational religious and/or political beliefs. Any dissemination, distribution or copying of this E-Mail is not authorized (either explicitly or implicitly) and constitutes and irritating social faux-pas. Also, any E-Mail sent from the originating author claims ownership of all E-Mail sent and received on local computer(s) in addition to all non-solicited E-Mails received. CAUTION: Unless the word 'absquatulation' has been used in its correct context somewhere other than in this warning, it does not have any legal nor grammatical use and may be ignored. No animals were harmed in the transmission of this E-Mail, although the kelpie next door is living on borrowed time, let me tell you. Those of you with an overwhelming fear of the unknown will be gratified to learn that there is no hidden message revealed by reading this warning backwards, so just ignore that alert notice from Microsoft. ANTIDOTE: However, by pouring a complete circle of salt around yourself and your computer you can ensure that no harm befalls you and your pets. If you have received this E-Mail in error, please add some nutmeg and egg whites, whisk and place in a warm oven for 40 minutes.

Sent with ProtonMail Secure Email.

-------- Original Message -------- Subject: Re: [nextcloud/nextcloudpi] How do I know if external storage works? (#101) Local Time: August 11, 2017 11:10 PM UTC Time: August 12, 2017 3:10 AM From: notifications@github.com To: nextcloud/nextcloudpi nextcloudpi@noreply.github.com Shadowstreik admin@ragnaranchorage.com, Manual manual@noreply.github.com

Okay.. I was wrong, my windows iso (3.27 GB) didn't upload. And thinking about it, I think the only reason I was able to upload 25.9 GB of breaking bad was because each episode was less than 2GB, the total size doesn't matter, what matters is the size per file.

In the case of the windows iso, it was a single file that was more than 2GBs, that's why it failed.

Am I right on this?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[ObfuscateTheWeb]

What was once a total of of a few hundred gigs now totals under 50.

That's interesting @Shadowstreik how did you manage to compress that much data? A few hundred gigs down to under 50?! That's some crazy compression..

Mind sharing your method of compression?

Handbrake did all of the conversions. What's the point of filetype consistency? Does it somehow reduce file size and furthers compression?

[ghost]

They are only totals. Each one, individually, is much smaller.

Using Handbrake is my method. It does all the work. Let's say something is a gig in size and an mov. Handbrake going from mov to mkv would, lets say, make it a .8 gig file.

The point of consistency... organization and cleanliness. Nothing but, really.

-- CONFIDENTIALITY NOTE: Some E-Mail communication may occur using a secure method. The information transmitted, including attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material. Any review, re-transmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender via E-Mail, only, and destroy any copies of this information. IMPORTANT: This E-Mail is intended for the use of the individual addressee(s) named above and may contain information that is confidential, privileged or unsuitable for overly sensitive persons with low self-esteem, no sense of humor or irrational religious and/or political beliefs. Any dissemination, distribution or copying of this E-Mail is not authorized (either explicitly or implicitly) and constitutes and irritating social faux-pas. Also, any E-Mail sent from the originating author claims ownership of all E-Mail sent and received on local computer(s) in addition to all non-solicited E-Mails received. CAUTION: Unless the word 'absquatulation' has been used in its correct context somewhere other than in this warning, it does not have any legal nor grammatical use and may be ignored. No animals were harmed in the transmission of this E-Mail, although the kelpie next door is living on borrowed time, let me tell you. Those of you with an overwhelming fear of the unknown will be gratified to learn that there is no hidden message revealed by reading this warning backwards, so just ignore that alert notice from Microsoft. ANTIDOTE: However, by pouring a complete circle of salt around yourself and your computer you can ensure that no harm befalls you and your pets. If you have received this E-Mail in error, please add some nutmeg and egg whites, whisk and place in a warm oven for 40 minutes.

Sent with ProtonMail Secure Email.

-------- Original Message -------- Subject: Re: [nextcloud/nextcloudpi] Errors with big files (over 2GB) (#101) Local Time: August 12, 2017 1:33 PM UTC Time: August 12, 2017 5:33 PM From: notifications@github.com To: nextcloud/nextcloudpi nextcloudpi@noreply.github.com Shadowstreik admin@ragnaranchorage.com, Mention mention@noreply.github.com

What was once a total of of a few hundred gigs now totals under 50.

That's interesting @Shadowstreik how did you manage to compress that much data? A few hundred gigs down to under 50?! That's some crazy compression..

Mind sharing your method of compression?

Handbrake did all of the conversions. What's the point of filetype consistency? Does it somehow reduce file size and furthers compression?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[nachoparker]

Then ditch everyone who is using a pi 2 ! Lol.. force all users to go out and get a pi 3 model b

Haha, not my decision though, but Raspbian developers

[delfer]

Fixed by docker-library/php#901 Tested in Docker image nextcloud:17-apache (nextcloud@sha256:58836d449406b396c1c119c2661577fd7d8284ff41a7d181a0e2be232d2757fc, 2019-10-24T05:04:49.946856094Z)