Closed camjac251 closed 4 years ago
sergix44
commented
4 years ago I did some tests with 1-2 gb files with the server (bare metal, not a vm), but the time needed to move the file to storage is about a few seconds (with ssd storage), you should check how much memory you give to php, and the disk read and write speeds, more than a minute seems too much, there must be something wrong.
camjac251
commented
4 years ago Huh that is interesting. Do you use any php caching options? I was hoping to run this on a 2GB Ram VPS later (although my testing has been on 4GB) but it's on an NVME SSD with 1GB/s speeds. Is it uploading the file to ramdisk before offloading to ssd for you?
sergix44
commented
4 years ago Php 7.3 with max memory at 512 MB, opcache enabled, without ramdisk or anything like that. The server is a small cheap Kimsufi from OVH, nothing special.
Huh that is interesting. Do you use any php caching options? I was hoping to run this on a 2GB Ram VPS later (although my testing has been on 4GB) but it's on an NVME SSD with 1GB/s speeds. Is it uploading the file to ramdisk before offloading to ssd for you?
If you are waiting 1 minute after the file is moved, your disk is reading/writing at ~90 MB/s for a 5 GB of file.
camjac251
commented
4 years ago I just made a new instance to test it again. I set the php max memory to 512M this time and made sure opcache was enabled (by default it is for me), opcache memory was left at 128MB. The file upload took time after it was done uploading again. Using opcache-gui, during the transfer it wasn't using any more cache
total memory: 128.00MB
used memory: 11.41MBI tried testing how the file transfer works in PHP. For this I used the browser uploader this time instead of Sharex. I also changed the tmp file upload path for php to /tmp/php and changed the file group to www-data. As it was uploading, I couldn't see any new files being made with ncdu.
The size of everything was the same up until the upload was at the end. Nothing was in /tmp/php. Then after it started to slowly appear, It was writing at about 200MB/s. But then I think after it took that file in the php cache and wrote it to xbackbone storage. I'm wondering where it saves before going to /tmp/php
sergix44
commented
4 years ago I just made a new instance to test it again. I set the php max memory to 512M this time and made sure opcache was enabled (by default it is for me), opcache memory was left at 128MB. The file upload took time after it was done uploading again. Using opcache-gui, during the transfer it wasn't using any more cache
total memory: 128.00MB used memory: 11.41MB
Remember that the opcache (operational code cache) is only use to cache php instructions for faster execution, not to cache files or data.
camjac251
commented
4 years ago I wonder how your install handles a 2GB file uploading through the script. I tested the disk writes on this linode and 400MB/500MB R/W so it should be fast. Is there a chance to have chunked uploads or something like html5 uploading?
sergix44
commented
4 years ago The server has 2 ssd in soft raid 1, but I don't think it could make such a huge difference.
The chucked uploads in this case can only make it worse, since once the last chunk is received it should read all the individual fragments and write the file to the storage.
In addition, ShareX is not using the chunked upload, so it will not work.
Hi there
I really like the look of this file upload manager. I've been looking to find something to replace uPste ever since it stopped development and became a security risk and this has many more features.
From my experience with PHP, large files take a while to go from the php temporary path to the destination. I'm testing XBackBone on a Linode Dedicated 4GB: 2 CPU, 80GB Storage, 4GB RAM node with Ubuntu 18.04 and php 7.2/nginx/sqlite. With ShareX uploading a 5GB file, the initial upload is really fast, almost saturates the gigabit speeds. After the file is done uploading, it took a minute before the link was made and sound went off in ShareX though.
Do you know if there's a way to optimize PHP for large file uploads? If there could be a way to have it upload to storage directly instead of to php temp cache, but it looks like it can't with PHP. I was thinking of moving away from PHP for finding a replacement but your app really looks the best :)