search

stonith404 / pingvin-share

A self-hosted file sharing platform that combines lightness and beauty, perfect for seamless and efficient file sharing.
https://stonith404.github.io/pingvin-share/
BSD 2-Clause "Simplified" License
2.83k stars 206 forks source link

🚀 Feature: Performance download and upload large files #186

Open webysther opened 1 year ago

webysther commented 1 year ago

🔖 Feature description

I got severe limitation in bandwith when upload and download files, the zip creation in background is great but when download the file works like in stream which drop (in my case) from 1Gbe to 500Mbps my bandwith and in upload drops from 1Gbe to 250Mbps.

🎤 Pitch

I used the file station on synology NAS with same nginx and got max. This is clear when talk about files bigger than 10GB, I share photos with friends and the bit problem I have is the need to create the zip every time, this is fixed on pingvin but I got limited on bandwith.

I think send the Content-lenght will improve the browser handle and using buffer (as .env VAR?)

Increase of highWaterMark from default (64kb) to 1mb maybe solve the problem, this maybe use from a .env too.

pierrbt commented 1 year ago

@webysther Are you sure the limitations don't come from your server itself / hard drive ? I'll check myself and I'll tell you 😉

webysther commented 1 year ago

@webysther Are you sure the limitations don't come from your server itself / hard drive ? I'll check myself and I'll tell you 😉

Pretty sure '...file station on synology NAS with same nginx and got max...', I think is a optimization scenario, even with google drive I got max bandwith with much less resource usage.

Look, if a try a 16GB files the project need to create 250k chunks, with 1mb chunk is talk about 15k chunks, which is more CPUxRAM balance.

webysther commented 1 year ago

With 64kb chunk every piece need about 128k of ram and 1mb chunk is about 2mb of ram, this a low price in cost to get really great performance boost, take a look.