search

immich-app / immich

High performance self-hosted photo and video management solution.
https://immich.app
GNU Affero General Public License v3.0
45.08k stars 2.18k forks source link

[Feature]: Upload large files in chunks #567

Closed Eidenz closed 1 year ago

Eidenz commented 2 years ago

Feature detail

Currently, uploading is done in a very simple way: upload the whole file directly to the server. However, some users may have limits on their reverse proxy preventing uploading such files.

Uploading in chunks (like LibrePhotos does for exemple) would allow bypassing this problem.

With immich not being able to scan local folders, I think this feature might be needed.

Platform

Server

bo0tzz commented 2 years ago

It probably can't hurt to add the chunking, but I think issues like this should generally be solved on the proxy side. As for watching a folder, we're planning to add an option for that to the cli.

zackpollard commented 1 year ago

I think this has extra benefits besides just the reverse proxy side of things. If we have this, then uploading large files on slow or unstable connections would mean they could be resumed later rather than the entire upload failing. Temporary files could be kept around for a configurable amount of time and if the file continues to be uploaded, negotiation between client and server could be handled so that the client resumes from where it was before.

fyfrey commented 1 year ago

For reliable upload we could implement the tus protocol https://tus.io/

JVT038 commented 1 year ago

Additionally, chunked uploading would solve an issue for Cloudflare users. Cloudflare only allows a max upload size of 100MB. Using chunked uploading, large files can still be uploaded and solve the Cloudflare issue. Currently, when uploading a large file, it shows this error in the logcat: flutter : ERROR backupAsset: SocketException: Broken pipe (OS Error: Broken pipe, errno = 32), address = immich.mydomain.com, port = 49742 and the file does not get uploaded.

IrwenXYZ commented 1 year ago

would like to see this feature too. cloudflare is blocking many of my uploads right now