search

Vinelab / cdn

CDN Assets Manager Package for Laravel.
MIT License
214 stars 71 forks source link

Long time to upload for lots of files #84

Closed dragosperca closed 8 years ago

dragosperca commented 8 years ago

this is not an issue per-se, but more of a request.

My use case scenario is push "/public/assets" to S3/Cloudfront. When this folder contains thousands of files (it can happen with lots of javascript packages, images, etc) the upload can take anywhere between 5 to 10 minutes. During this time, the app should be in maintenance mode and it's a killer to keep it 10 mins like this.

What if, before pushing files to CDN, you'd check if the file signature (MD5 hash) has changed since last push. You'd store a file in /storage/app that will contain all signatures previously computed. If a file isn't there, then we need to upload it. If a signature is different for a file, then we need to upload it. If the signature is found and is identical, skip the file.

With this approach, uploading 3000 files the first time would still take 5 minutes, but the second time it's going to find only 1 or 2 files changed, so the upload will finish in less than a second.

Obviously, I'm going to fork your package and try it on my own, but I'm interested in your thoughts and if you're open for a pull request on this.

dragosperca commented 8 years ago

err, i've seen a similar issue for this https://github.com/Vinelab/cdn/issues/26 sorry about it

Mulkave commented 8 years ago

@dragosperca no worries.

Mulkave commented 8 years ago

@dragosperca Just a poke to tell you that this has been worked out at #90, you can try it out if you wish.