Invalidate deleted files

[saneef]

Is there a way I can invalidate files which are been deleted by s3cmd sync?

[mdomsch]

sync --cf-invalidate invalidates the newly uploaded files. It doesn't presently invalidate deleted files. Doing so would seem to make sense though. It would require a change to _invalidate_on_cf() to take a list of objects rather than relying on the cmd_sync_local2remote() function scope uploaded_objects_list list, and to get the remote_list list out of _child() and into function scope as well. Seems pretty straightforward to do though, if you wanted to try your hand at a patch.

On Thu, Mar 27, 2014 at 2:20 AM, Saneef Ansari notifications@github.comwrote:

Is there a way I can invalidate files which are been deleted by s3cmd sync ?

— Reply to this email directly or view it on GitHubhttps://github.com/s3tools/s3cmd/issues/305 .

[mdomsch]

In reading about CloudFront a little more, is there a need to actually invalidate deleted files? Invalidation has a cost [1] of $0.005/object incurred after the first 1000/account/month, which could add up in a hurry. Just let them naturally expire from the cache and you'll avoid that. Not sure what your use case is specifically, but it seems to me that we'd prefer s3cmd to invalidate objects that have changed (e.g. a yum repomd.xml file that changes every time createrepo is run), not invalidate every file that is uploaded, or deleted. We can actually tell which files would change on sync local2remote, but we don't track it that way currently

• we just keep the list of what to upload, delete, or remote copy. I expect the current behavior of s3cmd sync --cf-invalidate would get really expensive in a hurry, issuing invalidation requests on every uploaded file, when you really only care to invalidate those that have changed (thus requiring the upload).

there's a side case here for remote copies also being a change we would want to account for.

If anyone wants to take a stab at all this, be my guest. I haven't used CloudFront myself, and won't any time soon, so waiting on me to write the code will be an exercise in the halting problem [2].

[1] http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Invalidation.html#PayingForInvalidation [2] https://en.wikipedia.org/wiki/Halting_problem

On Thu, Mar 27, 2014 at 7:25 AM, Matt Domsch matt@domsch.com wrote:

sync --cf-invalidate invalidates the newly uploaded files. It doesn't presently invalidate deleted files. Doing so would seem to make sense though. It would require a change to _invalidate_on_cf() to take a list of objects rather than relying on the cmd_sync_local2remote() function scope uploaded_objects_list list, and to get the remote_list list out of _child() and into function scope as well. Seems pretty straightforward to do though, if you wanted to try your hand at a patch.

On Thu, Mar 27, 2014 at 2:20 AM, Saneef Ansari notifications@github.comwrote:

Is there a way I can invalidate files which are been deleted by s3cmd sync?

— Reply to this email directly or view it on GitHubhttps://github.com/s3tools/s3cmd/issues/305 .