search

peak / s5cmd

Parallel S3 and local filesystem execution tool.
MIT License
2.71k stars 240 forks source link

fix: sync glacier objects overwrite #734

Open 4o4x opened 4 months ago

4o4x commented 4 months ago

This pull request addresses a bug (#712) in the sync command where objects in Glacier storage at the destination were being overwritten during synchronization. closes #712

Bug Details:

During the sync process, if an object exists in both the source and destination, and the destination object is in Glacier storage, it was incorrectly being overwritten by the source object.

Fixes:

Previously, if either the source or destination object was in Glacier storage, it was skipped. In the new version, only the source object is skipped, but the destination object is not. If the destination object is skipped, it will be considered as only in the source during the comparison phase, leading to it being overwritten by the source object.

4o4x commented 4 months ago

I wrote a script to observe how the sync command behaves with Glacier objects in AWS CLI. I used these flags in every possible combination:

The most important result obtained:

If the object in the source bucket is a Glacier object, no matter which flags are used, we cannot do anything to the object. Here are the test results:

delete.md sync.md

ilkinulas commented 4 months ago

I wrote a script to observe how the sync command behaves with Glacier objects in AWS CLI. I used these flags in every possible combination:

  • --delete
  • --ignore-glacier-warnings
  • --force-glacier-transfer
  • --storage-class=STANDARD | GLACIER

The most important result obtained:

If the object in the source bucket is a Glacier object, no matter which flags are used, we cannot do anything to the object. Here are the test results:

delete.md sync.md

Thank you for your efforts on this. What happens if the src or dst object's storage class is glacier deep archive? Does GLACIER also covers DEEP ARCHIVE?

igungor commented 1 month ago

@4o4x could you take a look at the PR description if it's still correct after the review changes please? thanks.

4o4x commented 1 month ago

@4o4x could you take a look at the PR description if it's still correct after the review changes please? thanks.

I have removed compare strategy part 👍

igungor commented 1 month ago

@sonmezonur @ilkinulas PTAL

4o4x commented 3 weeks ago

@ilkinulas @sonmezonur PTAL