peak / s5cmd

Parallel S3 and local filesystem execution tool.
MIT License
2.58k stars 226 forks source link

Preserving timestamps of files via a flag? #532

Open Ahuge opened 1 year ago

Ahuge commented 1 year ago

Hi, I have a use case where I am using s5cmd as a "stash" for user's uncommitted files from Perforce and this has been working great in terms of speed, however Perforce takes the timestamps on the files in account when determining if the user need to refresh their working space.

I am looking to see if we can include a flag that will record timestamps of files in the s3 metadata and then re-apply them during download. I'll see if I can get a PR up to help with this.

Thanks! -Alex

ajw725 commented 1 year ago

this is also critical for the sync command that was introduced in 2.0.0. aws s3 sync does preserve the timestamps of the files, so if you sync a directory from S3 to local and then turn right back around and sync it back to S3, it's a no-op because the sizes and timestamps haven't changed. if you try to do that with s5cmd, it copies every single file back up to S3, because it changes the last modified timestamps when it syncs the directory down from S3.

(the actual use case here involves changing/adding some files between the download and the upload.)

bounlu commented 1 year ago

Yes, I also agree that s5cmd should preserve timestamp as it would be in case of aws s3. I got into below issue today trying to figure out why my files look older than expected:

$ aws s3 ls s3://xxx
2023-08-02  14:16:07    50618045579 abc_1_val_1.fq.gz
2023-08-02  14:16:07    45907164979 abc_2_val_2.fq.gz
2023-08-02  16:25:17    74599091180 abc_1_val_1.fq.gz
2023-08-02  16:25:17    67202267370 abc_2_val_2.fq.gz
2023-08-02  16:05:07    68417358438 abc_1_val_1.fq.gz
2023-08-02  16:05:07    61942134475 abc_2_val_2.fq.gz
2023-08-02  15:06:47    60423742857 abc_1_val_1.fq.gz
2023-08-02  15:06:47    54783772662 abc_2_val_2.fq.gz
2023-08-02  14:44:29    51372815911 abc_1_val_1.fq.gz
2023-08-02  14:44:29    47378846706 abc_2_val_2.fq.gz

$ s5cmd ls s3://xxx
2023/08/02  06:16:07    50618045579 abc_1_val_1.fq.gz
2023/08/02  06:16:07    45907164979 abc_2_val_2.fq.gz
2023/08/02  08:25:17    74599091180 abc_1_val_1.fq.gz
2023/08/02  08:25:17    67202267370 abc_2_val_2.fq.gz
2023/08/02  08:05:07    68417358438 abc_1_val_1.fq.gz
2023/08/02  08:05:07    61942134475 abc_2_val_2.fq.gz
2023/08/02  07:06:47    60423742857 abc_1_val_1.fq.gz
2023/08/02  07:06:47    54783772662 abc_2_val_2.fq.gz
2023/08/02  06:44:29    51372815911 abc_1_val_1.fq.gz
2023/08/02  06:44:29    47378846706 abc_2_val_2.fq.gz