Closed KristofMorva closed 7 months ago
Just to give one more insight, I have tried with --dryrun
and it seems that there are more files it wants to upload, the AWS CLI just seems to exit after some upload:
with Shutdown request received in result processing thread, shutting down result thread.
. There are no errors preceding it, so I'm a little confused as to why it quits, and why it tries to re-upload the files that are already in the S3 bucket. I have attached the sanitized log in case somebody can see what I can't: log.txt
Also, for reference, rclone sync C:\Same\Path remote:bucket-name --size-only --fast-list --progress --config=""
correctly does not try to upload anything (as everything is up to date on the server).
Hi @KristofMorva, thanks for reaching out. I'm looking into this issue, can you confirm whether the S3 bucket you're using is a directory bucket or not?
Hi @KristofMorva, thanks for reaching out. I'm looking into this issue, can you confirm whether the S3 bucket you're using is a directory bucket or not?
Hey! Sorry, my S3 lingo is pretty bad (it's the first time I'm using it), but my bucket does have folders (not all files are dumped in the root) if that's what you mean :)
No worries! By directory bucket, I'm referring to these. They're listed separately in the S3 console, and all objects in those buckets use the S3 Express One Zone storage class.
No worries! By directory bucket, I'm referring to these. They're listed separately in the S3 console, and all objects in those buckets use the S3 Express One Zone storage class.
Ah okay! As mentioned in my post, I'm using Cloudflare R2, and sadly I'm unsure what's the underlying architecture there. It does not seem to have different options, it has a single-trier pricing (storage + read/write transactions).
Although the problematic files do not follow each other alphabetically
If you use aws s3api list-objects-v2
on the bucket, is the output in alphabetical order? I suspect this might be the same issue as https://github.com/aws/aws-cli/issues/8470, which I'm working on.
Although the problematic files do not follow each other alphabetically
If you use
aws s3api list-objects-v2
on the bucket, is the output in alphabetical order? I suspect this might be the same issue as #8470, which I'm working on.
Yes, they do appear to be in alphabetical order. Thank you for the quick follow-up! :)
Hi there! I'm glad you were able to get this working, but as a reminder, the AWS SDKs and the AWS CLI are designed for and intended for use with AWS. We cannot support the CLI or the SDK in non-AWS environments.
Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.
Describe the bug
Hey, I've seen several tickets for this issue, but none of the solutions seemed to be fitting, so my hope is that it's a regression in a recent version.
While syncing from a local folder to a remote S3-compatible server (Cloudflare R2), some seemingly random files keep getting re-uploaded.
Expected Behavior
When I sync right after syncing with the same command, I expect sync to do nothing.
Current Behavior
It instead uploads some of the files again and again.
For example,
Binaries\DotNET\IOS\DeploymentInterface.dll
is re-uploaded, whileBinaries\DotNET\IOS\DeploymentInterface.pdb
is perfectly fine (same folder, file name besides the extension, etc).Please note that the whole repository is 150.000+ files, dll extension is very common, and only 16 specific files are re-uploaded. What is common about the files:
Engine\Binaries
), but not a common sub-folder (and there are loads of other files of the same type in this folder, so it's not directly a problem with the whole folder)The debug log says
file does not exist at destination
. Is there any way I can debug further, i.e. why it thinks that it does not exist? Like some Super Duper Deep Debug Mode? It is clearly in the XML it received.Here's a log excerpt (deliberately didn't obfuscate any file names):
Reproduction Steps
The command I run is:
Possible Solution
Maybe some more detailed debugging logs could shed some light as to why it fails to find these files in the XML?
Additional Information/Context
While testing my script, I've been terminating AWS CLI very frequently, and the mentioned
Engine\Binaries
folder is on the topmost of the root folder lexically. Although the problematic files do not follow each other alphabetically, is it possible for something to be corrupted due to the interruptions? Or am I being blind and the problem is obvious?CLI version used
2.15.10
Environment details (OS name and version, etc.)
Windows 10.0.19045