search

rclone / rclone

"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files
https://rclone.org
MIT License
44.98k stars 4.03k forks source link

Azure Blob Storage with Archived files cannot be updated #4735

Open Nimce opened 3 years ago

Nimce commented 3 years ago

What is the problem you are having with rclone?

Using Rclone with Azure Blob Storage, when you try to update a file which is in archived state, I get the following error:

020-11-03 21:06:04 ERROR : FILE: Failed to copy: write error: -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, github.com/Azure/azure-storage-blob-go@v0.10.0/azblob/zc_storage_error.go:42 ===== RESPONSE ERROR (ServiceCode=BlobArchived) ===== Description=This operation is not permitted on an archived blob.

I have Soft Deletion turned ON in the blob, same effect (tried with Soft Deletion turned off, same effect as well).

What is your rclone version (output from rclone version)

rclone v1.53.2

Which OS you are using and how many bits (e.g. Windows 7, 64 bit)

Windows 10 64 bit

Which cloud storage system are you using? (e.g. Google Drive)

Azure Blob Storage

The command you were trying to run (e.g. rclone copy /tmp remote:tmp)

rclone sync E: MyAzureBlob: --progress

A log from the command with the -vv flag (e.g. output from rclone -vv copy /tmp remote:tmp)

` 2020-11-03 21:08:20 ERROR : : Failed to copy: write error: -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, github.com/Azure/azure-storage-blob-go@v0.10.0/azblob/zc_storage_error.go:42 ===== RESPONSE ERROR (ServiceCode=BlobArchived) ===== Description=This operation is not permitted on an archived blob. RequestId: Time:**, Details: Code: BlobArchived PUT * Authorization: REDACTED Content-Length: [310669] User-Agent: [rclone/v1.53.2] X-Ms-Client-Request-Id: [] X-Ms-Date: [**] X-Ms-Version: [2019-02-02]

RESPONSE Status: 409 This operation is not permitted on an archived blob. Content-Length: [233] Content-Type: [application/xml] Date: [**] Server: [Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0] X-Ms-Error-Code: [BlobArchived] X-Ms-Version: [2019-02-02] `

nromyn commented 3 years ago

This occurs when rclone tries to update a file that has been moved to the Archive tier. I have a lifecycle policy that archives files older than X days old. This may not be an optimal configuration as rehydration of archived data is a) potentially costly b) time-consuming. I'm not familiar enough with the rehydration process to know whether this is something that can or should be automated. Each item to be rehydrated would need some kind of watch process to update/delete the item once it's rehydrated.

It is possible to issue a delete command to an archived item. There is a cost penalty to doing that.

This may be an issue when mounting the remote as a local filesystem - archived files are not immediately downloadable. Being able to automate the rehydration and retrieval of data would be a valuable addition to rclone for this scenario.

Nimce commented 3 years ago

I agree, or even automate the deletion of archived files and putting the new files instead.

kalmard0 commented 1 week ago

Just highlighting that this is still an issue and --azureblob-archive-tier-delete does not seem to make a difference. #3982 is related.