File delete fails for files in the format <GUID>##9999.csv

maps-mike commented 2 months ago

Preflight Checklist

[X] I have installed the latest version of Storage Explorer.
[X] I have checked existing resources, including the troubleshooting guide and the release notes.
[X] I have searched for similar issues.

Storage Explorer Version

1.33.1

Regression From

No response

Architecture

x64

Storage Explorer Build Number

20240410.2

Platform

Windows

OS Version

Windows 10

Bug Description

File delete fails for files in the format ##9999.csv

Steps to Reproduce

Create ADLS G2 storage account create folder add a file called ab9f89ba-f069-44b1-84a6-7dda8da39147##7587.csv try to delete file using storage explorer

Actual Experience

Error message in storage explorer:

Deletion of 'ab9f89ba-f069-44b1-84a6-7dda8da39147##7587.csv' from 'landing/Transactional/upload_api_sandbox/20240430/' failed: 0 completed, 1 failed (used name and key) Started at: 01/05/2024 08:17, Duration: 4 seconds

Error in log:

2024/05/01 07:17:14 ERR: [P#0-T#0] https://adlsdatahubraw.dfs.core.windows.net/landing/Transactional/upload_api_sandbox/20240430/ab9f89ba-f069-44b1-84a6-7dda8da39147%23%237587.csv: 404: DELETE ERROR -404 The specified path does not exist.. X-Ms-Request-Id:4666f2af-401f-000c-0997-9b4c15000000

Expected Experience

File is deleted

Additional Context

Other files can be deleted without issue it is just files in the format ([GUID]##9999.csv) above that cannot be deleted.

The problem files can still be deleted by other mechanisms - I usually use the Azure portal.

craxal commented 2 months ago

@maps-mike My guess is the double hash (##) is what's causing the issue. Some characters need special handling. Strange, though, as the URL in the error message looks correct. Can you confirm that URL of the blob used by Storage Explorer is the same one used by the Azure portal? You can check this by looking at the blob's properties both in Storage Explorer and the portal.

craxal commented 2 months ago

@maps-mike I am not able to reproduce this issue, either by deleting one or multiple blobs. Can you add any more details about what exactly you were doing? When you refresh the explorer view, are the blobs you're trying to delete still present?

craxal commented 1 month ago

@maps-mike Have you had a chance to reproduce this issue and gather more information? Anything you can provide can help us investigate further.

maps-mike commented 1 month ago

Hi Craig – I’ve just done a test this morning, the exact process is:

Upload 2 CSV files to a file path “landing/Transactional/error”
- Note path is an ADLSG2 file system, NOT Blob storage
  - Performance: Standard
  - Replication: LRS
  - Account Kind: StorageV2 (general purpose v2)
  - Firewall IS enabled.
- Files:
  - 35ba9a88-09d6-4c9b-bf7f-3231a283c443##7580.csv
  - 2237f762-77f7-4039-a531-968a442359b5##7581.csv
    - Files contain 2 lines of CSV data (header & 1 row)
    - File size is just over 1K
I click “delete” in Storage Explorer
Storage explorer confirms “are you sure” with correct file name
I click “Yes”
Failure message returned, file still present in filesystem. @.***

Copy AZ command to clipboard gives me this (XXX for sensitive info)

$env:AZCOPY_CONCURRENCY_VALUE = "AUTO"; $env:AZCOPY_CRED_TYPE = "SharedKey"; $env:ACCOUNT_NAME = "XXX"; $env:ACCOUNT_KEY = "XXX"; ./azcopy.exe remove https://XXX.dfs.core.windows.net/landing/Transactional/error/35ba9a88-09d6-4c9b-bf7f-3231a283c443%23%237580.csv --from-to=BlobFSTrash --recursive --log-level=INFO; $env:AZCOPY_CONCURRENCY_VALUE = ""; $env:AZCOPY_CRED_TYPE = ""; $env:ACCOUNT_NAME = ""; $env:ACCOUNT_KEY = "";

Log contains this:

2024/05/13 06:20:02 AzcopyVersion 10.23.0 2024/05/13 06:20:02 OS-Environment windows 2024/05/13 06:20:02 OS-Architecture amd64 2024/05/13 06:20:02 Log times are in UTC. Local time is 13 May 2024 07:20:02 2024/05/13 06:20:03 ISO 8601 START TIME: to copy files that changed before or after this job started, use the parameter --include-before=2024-05-13T06:19:57Z or --include-after=2024-05-13T06:19:57Z 2024/05/13 06:20:03 Authenticating to destination using SharedKey 2024/05/13 06:20:03 Authenticating to source using SharedKey 2024/05/13 06:20:03 Job-Command remove https://XXX.dfs.core.windows.net/landing/Transactional/error/35ba9a88-09d6-4c9b-bf7f-3231a283c443%23%237580.csv --output-type=json --cancel-from-stdin --from-to=BlobFSTrash --recursive --log-level=INFO 2024/05/13 06:20:03 Number of CPUs: 8 2024/05/13 06:20:03 Max file buffer RAM 4.000 GB 2024/05/13 06:20:03 Max concurrent network operations: will be dynamically tuned up to 3000 (Based on auto-tuning limit. Set AZCOPY_CONCURRENCY_VALUE environment variable to override) 2024/05/13 06:20:03 Check CPU usage when dynamically tuning concurrency: true (Based on hard-coded default. Set AZCOPY_TUNE_TO_CPU environment variable to true or false override) 2024/05/13 06:20:03 Max concurrent transfer initiation routines: 64 (Based on hard-coded default. Set AZCOPY_CONCURRENT_FILES environment variable to override) 2024/05/13 06:20:03 Max enumeration routines: 16 (Based on hard-coded default. Set AZCOPY_CONCURRENT_SCAN environment variable to override) 2024/05/13 06:20:03 Parallelize getting file properties (file.Stat): false (Based on AZCOPY_PARALLEL_STAT_FILES environment variable) 2024/05/13 06:20:03 Max open files when downloading: 2147479959 (auto-computed) 2024/05/13 06:20:03 Final job part has been created 2024/05/13 06:20:03 Trying 4 concurrent connections (initial starting point) 2024/05/13 06:20:03 Final job part has been scheduled 2024/05/13 06:20:03 WARN: [P#0-T#0] Displayed file count will be either 1 or based upon list-of-files entries, and thus inaccurate, as deletes are performed recursively service-side. 2024/05/13 06:20:04 ==> REQUEST/RESPONSE (Try=1/396.6515ms, OpTime=619.7381ms) -- RESPONSE SUCCESSFULLY RECEIVED HEAD https://XXX.blob.core.windows.net/landing/Transactional%2Ferror%2F35ba9a88-09d6-4c9b-bf7f-3231a283c443%23%237580.csv X-Ms-Request-Id: [774e3d22-101e-0087-25fd-a44878000000]

2024/05/13 06:20:04 ==> REQUEST/RESPONSE (Try=1/39.9697ms, OpTime=260.7882ms) -- RESPONSE SUCCESSFULLY RECEIVED DELETE https://XXX.dfs.core.windows.net/landing/Transactional/error/35ba9a88-09d6-4c9b-bf7f-3231a283c443?recursive=false#%237580.csv X-Ms-Request-Id: [19fc7ae5-301f-0016-05fd-a42dca000000]

2024/05/13 06:20:04 ERR: [P#0-T#0] https://XXX.dfs.core.windows.net/landing/Transactional/error/35ba9a88-09d6-4c9b-bf7f-3231a283c443%23%237580.csv: 404: DELETE ERROR -404 The specified path does not exist.. X-Ms-Request-Id:19fc7ae5-301f-0016-05fd-a42dca000000

2024/05/13 06:20:04 JobID=d2cd78e0-5bab-0444-6f05-3f610e137603, Part#=0, TransfersDone=1 of 1 2024/05/13 06:20:04 all parts of entire Job d2cd78e0-5bab-0444-6f05-3f610e137603 successfully completed, cancelled or paused 2024/05/13 06:20:04 is part of Job which 1 total number of parts done 2024/05/13 06:20:05 PERF: primary performance constraint is Unknown. States: T: 0, GRs: 4 2024/05/13 06:20:05 100.0 %, 0 Done, 1 Failed, 0 Pending, 0 Skipped, 1 Total, 2024/05/13 06:20:05 Closing Log

Using portal to try delete:
- Go to Stroage account
- Choose Storage Browser
- Blob containers ->
- Click tick box next to file
- Click delete, get confirmation message:
- @.***
- Click Yes, file deleted in Portal & Storage explorer

Comparing URL in Storage Explorer & portal:

Portal https://PATH/35ba9a88-09d6-4c9b-bf7f-3231a283c443##7580.csv Storage Explorer: https://PATH/35ba9a88-09d6-4c9b-bf7f-3231a283c443%23%237580.csv

I agree that its got to be the ## characters – I’m not convinced the correct escape sequence is being used in the command, or maybe its my version of AZCopy?

Anyways, hope this helps.

Mike

craxal commented 1 month ago

@maps-mike For testing purposes, can you upload and delete some files with some variations? For example, does the issue still reproduce if:

You only have one "#" character
You have no "#" characters
You change the directory structure, such as moving the file to the root
You delete multiple files at once

Strictly speaking, "#" characters are not legal in a URL path. Azure portal might be un-escaping characters for display convenience.

The fact that I still can't repo the issue suggests we are escaping the "#" characters correctly:

./azcopy remove "https://XXX.dfs.core.windows.net/test01/01234567-89ab-cdef-0123-456789abcdef%23%231234.csv" --from-to=BlobFSTrash --recursive --log-level=INFO;

maps-mike commented 1 month ago

Hi Craig – results & observations below:

You only have one "#" character.
- [Mike] Same problem – error deleting.
You have no "#" characters
- [Mike] Success – deletes as expected.
You change the directory structure, such as moving the file to the root
- [Mike] Same problem – error deleting (from root & other folder paths)
You delete multiple files at once
- [Mike] Same problem – error deleting.

I have observed the problem only occurs on ADLS Gen2 Storage accounts – if I upload ## files to a Blob storage account the delete works as expected.

Note – I copied & renamed the same file in all tests so all other factors should be consistent.

Short file name cush as “3#.csv” fail as well.

No worries if this one goes down as “cannot reproduce”, I have a workaround, though I’m equally happy to try more tests.

Mike

craxal commented 1 month ago

@maps-mike Thank you for that information!

Ah ha! Sorry, I should have tried this first: I was able to reproduce the issue using our 1.33.0 release. It doesn't reproduce in our working builds now, so I suspect the issue has already been addressed (by updating our version of AzCopy, probably).

Here is a link to a private build. Can you download, install, and give it a try to see if the problem persists?

[LINK REMOVED]

maps-mike commented 1 month ago

Thanks Craig, I’ll do that – the install needs .Net 8 (& admin rights) so I’ll need to get IT to do that install so might take a little while to get back but sounds like it’ll be sorted!

Mike

craxal commented 1 month ago

@maps-mike Any progress? If you need a new link, please let me know.

maps-mike commented 1 month ago

Hi Craig – sorry I was a bit slow, got the new version installed now & can confirm the issue is resolved, I can delete these ## files as expected!

Thank You!

craxal commented 1 month ago

Excellent! Our next release (1.34.0) contains an AzCopy update, so that means the next release should resolve your issue.

microsoft / AzureStorageExplorer