search

Azure / azure-storage-azcopy

The new Azure Storage data transfer utility - AzCopy v10
MIT License
602 stars 216 forks source link

azcopy url-decodes slashes from directory names #2753

Open frank-siebert-tracetronic opened 1 month ago

frank-siebert-tracetronic commented 1 month ago

Which version of the AzCopy was used?

10.24.0

Note: The version is visible when running AzCopy without any argument

Which platform are you using? (ex: Windows, Mac, Linux)

Windows

What command did you run?

azcopy copy "C:/Users/XXX/testproject/testfolder%5C_something_wrong" "https://xxx.blob.core.windows.net/zzz?sp=racw&st=2024-07-22T10:00:55Z&se=2024-07-22T18:00:55Z&spr=https&sv=2022-11-02&sr=c&sig=abc" --recursive

Note: Please remove the SAS to avoid exposing your credentials. If you cannot remember the exact command, please retrieve it from the beginning of the log file.

What problem was encountered?

I have some directories that have backward slashes in their names (they originally come from unix systems), so they get url encoded on my disk (therefore the %5C in the directory name). What I am trying to do is upload them in the exact same manner, so no decoding taking place, just the plain "%5C" within the directory name. However what is happening instead is the %5C gets replaced with a slash and then gets interpreted as a seperator, so in the storage I will get a subdirectory named "_something_wrong"

Expected: storageSource/testfolder%5C_something_wrong Actual: storageSource/testfolder/_something_wrong

I have tried to do a double url encoding by renaming the directory to %255C but that didn't work either and will not be decoded at all, so it stays that way and therefore is also not a solution.

How can we reproduce the problem in the simplest way?

Create a simple folder locally with a %5C in the directory name and transfer it to an arbitrary storage container via azcopy copy.

Have you found a mitigation/solution?

No

ashruti-msft commented 1 month ago

Hi, I was not able to reproduce this issue. Test scenario: I created a folder named "new_%5Cthing" on local and copied it to storage using azcopy. Storage showed a folder created with the name "new%5C_thing". Please let me know if I understood your issue correctly or not. Thanks.

ashruti-msft commented 1 month ago

Okay I can reproduce this on windows not on linux. I'll get back to you regarding this.

frank-siebert-tracetronic commented 1 month ago

Okay I can reproduce this on windows not on linux. I'll get back to you regarding this.

Hi, are there any updates on this issue?

ashruti-msft commented 1 month ago

Hi, can you please share your debug logs for this run. Thanks!

frank-siebert-tracetronic commented 1 month ago

Hi, can you please share your debug logs for this run. Thanks!

Well, those are corporate specific logs, so this is unfortunately not possible. I thought you could reproduce this. What part of the logs do you need?

Data-Engineer-Joe commented 1 month ago

Hello @ashruti-msft, you said you can reproduce this issue. What's the next step here, this behaviour hurts pretty much because it rips our hive partitioning into pieces. Please follow up on this. I hope there will be a fix. Thanks.

jschelter-co commented 3 weeks ago

Hello @ashruti-msft. I'd also like to ask if you've had the chance to look more closely into this issue. It really is a major blocker on our side. Any feedback is highly appreciated. Thanks a lot.

Data-Engineer-Joe commented 2 weeks ago

Hello @ashruti-msft, could you please leave a comment on your progress regarding this issue?

ashruti-msft commented 2 weeks ago

Hi @Data-Engineer-Joe I'll discuss this issue with the team and get back to you.

Data-Engineer-Joe commented 1 day ago

Hello @ashruti-msft and @adreed-msft , do you have any news on the issue? It's really a major problem for us! Please give me some feedback, what do you plan or if you can offer something. Thank you.