search

kpfaulkner / azurecopy

copy blobs between azure, s3 and local storage
Apache License 2.0
36 stars 13 forks source link

Non blocking errors when filename contains [1] #31

Open OskarKlintrot opened 7 years ago

OskarKlintrot commented 7 years ago

Command

> ./azurecopy.exe -i https://mybucket.s3.amazonaws.com/myblob -o https://myaccount.blob.core.windows.net/mycontainer -blobcopy

gives

StartCopyFromBlob error msg Fjärrservern returnerade ett fel: (403) Förbjuden.
StartCopyFromBlob error stack    vid Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteSync[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext)
   vid azurecopy.AzureBlobCopyHandler.StartCopy(BasicBlobContainer origBlob, String DestinationUrl, DestinationBlobType destBlobType, Boolean skipIfExists)
Unable to start copying pexels-photo-303024[1].jpeg
Fjärrservern returnerade ett fel: (403) Förbjuden.

would probably translate to

Remote server returned an error: (403) Forbidden.

This is the only file that we couldn't copy and the only file containing [1] (both in the filename and path, which is technically the same).

ps. Thanks for an awesome program, we owe you a beer! ds.

kpfaulkner commented 7 years ago

Hi

Just to check, in the example the blob is called "myblob" but in reality it has the characters [1] in it? ie something like myblob[1] ?

Correct?

if so, I'll try and repro it here and figure out a fix. I bet those []'s will cause an issue.

Cheers

Ken

On Wed, Oct 11, 2017 at 1:33 AM, Oskar Klintrot notifications@github.com wrote:

Command

./azurecopy.exe -i https://mybucket.s3.amazonaws.com/myblob -o https://myaccount.blob.core.windows.net/mycontainer -blobcopy

gives

StartCopyFromBlob error msg Fjärrservern returnerade ett fel: (403) Förbjuden. StartCopyFromBlob error stack vid Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteSync[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext) vid azurecopy.AzureBlobCopyHandler.StartCopy(BasicBlobContainer origBlob, String DestinationUrl, DestinationBlobType destBlobType, Boolean skipIfExists) Unable to start copying pexels-photo-303024[1].jpeg

Fjärrservern returnerade ett fel: (403) Förbjuden.

would probably translate to

Remote server returned an error: (403) Forbidden.

This is the only file that we couldn't copy and the only file containing [1] (both in the filename and path, which is technically the same).

ps. Thanks for an awesome program, we owe you a beer! ds.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/kpfaulkner/azurecopy/issues/31, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHGUopMfnpNuTuiyB5NaCHTVtY_NxYtks5sq4BHgaJpZM4P0CUu .

OskarKlintrot commented 7 years ago

Thank God for the history in Powershell! No, in the example I had just copy/pasted the example from the readme. This is the "actual" command from my Powershell history:

> ./azurecopy.exe -i https://mybucket.s3.amazonaws.com -o https://myaccount.blob.core.windows.net/mycontainer -blobcopy

So it tried to copy everything from S3 to Blob Storage but failed on that file with a [1] in it. I haven't looked in the source but I guess it assumed it was an array and tried to iterate over it or something like that.

Sorry for the confusion!

kpfaulkner commented 7 years ago

Hi

Ok, I'll repro this later today/tonight and sort out a fix.

Cheers

Ken

On Thu, Oct 12, 2017 at 4:32 PM, Oskar Klintrot notifications@github.com wrote:

Thank God for the history in Powershell! No, in the example I had just copy/pasted the example from the readme. This is the "actual" command from my Powershell history:

./azurecopy.exe -i https://mybucket.s3.amazonaws.com -o https://myaccount.blob.core.windows.net/mycontainer -blobcopy

So it tried to copy everything from S3 to Blob Storage but failed on that file with a [1] in it. I haven't looked in the source but I guess it assumed it was an array and tried to iterate over it or something like that.

Sorry for the confusion!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kpfaulkner/azurecopy/issues/31#issuecomment-336025239, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHGUlfoVetxQBu2Z88quKIU_ZKVnp6fks5sraRngaJpZM4P0CUu .

kpfaulkner commented 7 years ago

Hi

I just tried it with a blob called test1[1], copying from S3 to Azure and it worked fine. No issues at all. Can you give me the entire blobname? Maybe there is something else that combines with the [1] that causes the problem.

Thanks

Ken

On Thu, Oct 12, 2017 at 5:19 PM, Ken Faulkner ken.faulkner@gmail.com wrote:

Hi

Ok, I'll repro this later today/tonight and sort out a fix.

Cheers

Ken

On Thu, Oct 12, 2017 at 4:32 PM, Oskar Klintrot notifications@github.com wrote:

Thank God for the history in Powershell! No, in the example I had just copy/pasted the example from the readme. This is the "actual" command from my Powershell history:

./azurecopy.exe -i https://mybucket.s3.amazonaws.com -o https://myaccount.blob.core.windows.net/mycontainer -blobcopy

So it tried to copy everything from S3 to Blob Storage but failed on that file with a [1] in it. I haven't looked in the source but I guess it assumed it was an array and tried to iterate over it or something like that.

Sorry for the confusion!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kpfaulkner/azurecopy/issues/31#issuecomment-336025239, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHGUlfoVetxQBu2Z88quKIU_ZKVnp6fks5sraRngaJpZM4P0CUu .

OskarKlintrot commented 7 years ago

The name should be:

images/pexels-photo-303024[1]/original/pexels-photo-303024[1].jpeg

There where also a few that hade namings like this:

images/pexels-photo-303024[1]/w_500-h_300/pexels-photo-303024[1].jpeg

I just got a bunch of the error message inserted in my first post so unfortunately I have no clue which one caused the issue.

kpfaulkner commented 7 years ago

Cheers, I'll give that a go.

On Thu, Oct 12, 2017 at 7:02 PM, Oskar Klintrot notifications@github.com wrote:

The name should be:

images/pexels-photo-303024[1]/original/pexels-photo-303024[1].jpeg

There where also a few that hade namings like this:

images/pexels-photo-303024[1]/w_500-h_300/pexels-photo-303024[1].jpeg

I just got a bunch of the error message inserted in my first post so unfortunately I have no clue which one caused the issue.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kpfaulkner/azurecopy/issues/31#issuecomment-336052126, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHGUnJtg3PQScJREorXk5ggc8RvcvPyks5srcePgaJpZM4P0CUu .

OskarKlintrot commented 7 years ago

Great, thanks!

kpfaulkner commented 7 years ago

hmmmm still no luck for me. It's working fine.

Can you copy/paste some of the output from when you ran it?

On Thu, Oct 12, 2017 at 7:17 PM, Oskar Klintrot notifications@github.com wrote:

Great, thanks!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kpfaulkner/azurecopy/issues/31#issuecomment-336055699, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHGUhE48CbEAZYimOwPHBTyHWc7_Gkgks5srcswgaJpZM4P0CUu .

OskarKlintrot commented 7 years ago

Sorry, I don't have more than that. The blob storage container is now in production so I don't really want to re-run the command...

kpfaulkner commented 7 years ago

fair enough. I'll try some alternatives and try and see if I can get any similar errors.

Thanks

Ken

On Thu, Oct 12, 2017 at 7:46 PM, Oskar Klintrot notifications@github.com wrote:

Sorry, I don't have more than that. The blob storage container is now in production so I don't really want to re-run the command...

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kpfaulkner/azurecopy/issues/31#issuecomment-336063074, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHGUkQMyiw4mdYycIYVVxicZ0mnc1Nnks5srdH3gaJpZM4P0CUu .

kpfaulkner commented 7 years ago

Ahhh able to repro! I missed out the -blobcopy flag. Ok, will investigate further.

From: Oskar Klintrot Sent: Thursday, 12 October 2017 7:46 PM To: kpfaulkner/azurecopy Cc: Ken Faulkner; Comment Subject: Re: [kpfaulkner/azurecopy] Non blocking errors when filename contains[1] (#31)

Sorry, I don't have more than that. The blob storage container is now in production so I don't really want to re-run the command... — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

kpfaulkner commented 7 years ago

Have checked if its the S3 or Azure side. It's definitely Azure, but unsure if there is anything I can do about it. The shared URL that S3 makes is definitely valid (ie I can copy it to a browser and can download manually). I'm still investigating if there is anything I can do on the Azure side. I don't believe its the Azure name containing [1]... but it's more the fact when we tell Azure to copy the file, it dies.

Honestly unsure if there is anything I can do about it, but will keep investigating.

Cheers

Ken