kpfaulkner / azurecopy

copy blobs between azure, s3 and local storage
Apache License 2.0
36 stars 13 forks source link

Crashes trying to list Amazon files #4

Closed apenngrace closed 9 years ago

apenngrace commented 9 years ago

azurecopy works fine when I list my Azure files, but it crashes if I try to list my Amazon files.

For example, azurecopy exists on Unknown error with this command: "azurecopy -list https://s3.amazonaws.com/mybucket".

Here is the trace I got when I used the "-db" flag:

Unknown error generated. Please report to Github page https://github.com/kpfaulkner/azurecopy/issues .  Can view underlying stacktrace by adding -db flag.
   at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleException(IExecutionContext executionContext, HttpErrorResponseException exception)
   at Amazon.Runtime.Internal.ExceptionHandler`1.Handle(IExecutionContext executionContext, Exception exception)
   at Amazon.Runtime.Internal.ErrorHandler.ProcessException(IExecutionContext executionContext, Exception exception)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.GenericHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.GenericHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.GenericHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.GenericHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.GenericHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.GenericHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.GenericHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.GenericHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.GenericHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.GenericExceptionHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.GenericExceptionHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.MetricsHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RuntimePipeline.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.AmazonServiceClient.Invoke[TRequest,TResponse](TRequest request, IMarshaller`2 marshaller, ResponseUnmarshaller unmarshaller)
   at Amazon.S3.AmazonS3Client.GetBucketLocation(GetBucketLocationRequest request)
   at Amazon.S3.AmazonS3Client.GetBucketLocation(String bucketName)
   at azurecopy.Utils.S3Helper.GenerateS3Client(String accessKey, String secretKey, String bucketName)
   at azurecopy.S3Handler.ListBlobsInContainer(String baseUrl)
   at azurecopycommand.Program.DoList()
   at azurecopycommand.Program.Process()
   at azurecopycommand.Program.Main(String[] args)
kobynet commented 9 years ago

+1

kpfaulkner commented 9 years ago

Sorry for late response... for some reason Github didn't notify me (although it is elsewhere). Will investigate.

Which version was being used?

kobynet commented 9 years ago

i was using latest stable (1.0.0)

kpfaulkner commented 9 years ago

Thanks, I'll look into that now.

kobynet commented 9 years ago

It might worth checking to migrate to new aws sdk which has exasperated package for each service

kpfaulkner commented 9 years ago

Just downloaded 1.0.0.... modified the config file with my keys/region etc.... and did the following:

azurecopy.exe -list https://s3.amazonaws.com/testken123 Test.cmd (Test.cmd https://s3.amazonaws.com/testken123Test.cmd) Test2.cmd (Test2.cmd https://s3.amazonaws.com/testken123Test2.cmd) Test.cmd (test/Test.cmd https://s3.amazonaws.com/testken123test/Test.cmd) Test2.cmd (test/Test2.cmd https://s3.amazonaws.com/testken123test/Test2.cmd) Test2.cmd (test/ken/Test2.cmd https://s3.amazonaws.com/testken123test/ken/Test2.cmd) test2 (test/ken/test2 https://s3.amazonaws.com/testken123test/ken/test2) Operation took 2029 ms faulkner@DEV C:\temp\azurecopy-1.0.0\azurecopy

Are you setting the values for:

AWSAccessKeyID AWSSecretAccessKeyID AWSRegion

?

Thanks

Ken

kobynet commented 9 years ago

There was a problem with my aws account, so it seems to work now.

kobynet commented 9 years ago

@apenngrace it might worth trying https://mybucket.s3.amazonaws.com/ instead of https://s3.amazonaws.com/mybucket

apenngrace commented 9 years ago

Thanks!

Sent from my iPhone

On Jul 30, 2015, at 3:24 AM, Koby notifications@github.com wrote:

@apenngrace it might worth trying https://mybucket.s3.amazonaws.com/ instead of https://s3.amazonaws.com/mybucket

— Reply to this email directly or view it on GitHub.

kpfaulkner commented 9 years ago

Although for most operation both URL formats (https://bucketname.s3.amazonaws.com and https://s3.amazonaws.com/bucketname) were handled, admittedly there was a bug for the "list" command (when using the first option).

I've just released 1.0.1 that fixes the problem. In saying that though, the https://s3.amazonaws.com/bucketname format should have always worked.

Thanks for the heads up on the issue.

Ken