Error trying to copy page blob from AZ to AWS

[eosfor]

Unknown error generated. Please report to Github page https://github.com/kpfaulkner/azurecopy/issues . Can view underlying stacktrace by adding -db flag. at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteSync[T](RESTCommand1 cmd, IRetryPolicy policy, OperationContext operationContext) at Microsoft.WindowsAzure.Storage.Blob.CloudPageBlob.DownloadRangeToStream(Stream target, Nullable1 offset, Nullable`1 length, AccessCondition accessCondition, BlobRequestOptions options, OperationContext operationContext) at Microsoft.WindowsAzure.Storage.Blob.CloudPageBlob.DownloadToStream(Stream target, AccessCondition accessCondition, BlobRequestOptions options, OperationContext operationContext) at azurecopy.AzureHandler.ReadPageBlob(ICloudBlob blobRef, String fileName) at azurecopy.AzureHandler.ReadBlob(String url, String filePath) at azurecopycommand.Program.DoNormalCopy() at azurecopycommand.Program.Process() at azurecopycommand.Program.Main(String[] args)

[kpfaulkner]

Currently attempting to repro.

[eosfor]

Try page blob bigger than 1gb at least. better to create VM and then try to copy its files

[kpfaulkner]

Thanks, will try that.

[kpfaulkner]

Apart from -i and -o, which other parameters were used?

[eosfor]

Nothing. Actually it failed with both.

PS C:\temp\azurecopy-0.16.3\azurecopy> .\azurecopy.exe -o https://somestuff.s3-website-us-east-1.amazonaws.com -s3k -i https://some/some/some -ak aKey -v -db -d G:\toaws Copying https://some/some/some to https://somestuff.s3-website-us-east-1.amazonaws.com Unknown error generated. Please report to Github page https://github.com/kpfaulkner/azurecopy/issues . Can view underlying stacktrace by adding -db flag. at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleException(IExecutionContext executionContext, HttpErrorResponseException exception) at Amazon.Runtime.Internal.ExceptionHandler1.Handle(IExecutionContext executionContext, Exception exception) at Amazon.Runtime.Internal.ErrorHandler.ProcessException(IExecutionContext executionContext, Exception exception) at Amazon.Runtime.Internal.ErrorHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.GenericHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.GenericHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.GenericHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.RetryHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.GenericHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.GenericHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.GenericHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.GenericHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.GenericHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.GenericHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.GenericExceptionHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.GenericExceptionHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.MetricsHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.RuntimePipeline.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.AmazonServiceClient.Invoke[TRequest,TResponse](TRequest request, IMarshaller2 marshaller, ResponseUnmarshaller unmarshaller) at Amazon.S3.AmazonS3Client.GetBucketLocation(GetBucketLocationRequest request) at Amazon.S3.AmazonS3Client.GetBucketLocation(String bucketName) at azurecopy.Utils.S3Helper.GenerateS3Client(String accessKey, String secretKey, String bucketName) at azurecopy.S3Handler.WriteBlob(String url, Blob blob, Int32 parallelUploadFactor, Int32 chunkSizeInMB) at azurecopycommand.Program.DoNormalCopy() at azurecopycommand.Program.Process() at azurecopycommand.Program.Main(String[] args)

[kpfaulkner]

I believe the original issue was due to the application being compiled as 32bit as opposed to 64 bit. My testing ended up with an out of memory issue. I've now generated a pre-release 1.2.0 (https://github.com/kpfaulkner/azurecopy/releases/tag/1.2.0) which is 64 bit. Please try that version and let me know if that solves your issue.

The thing is, if you use the -d flag (which indicates that the blob should be downloaded to local storage instead of keeping it in memory) I wouldn't get the problem. Please let me know if 1.2.0 fixes your problem, if not, I'll continue investigating.

[eosfor]

Let me clarify one point here. We were trying different tool under the same conditions - cherrysafe.com and it also failed. We were told that the issue somehow related to sparse files. Meaning that page blobs are sparse but S3 does not support that so the only way to copy them properly is to download them first and then upload.

[kpfaulkner]

Interesting. I was initially just trying with block blobs then I noticed this memory issue (hence wanted to know if that fixed it for you). I'll be trying page blob tonight.

Given you're going from Azure to AWS I think you'll have to download locally then upload to S3 (via whichever tool you like). If going TO Azure then there is an option that allows uploading without having to download first, but only going TO Azure.

If you can check if the x64 version fixes your problem, please let me know. Otherwise I'll continue investigating.