kpfaulkner
commented
8 years ago Hi
I certainly haven't tried with volumes that large. Hmmm looking at the stack trace you supplied its the Amazon client library that is dying when trying to get the list of S3 blobs. The only thing I can suggest is perform multiple copies but targeting subdirectories.
ie if you have /media/photogroup1/.... and /media/photogroup2/.... then try 2 separate commands, one on each virtual directory.
I can investigate if there is an alternative way to get the list of blobs (maybe I could segment it within azurecopy itself), but that may take a little while to sort out.
Do you think performing multiple commands each targeting directories will work for you?
Thanks
Ken
On Wed, Jan 20, 2016 at 1:16 PM, Simon Horn notifications@github.com wrote:
Hi,
I've tried copying over the ~15TB worth of S3 files to an Azure target and received the debug error below There are ~6 million files in so maybe it was too much to handle? I saw the process use up to ~4GB of memory
If I target a much smaller virtual directory it works
PS C:\admin\azurecopy-115\azurecopy> \azurecopyexe -i https://s3amazonawscom/media/photos/ -o https://mediablobcorewindowsnet/photos -v -db -blobcopy GetHandler start GetHandler retrieved azurecopyS3Handler GetHandler start GetHandler retrieved azurecopyAzureHandler Unknown error generated Please report to Github page https://githubcom/kpfaulkner/azurecopy/issues Can view underly ing stacktrace by adding -db flag at AmazonRuntimeInternalUnmarshallerUnmarshall(IExecutionContext executionContext) at AmazonRuntimeInternalUnmarshallerInvokeSync(IExecutionContext executionContext) at AmazonRuntimeInternalPipelineHandlerInvokeSync(IExecutionContext executionContext) at AmazonS3InternalAmazonS3ResponseHandlerInvokeSync(IExecutionContext executionContext) at AmazonRuntimeInternalPipelineHandlerInvokeSync(IExecutionContext executionContext) at AmazonRuntimeInternalErrorHandlerInvokeSync(IExecutionContext executionContext) at AmazonRuntimeInternalPipelineHandlerInvokeSync(IExecutionContext executionContext) at AmazonRuntimeInternalCallbackHandlerInvokeSync(IExecutionContext executionContext) at AmazonRuntimeInternalPipelineHandlerInvokeSync(IExecutionContext executionContext) at AmazonRuntimeInternalSignerInvokeSync(IExecutionContext executionContext) at AmazonRuntimeInternalPipelineHandlerInvokeSync(IExecutionContext executionContext) at AmazonRuntimeInternalCredentialsRetrieverInvokeSync(IExecutionContext executionContext) at AmazonRuntimeInternalPipelineHandlerInvokeSync(IExecutionContext executionContext) at AmazonRuntimeInternalRetryHandlerInvokeSync(IExecutionContext executionContext) at AmazonRuntimeInternalPipelineHandlerInvokeSync(IExecutionContext executionContext) at AmazonRuntimeInternalCallbackHandlerInvokeSync(IExecutionContext executionContext) at AmazonRuntimeInternalPipelineHandlerInvokeSync(IExecutionContext executionContext) at AmazonS3InternalAmazonS3KmsHandlerInvokeSync(IExecutionContext executionContext) at AmazonRuntimeInternalPipelineHandlerInvokeSync(IExecutionContext executionContext) at AmazonRuntimeInternalEndpointResolverInvokeSync(IExecutionContext executionContext) at AmazonRuntimeInternalPipelineHandlerInvokeSync(IExecutionContext executionContext) at AmazonS3InternalAmazonS3PostMarshallHandlerInvokeSync(IExecutionContext executionContext) at AmazonRuntimeInternalPipelineHandlerInvokeSync(IExecutionContext executionContext) at AmazonRuntimeInternalMarshallerInvokeSync(IExecutionContext executionContext) at AmazonRuntimeInternalPipelineHandlerInvokeSync(IExecutionContext executionContext) at AmazonS3InternalAmazonS3PreMarshallHandlerInvokeSync(IExecutionContext executionContext) at AmazonRuntimeInternalPipelineHandlerInvokeSync(IExecutionContext executionContext) at AmazonRuntimeInternalCallbackHandlerInvokeSync(IExecutionContext executionContext) at AmazonRuntimeInternalPipelineHandlerInvokeSync(IExecutionContext executionContext) at AmazonS3InternalAmazonS3ExceptionHandlerInvokeSync(IExecutionContext executionContext) at AmazonRuntimeInternalPipelineHandlerInvokeSync(IExecutionContext executionContext) at AmazonRuntimeInternalErrorCallbackHandlerInvokeSync(IExecutionContext executionContext) at AmazonRuntimeInternalPipelineHandlerInvokeSync(IExecutionContext executionContext) at AmazonRuntimeInternalMetricsHandlerInvokeSync(IExecutionContext executionContext) at AmazonRuntimeInternalRuntimePipelineInvokeSync(IExecutionContext executionContext) at AmazonRuntimeAmazonServiceClientInvokeTRequest,TResponse at AmazonS3AmazonS3ClientListObjects(ListObjectsRequest request) at azurecopyS3HandlerListBlobsInContainer(String containerName, String blobPrefix, Boolean debug) at azurecopycommandProgramGetSourceBlobList(IBlobHandler inputHandler) at azurecopycommandProgramDoNormalCopy() at azurecopycommandProgramProcess(Boolean debugMode) at azurecopycommandProgramMain(String[] args)
Cheers, Simon
— Reply to this email directly or view it on GitHub https://github.com/kpfaulkner/azurecopy/issues/7.
nightwallaby
Hi,
I've tried copying over the ~1.5TB worth of S3 files to an Azure target and received the debug error below. There are ~6 million files in so maybe it was too much to handle? I saw the process use up to ~4GB of memory.
If I target a much smaller virtual directory it works.
PS C:\admin\azurecopy-1.1.5\azurecopy> .\azurecopy.exe -i https://s3.amazonaws.com/media/photos/ -o https://media.blob.core.windows.net/photos -v -db -blobcopy GetHandler start GetHandler retrieved azurecopy.S3Handler GetHandler start GetHandler retrieved azurecopy.AzureHandler Unknown error generated. Please report to Github page https://github.com/kpfaulkner/azurecopy/issues . Can view underly ing stacktrace by adding -db flag. at Amazon.Runtime.Internal.Unmarshaller.Unmarshall(IExecutionContext executionContext) at Amazon.Runtime.Internal.Unmarshaller.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) at Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.ErrorHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.Signer.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.CredentialsRetriever.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.RetryHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) at Amazon.S3.Internal.AmazonS3KmsHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.EndpointResolver.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) at Amazon.S3.Internal.AmazonS3PostMarshallHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.Marshaller.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) at Amazon.S3.Internal.AmazonS3PreMarshallHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) at Amazon.S3.Internal.AmazonS3ExceptionHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.MetricsHandler.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.Internal.RuntimePipeline.InvokeSync(IExecutionContext executionContext) at Amazon.Runtime.AmazonServiceClient.Invoke[TRequest,TResponse](TRequest request, IMarshaller`2 marshaller, Response Unmarshaller unmarshaller) at Amazon.S3.AmazonS3Client.ListObjects(ListObjectsRequest request) at azurecopy.S3Handler.ListBlobsInContainer(String containerName, String blobPrefix, Boolean debug) at azurecopycommand.Program.GetSourceBlobList(IBlobHandler inputHandler) at azurecopycommand.Program.DoNormalCopy() at azurecopycommand.Program.Process(Boolean debugMode) at azurecopycommand.Program.Main(String[] args)
Cheers, Simon