Unknown Error

[alexlenail]

Hi Ken,

I'm hoping to use AzureCopy to move a fairly large volume of data from S3 to Azure Blobs. I haven't been able to get it to work, I run into this problem when it tries to copy the first file:

It seems like it manages to get the first 12MB across. Any idea what the problem might be here? Too long a filename? Too big a file?

Thanks,

--Alex

[kpfaulkner]

Hi

Hmmm it's been used to copy pretty large files (and also lots of files). Can you give me some stats on # and size of files?

Also, can you copy/paste the exact command you used? It was clipped off from the screen capture.

Thanks

Ken

[alexlenail]

Hi @kpfaulkner ,

Thanks for getting back to me so promptly! Sorry for clipping the image. Here's a better one:

I'm trying to move something on the order of 30TB. Maximum file size is 200GB. A couple thousand files I think.

[kpfaulkner]

Thanks for the updated screen shot. Ok, couple of things I'd suggest.

Firstly, add the -d flag. This will tell it to cache the file locally instead of in memory (that's the part which is currently blowing up for you).

If that works, great. But another thing I'd suggest (and maybe try this first) is to use the -blobcopy flag. This will copy directly from S3 to Azure without having to go via your machine, thus not using any of your bandwidth and will probably be a lot quicker for that volume of data.

Please let me know how you get along with it.

Cheers

Ken

[alexlenail]

Hi @kpfaulkner Just to be clear, is this either-or or both? I definitely want the -blobcopy flag it seems, but do I also want the -d flag?

[kpfaulkner]

Correct, its either-or. Both would work but I'd try the -blobcopy flag first.

[alexlenail]

Hi @kpfaulkner !

Progress was being made, I think I transferred some 300MB this time before it crashed, with the following error:

This looks maybe like it reached max retries? What do you think?

[kpfaulkner]

Hmmm can you give me more of that screen shot? (including params).

Also, can you add in the -db flag (if not already).

Thanks

Ken

On Mon, Nov 21, 2016 at 10:34 AM, Alexander Lenail <notifications@github.com

wrote:

Hi @kpfaulkner https://github.com/kpfaulkner !

Progress was being made, I think I transferred some 300MB this time before it crashed, with the following error: [image: screen shot 2016-11-20 at 6 32 54 pm] https://cloud.githubusercontent.com/assets/2761597/20467194/ef3e6206-af4f-11e6-8506-f474b6cd9852.png

This looks maybe like it reached max retries? What do you think?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kpfaulkner/azurecopy/issues/15#issuecomment-261815160, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHGUhvfl1_5HyUPFL_gpmdJGcwa4QA3ks5rANkigaJpZM4K3XDO .

[alexlenail]

Hi @kpfaulkner

Above the image that's shown is just pages and pages of

using blob copy [source] to [dest]

That said, the command that was run was this one:

(found by using the up arrow to show the last command run)

[alexlenail]

Any ideas? It looks like AzureCopy started transferring every single file at once as opposed to doing them sequentially. Maybe either end of the transfer freaked out when AzureCopy started simultaneously moving a couple thousand files? @kpfaulkner

[kpfaulkner]

Hi

Have attached a custom build with some extra debugging (and increased timeouts). The exception you're seeing is in Microsofts code and I can't figure out (yet) whats causing it, but will be interested to see what happens with this new version.

Please let me know how it goes.

Debug.zip

[alexlenail]

The zip you sent me didn't have an azurecopy.exe.Config file, so I copied the one I had into the directory. If that was incorrect let me know. The transfer started, I'll let you know if/once it fails, @kpfaulkner. Thanks for all the support!

[kpfaulkner]

Correct, it didn't have the config file (was thinking you could use the one you had). Sorry, probably should have mentioned that :)

On 23 Nov. 2016, at 04:31, Alexander Lenail notifications@github.com wrote:

The zip you sent me didn't have an azurecopy.exe.Config file, so I copied the one I had into the directory. If that was incorrect let me know. The transfer started, I'll let you know if/once it fails, @kpfaulkner. Thanks for all the support!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[kpfaulkner]

Have created another build with some better features :)

In your app.config can you add:

<!-- blobcopy timeouts -->
<add key="MaxExecutionTimeInMins" value="60" />
<add key="MaxServerTimeoutInMins" value="60" />
<add key="BlobCopyBatchSize" value="5" />

These values are options that we can tinker with connection timeouts. Also I was thinking about your idea about maybe Azure it having a fit at being given a lot of files at once. I've done this with 1000's of blobs at once, but maybe you've hit the limit :) The BlobCopyBatchSize chunks the uploads into XXX number of blobs.

Please let me know if this is any use.

Thanks Debug.zip

[alexlenail]

Curiously, the command completed this time, but I don't think it properly copied the data. If the command completes with the blobcopy flag does that mean it has completed the transfer or completed 'staging' the transfer? (since the computer that executes the command isn't involved in passing the data from one to the other).

[kpfaulkner]

Was that with the first or second binary I uploaded to this thread? Either way, it should have waited until the blobs are complete. The latest version was also more verbose about reporting errors.

If it was the latest binary, did you see the message "New Batch" appearing a bunch of times?

[alexlenail]

It was the first binary. Should I try again with the latest one? @kpfaulkner

[kpfaulkner]

Have you tried the latest version ( https://github.com/kpfaulkner/azurecopy/releases/tag/1.3.3 ) ?