Problem Listing AWS S3

[nightwallaby]

Hi,

I am using v1.1.3 I am unable to perform any function against AWS S3. Here is the stack trace trying to list the contents of a bucket.

PS C:\admin\azurecopy-1.1.3\azurecopy> .\azurecopy.exe -list https://rw-static.s3.amazonaws.com/ -db -v GetHandler start GetHandler retrieved azurecopy.S3Handler container name rw-static Unknown error generated. Please report to Github page https://github.com/kpfaulkner/azurecopy/issues . Can view underlying stacktrace by adding -db flag. at System.Collections.Generic.Dictionary`2.get_Item(TKey key) at azurecopy.Utils.S3Helper.GenerateS3Client(String accessKey, String secretKey, String bucketName) at azurecopy.S3Handler.ListBlobsInContainer(String containerName, String blobPrefix, Boolean debug) at azurecopycommand.Program.DoList(Boolean debugMode) at azurecopycommand.Program.Process(Boolean debugMode) at azurecopycommand.Program.Main(String[] args)

Any help with this issue would be appreciated.

Cheers, Simon

[kpfaulkner]

Hi Simon,

Which region is your S3 account setup for?

Thanks

Ken

On Mon, Jan 18, 2016 at 2:43 PM, Simon Horn notifications@github.com wrote:

Hi,

I am using v113 I am unable to perform any function against AWS S3 Here is the stack trace trying to list the contents of a bucket

PS C:\admin\azurecopy-113\azurecopy> \azurecopyexe -list https://rw-statics3amazonawscom/ -db -v GetHandler start GetHandler retrieved azurecopyS3Handler container name rw-static Unknown error generated Please report to Github page https://githubcom/kpfaulkner/azurecopy/issues Can view underlying stacktrace by adding -db flag at SystemCollectionsGenericDictionary`2get_Item(TKey key) at azurecopyUtilsS3HelperGenerateS3Client(String accessKey, String secretKey, String bucketName) at azurecopyS3HandlerListBlobsInContainer(String containerName, String blobPrefix, Boolean debug) at azurecopycommandProgramDoList(Boolean debugMode) at azurecopycommandProgramProcess(Boolean debugMode) at azurecopycommandProgramMain(String[] args)

Any help with this issue would be appreciated

Cheers, Simon

— Reply to this email directly or view it on GitHub https://github.com/kpfaulkner/azurecopy/issues/6.

[nightwallaby]

Hi Ken,

My region is ap-southeast-2, and the azurecopy.exe.Config file has this configuration in place.

Cheers, Simon

[kpfaulkner]

Hi Simon,

I see the problem. I think you must be the first person to try AzureCopy and use that region. Typo in my code I'm afraid. I'll build and release a patch release tonight and will let you know when it's available. Sorry for the issue.

Thanks

Ken

On Mon, Jan 18, 2016 at 3:11 PM, Simon Horn notifications@github.com wrote:

Hi Ken,

My region is ap-southeast-2, and the azurecopy.exe.Config file has this configuration in place.

Cheers, Simon

— Reply to this email directly or view it on GitHub https://github.com/kpfaulkner/azurecopy/issues/6#issuecomment-172420241.

[nightwallaby]

Hi Ken,

That's cool. Thanks for the quick response!

Cheers, Simon

On 18 January 2016 at 14:28, Ken Faulkner notifications@github.com wrote:

Hi Simon,

I see the problem. I think you must be the first person to try AzureCopy and use that region. Typo in my code I'm afraid. I'll build and release a patch release tonight and will let you know when it's available. Sorry for the issue.

Thanks

Ken

On Mon, Jan 18, 2016 at 3:11 PM, Simon Horn notifications@github.com wrote:

Hi Ken,

My region is ap-southeast-2, and the azurecopy.exe.Config file has this configuration in place.

Cheers, Simon

— Reply to this email directly or view it on GitHub <https://github.com/kpfaulkner/azurecopy/issues/6#issuecomment-172420241 .

— Reply to this email directly or view it on GitHub https://github.com/kpfaulkner/azurecopy/issues/6#issuecomment-172421445.

[kpfaulkner]

Have created a pre-release 1.1.5. URL is https://github.com/kpfaulkner/azurecopy/releases/tag/1.1.5

I've given it a quick test and it works for me. Please let me know how it goes for you.

Thanks

Ken

[nightwallaby]

That seems to be working now. Thank you very much.

I do however have a question about my particular use case. I currently have ~1.5 TB of images in an S3 bucket that I wish to copy to Azure for backup purposes. While I understand that folders don't strictly exist in S3 (the full folder/file path is actually just the object name) I have the files organised in 'folders' like

/////. It's everything underneath FOLDERNAME that I wish to copy. If I run azurecopy like: azurecopy.exe -i https://media.s3.amazonaws.com/photos/ -o https://media.blob.core.windows.net/photos ...will azurecopy enumerate all of the ~1.5 TB worth of files before actually starting the copy (be it with direct or blobcopy)? If I stop the copy and restart it, will it recopy files across that are already there? Cheers, Simon On 18 January 2016 at 14:57, Ken Faulkner notifications@github.com wrote: > Have created a pre-release 1.1.5. > URL is https://github.com/kpfaulkner/azurecopy/releases/tag/1.1.5 > > I've given it a quick test and it works for me. Please let me know how it > goes for you. > > Thanks > > Ken > > — > Reply to this email directly or view it on GitHub > https://github.com/kpfaulkner/azurecopy/issues/6#issuecomment-172425892.

[kpfaulkner]

Hi Simon,

Azurecopy will use the same "virtual directories" in Azure as it does in S3. ie if you have S3 bucket "media" and in that you have a blob called "foo/2005/12/1/myblob" it can copy that entire structure over to Azure so you'll preserve the "virtual dirs".

AzureCopy will traverse all the blobs (basically just getting a list, which should be pretty quick) then will start copying one by one. You can use the -blobcopy flag so it goes directly from Amazons DC to Microsofts DC without using any of your personal bandwidth.

eg. azurecopy.exe -i https://mybucket.s3.amazonaws.com/mydir/ -o https://myacct.blob.core.windows.net/mycontainer/ -blobcopy

Azurecopy doesn't check if blobs already exist at the destination, so if you interrupt the copying then start it again it will copy everything from scratch (I might be able to add a check in at some stage).

Hope it's useful!

Cheers

Ken

On Tue, Jan 19, 2016 at 8:51 AM, Simon Horn notifications@github.com wrote:

That seems to be working now. Thank you very much.

I do however have a question about my particular use case. I currently have ~1.5 TB of images in an S3 bucket that I wish to copy to Azure for backup purposes. While I understand that folders don't strictly exist in S3 (the full folder/file path is actually just the object name) I have the files organised in 'folders' like

/////. It's everything underneath FOLDERNAME that I wish to copy. If I run azurecopy like: azurecopy.exe -i https://media.s3.amazonaws.com/photos/ -o https://media.blob.core.windows.net/photos ...will azurecopy enumerate all of the ~1.5 TB worth of files before actually starting the copy (be it with direct or blobcopy)? If I stop the copy and restart it, will it recopy files across that are already there? Cheers, Simon On 18 January 2016 at 14:57, Ken Faulkner notifications@github.com wrote: > Have created a pre-release 1.1.5. > URL is https://github.com/kpfaulkner/azurecopy/releases/tag/1.1.5 > > I've given it a quick test and it works for me. Please let me know how it > goes for you. > > Thanks > > Ken > > — > Reply to this email directly or view it on GitHub > . — Reply to this email directly or view it on GitHub https://github.com/kpfaulkner/azurecopy/issues/6#issuecomment-172663704.

[nightwallaby]

Ok cool. Thank you very much Ken.

On 19 January 2016 at 08:05, Ken Faulkner notifications@github.com wrote:

Hi Simon,

Azurecopy will use the same "virtual directories" in Azure as it does in S3. ie if you have S3 bucket "media" and in that you have a blob called "foo/2005/12/1/myblob" it can copy that entire structure over to Azure so you'll preserve the "virtual dirs".

AzureCopy will traverse all the blobs (basically just getting a list, which should be pretty quick) then will start copying one by one. You can use the -blobcopy flag so it goes directly from Amazons DC to Microsofts DC without using any of your personal bandwidth.

eg. azurecopy.exe -i https://mybucket.s3.amazonaws.com/mydir/ -o https://myacct.blob.core.windows.net/mycontainer/ -blobcopy

Azurecopy doesn't check if blobs already exist at the destination, so if you interrupt the copying then start it again it will copy everything from scratch (I might be able to add a check in at some stage).

Hope it's useful!

Cheers

Ken

On Tue, Jan 19, 2016 at 8:51 AM, Simon Horn notifications@github.com wrote:

That seems to be working now. Thank you very much.

I do however have a question about my particular use case. I currently have ~1.5 TB of images in an S3 bucket that I wish to copy to Azure for backup purposes. While I understand that folders don't strictly exist in S3 (the full folder/file path is actually just the object name) I have the files organised in 'folders' like

/////. It's everything underneath FOLDERNAME that I wish to copy. If I run azurecopy like: azurecopy.exe -i https://media.s3.amazonaws.com/photos/ -o https://media.blob.core.windows.net/photos ...will azurecopy enumerate all of the ~1.5 TB worth of files before actually starting the copy (be it with direct or blobcopy)? If I stop the copy and restart it, will it recopy files across that are already there? Cheers, Simon On 18 January 2016 at 14:57, Ken Faulkner notifications@github.com wrote: > Have created a pre-release 1.1.5. > URL is https://github.com/kpfaulkner/azurecopy/releases/tag/1.1.5 > > I've given it a quick test and it works for me. Please let me know how > it > goes for you. > > Thanks > > Ken > > — > Reply to this email directly or view it on GitHub > < > https://github.com/kpfaulkner/azurecopy/issues/6#issuecomment-172425892 > . — Reply to this email directly or view it on GitHub

— Reply to this email directly or view it on GitHub https://github.com/kpfaulkner/azurecopy/issues/6#issuecomment-172666921.