Hooking up to a user's S3 bucket for output

[scripting]

Suppose I wanted to add a feature to one of my Electron apps that allowed a user to hook it up to an S3 location that they own and pay for.

What's the best way to provide access?

I wonder that's what IAM is for. Can it be used to authorize an app? It seems to work with OAuth. Is that true?

I need to dig in. If you know of any easy howto's please post a note here.

[antmuzic]

The IAM strategy to solve this problem would be:

User creates an IAM identity in their account and grants the proper access to that identity. User then generates a secret key and access key and shares those with your app. Your app then uses those credentials to write to the user's bucket. The nice thing is that these credentials only have access to the specified bucket.

This is a very old howto I found that still holds up: http://mikeferrier.com/2011/10/27/granting-access-to-a-single-s3-bucket-using-amazon-iam/

Using this howto, the user would share the secret key and access key instead of username and password.

I've seen this strategy employed by various apps. What comes to mind are things like wordpress backup to s3 plugins.

The other strategy for the user to give specific permissions to your app on their bucket. This can be granted by canonical user id (http://docs.aws.amazon.com/general/latest/gr/acct-identifiers.html) or by the email address on your account. That would get done in the user's aws console under s3 by changing permissions on the bucket. (http://docs.aws.amazon.com/AmazonS3/latest/UG/EditingBucketPermissions.html)

[scripting]

Sorry for not being more clear in this writeup.

Imagine how a user authorizes one of my apps to access their Twitter account.

That's the level of complexity I'm looking for. Something one of those users can do.

The features you describe (which I did know about) are the kinds of things you would ask a developer to do.

I'm looking to create tools for writers. Wondering if after all these years Amazon made this possible. There are so many docs to read -- I was hoping that someone who was more up on this stuff could help me find the new bit that makes this possible (if it exists).

[scripting]

BTW, it seems to me it pretty much has to be OAuth.

[antmuzic]

AWS is not really there yet for users looking for that level of complexity (I don't know that it'll ever be there).

There might be a Oauth path using IAM, but the level of complexity is going to be even worse than what I described. (Create users, groups, policies, set up oAuth on the account.) Now that I'm thinking about it, I've only ever seen AWS using oAuth to authenticate against an external source, not the other way around.

You could just ask for the access-key/secret-key with access to a bucket. Basic users would provide the root access-key/secret-key. More advanced users would know how to create a user with only write access to the specified bucket. Access-key/secret-key is better than the the userid/password for the account, but what you describe (an oAuth path) would be a ton better from both a security and an ease-of-use perspective.

[scripting]

Thanks for the update. I check in every few years to see if they got this obvious connection going. I suppose I could implement it myself and resell their storage, but I'm a programmer not a financier, and it's one of those kinds of things that as soon as I got it off the ground they'd copy it and they'd be right to. It's really their job.

On Sat, Aug 5, 2017 at 1:39 PM, antmuzic notifications@github.com wrote:

AWS is not really there yet for users looking for that level of complexity (I don't know that it'll ever be there).

There might be a Oauth path using IAM, but the level of complexity is going to be even worse than what I described. (Create users, groups, policies, set up oAuth on the account.) Now that I'm thinking about it, I've only ever seen AWS using oAuth to authenticate against an external source, not the other way around.

You could just ask for the access-key/secret-key with access to a bucket. Basic users would provide the root access-key/secret-key. More advanced users would know how to create a user with only write access to the specified bucket. Access-key/secret-key is better than the the userid/password for the account, but what you describe (an oAuth path) would be a ton better from both a security and an ease-of-use perspective.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/scripting/Scripting-News/issues/19#issuecomment-320457412, or mute the thread https://github.com/notifications/unsubscribe-auth/ABm9O14KIDWY6mfQ02hZMeRCQjutp3nuks5sVKi9gaJpZM4OuC99 .

[lisawilliams]

Calling in @TanyaSelvog on this one. Tanya runs an AWS meetup in Austin, TX and is doing a series called 100 Days of AWS. (In other news, I wonder if @-ing someone works here).

[scripting]

Okay I decided to dive into IAM permissions and see if I could get a writing tool to hook up to S3 at least in a way that programmers and adventurous users could figure it out.

But a simple Hello World program that uses the credentials gets permission denied so there's something wrong in the setup. Here's the data.

1. This is the spec I used to create the permissions.

2. It seems to be interpreted correctly by IAM.

3. The test app appears to be loading the credentials properly. I have two sets of credentials in my credentials file. One for me and one for the fictitious new user. The test app works when the credentials are mine, and fails with Access Denied when I use the other creds.

4. BTW, here's the howto I prepared. Maybe you can spot the problem in there? I won't release the howto until my code works. ;-)

Any help much appreciated. ;-)

[benzado]

I looked over the spec, etc. and I can't see a problem. :-(

Your howto recommends creating a managed policy, attaching the policy to a group, then adding a user to the group... that's The Right Way if you are managing a lot of users/groups/policies/etc. but if you want to keep it simple, you can add a policy directly to the user. The steps would be

1. Create user
2. On user's Permissions tab, click "Add inline policy"

Who knows, maybe that will magically fix it?

[scripting]

Interesting. I tried to attach a policy to a user and couldn't figure out what the steps are. And I know that sometimes AWS responds to those kinds of do-overs. Back in a few minutes. ;-)

[scripting]

It didn't work.

[FTWynn]

Dave, I don't know if you're open to Google Cloud Platform instead of Amazon (disclosure, I work in Mountain View), but there are two options down that path:

Use Application storage on Google Drive

This technically uses Drive space instead of a Cloud bucket, but it would still be owned by the user. The only drawback is that the application folder data isn't viewable in the Drive UI, so you'd need to build an export function into the app.

Documentation for App folders

Use a Google Cloud Storage Bucket

This is basically the same as the S3 approach, but since Google Cloud is accessed via Google accounts (gmail, apps for domain, emails just registered with this link), you don't have to reimplement OAuth yourself.

You can then interact with the REST API to create buckets and get/put objects.

https://cloud.google.com/storage/docs/creating-buckets

In either case, you might get more of what you want since OAuth is already tied into the identity mechanism, writers are more likely to already have a Google account, and it works the same way as Dropbox and/or S3.

[scripting]

I'm always open to new ideas esp if they solve this problem that's been around so long! ;-)

The files have to be accessible over HTTP at a domain the user controls. This is a publishing application. And the user has to have full freedom to move.

Pretty sure Google Drive can't do that (we've been down this road before) but I am not familiar with Google Cloud Storage Bucket.

On Wed, Aug 9, 2017 at 7:43 PM, David Wynn notifications@github.com wrote:

Dave, I don't know if you're open to Google Cloud Platform instead of Amazon (disclosure, I work there), but there are two options down that path: Use Application storage on Google Drive

This technically uses Drive space instead of a Cloud bucket, but it would still be owned by the user. The only drawback is that the application folder data isn't viewable in the Drive UI, so you'd need to build an export function into the app.

Documentation for App folders https://developers.google.com/drive/v3/web/appdata Use a Google Cloud Storage Bucket

This is basically the same as the S3 approach, but since Google Cloud is accessed via Google accounts (gmail, apps for domain, emails just registered with this link https://accounts.google.com/SignUpWithoutGmail?), you don't have to reimplement OAuth yourself.

You can then interact with the REST API to create buckets and get/put objects.

https://cloud.google.com/storage/docs/creating-buckets

In either case, you might get more of what you want since OAuth is already tied into the identity mechanism, writers are more likely to already have a Google account, and it works the same way as Dropbox and/or S3.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/scripting/Scripting-News/issues/19#issuecomment-321411097, or mute the thread https://github.com/notifications/unsubscribe-auth/ABm9OxAYjmXBfeXKiy6DGDkn9rxqPltRks5sWkQugaJpZM4OuC99 .

[FTWynn]

The files have to be accessible over HTTP at a domain the user controls. This is a publishing application. And the user has to have full freedom to move.

Definitely Storage Bucket then.

I just started building a static site out of one of my buckets. All it takes is a CNAME in the DNS record. Everything else should be pretty close to S3 API semantics.

https://cloud.google.com/storage/docs/hosting-static-website

My terrible website for embarrassment purposes

[scripting]

Okay then, let's have a look at making this work.

It'll take me a couple of days probably to swing back around to it. Will you be getting cc's of posts on this thread? I'll probably have questions if you don't mind helping out.

Dave

On Wed, Aug 9, 2017 at 8:01 PM, David Wynn notifications@github.com wrote:

The files have to be accessible over HTTP at a domain the user controls. This is a publishing application. And the user has to have full freedom to move.

Definitely Storage Bucket then.

I just started building a static site out of one of my buckets. All it takes is a CNAME in the DNS record. Everything else should be pretty close to S3 API semantics.

https://cloud.google.com/storage/docs/hosting-static-website

My terrible website for embarrassment purposes http://www.ftwynn.com

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/scripting/Scripting-News/issues/19#issuecomment-321413565, or mute the thread https://github.com/notifications/unsubscribe-auth/ABm9O3Dk40CfQop0JyVVTn6yEhBszGNmks5sWkhCgaJpZM4OuC99 .

[FTWynn]

Yep. Will do.

[scripting]

I created a bucket and uploaded a file.

https://storage.googleapis.com/yachty-post-5874/earth.png

Next I have to figure out how to assign a name to that location, something like

http://goog.bloatware.org/

I'll be back in a bit..

[scripting]

I found the doc I needed.

https://cloud.google.com/storage/docs/hosting-static-website

Dave

[scripting]

Well I got stuck trying to verify the CNAME I want to use which is test.bloatware.org.

I couldn't decypher this --

I also couldn't figure out what they're asking me to do with the TXT record.

If you could help me figure out what they're asking me to do, I will do it and then get on to the next step.

[scripting]

BTW, I can tell right now this isn't the answer. It sounds like they may have all the pieces, but the UI for creating a bucket doesn't need to be any more complicated than creating a GMail account. Or even a new Google Docs spreadsheet. People think this kind of stuff has to be incomprehensible, but really it doesn't.

Re verification --

1. Amazon doesn't make you jump through this hoop for S3.

2. A simpler way to do it would be to tell you to point the CNAME you want to use to some name which points to the Google hosting server. So once you've done this that's all you have to do. This does not need to be a multi-step process, if it's needed at all.

Dave

[dubsquared]

Howdy, Dave! Just to clarify, are you trying to connect your app to S3 so you can provide a feature to a blob object that is hosted there? Trying to fully understand the use case and ask. We do a lot of work integrating with cloud drives, so this is very interesting to me. Thanks!

[FTWynn]

I agree there could be a better flow for this. It's definitely built for engineers, not so much poets.

It's asking you to verify your domain so you can just use a CNAME to address the bucket as listed here: https://cloud.google.com/storage/docs/xml-api/reference-uris#cname . One thing to note is you'll need to name the bucket the same as the domain, bloatware.com in this case.

You shouldn't need the TXT record with the CNAME verification.

[scripting]

Have you tried creating a bucket with a cname? If so, at the verification step, what exactly did you do? Their instructions were incomplete. I guessed what they were asking me to do, but they weren't happy.

I wanted to name the bucket test.bloatware.org, btw, not bloatware.org.

On Saturday, August 12, 2017, David Wynn notifications@github.com wrote:

I agree there could be a better flow for this. It's definitely built for engineers, not so much poets.

It's asking you to verify your domain so you can just use a CNAME to address the bucket as listed here: https://cloud.google.com/ storage/docs/xml-api/reference-uris#cname . One thing to note is you'll need to name the bucket the same as the domain, bloatware.com in this case.

You shouldn't need the TXT record with the CNAME verification.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/scripting/Scripting-News/issues/19#issuecomment-321956926, or mute the thread https://github.com/notifications/unsubscribe-auth/ABm9O-jy-MfrfNiUkR7BlxaKJQio_Zn7ks5sXSpxgaJpZM4OuC99 .

-- Typed on an iPad with fat fingers.

[FTWynn]

You should be able to name the bucket test.bloatware.org, I believe. There shouldn't be an issue there.

The whole point of the domain registration process is to get the domain to show up on this page for you:

https://www.google.com/webmasters/verification/home?hl=en

Once it's there, that should be it. If the CNAME route isn't working, I would try the TXT record method.

[scripting]

When you try it, let me know what you did. I did try it and as I said the instructions are incomplete.

On Sat, Aug 12, 2017 at 4:17 PM, David Wynn notifications@github.com wrote:

You should be able to name the bucket test.bloatware.org, I believe. There shouldn't be an issue there.

The whole point of the domain registration process is to get the domain to show up on this page for you:

https://www.google.com/webmasters/verification/home?hl=en

Once it's there, that should be it. If the CNAME route isn't working, I would try the TXT record method.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/scripting/Scripting-News/issues/19#issuecomment-322003439, or mute the thread https://github.com/notifications/unsubscribe-auth/ABm9O-PWTkFojUC8WRHdcwNf6HRkM2FVks5sXghwgaJpZM4OuC99 .

[scripting]

To @dubsquared -- here's the use-case fully explained..

https://github.com/scripting/Scripting-News/issues/21

Dave

[windley]

AWS Cognito allows users accessing S3, but not, as far as I can tell, in their own account.

The disconnect seems to be that creating an S3 bucket is meant for developers, not random Amazon users. As you point out, Dave, that doesn't have to be, but it seems to be the case.

[scripting]

With all the other things Amazon sells, why not sell a virtual bucket??

https://www.amazon.com/s/ref=nb_sb_noss_1?url=search-alias%3Daps&field-keywords=bucket

Dave

On Tue, Sep 5, 2017 at 3:32 PM, Phil Windley notifications@github.com wrote:

AWS Cognito allows users accessing S3 http://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_examples_s3_cognito-bucket.html, but not, as far as I can tell, in their own account.

The disconnect seems to be that creating an S3 bucket is meant for developers, not random Amazon users. As you point out, Dave, that doesn't have to be, but it seems to be the case.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/scripting/Scripting-News/issues/19#issuecomment-327279371, or mute the thread https://github.com/notifications/unsubscribe-auth/ABm9OyjFXulDXgh3CuEN63n2VHON_3nKks5sfaHGgaJpZM4OuC99 .