AWS SSO based connections are too difficult to use

michael-crawford commented 2 years ago

I stopped using Cyberduck a couple years ago as multi-account became increasingly common, and I moved to use of aws-vault to store credentials in the MacOS or Windows keychain or use AWS SSO (pre-control tower), neither of which worked with Cyberduck, making it increasingly difficult to use without keeping a duplicate set of credentials.

I decided to try it again today, and noticed the new credentials option to obtain credentials from ~/.aws/config, where I have a bunch of SSO based credentials setup for many accounts organized via AWS Control Tower. For the first account, I added the bookmark, specified the profile name (based on SSO), and then double-clicked to open - nothing.

I figured, just as with the command line, I needed to first run "aws sso login --profile $profile", which I did, then also confirmed on the same CLI I could see data using a simple command like "aws ec2 describe-regions --profile $profile", so when I went back into Cyberduck and then clicked on the bookmark, it worked. I was glad to see this appeared to make Cyberduck usable again for large-scale / enterprise-level AWS multi-account scenarios!

So, I then went in and setup a bunch of other bookmarks based on other SSO profiles. But when I next tried to double-click these profiles, they would all hang. I went into the CLI, confirmed the profile worked, then once at least one CLI command was run against the profile (note I was never prompted for another aws sso login command on the CLI once the first one was done) on the CLI, Cyberduck would then work against that profile. OK, this is not good, I may have dozens of accounts, so this is going to make this so inconvenient again, just like before, to where the cons outweigh the pros...

I Googled your docs and found a reference https://docs.cyberduck.io/protocols/s3/ where it indicates both the aws sso login and at least one command like aws sts get-caller-identity needs to be run to get the STS tokens you need into cache for use. So, at least with the current design, I see what's happening.

The problem is, this effectively makes Cyberduck difficult to use, so much so I'm unlikely to want to use it much still. So, I thought I'd suggest a couple of improvements which you could make to reduce developer friction on use.

[ ] When I setup the bookmark, using the "S3 (Credentials from AWS Command Line Interface)" type, you have the name of the profile in the ~/.aws/config file, which should be readable by your program, so you can look into that profile and detect that it's based on SSO. Why do I have to run the "aws sts get-caller-identity --profile $profile"? Why can't you run this for me, some exec method to run a shell command as the user?
[ ] Same as (1), if you detect that this is an SSO profile, if you try to run (1), and that fails with an error code indicating the user has not logged in yet, why can't you run the "aws sso login --profile $profile" command for me? This does interact with the browser, not sure if that complicates things, and if so this does only need to be run once, so it's less onerous, but if you can't run it, at least popping up a dialog box saying "Open a terminal and run 'aws sso login --profile $profile', then retry" or similar.

The problem is (1) when you have a large number of accounts. It makes this program less usable than it should/could be. running a couple of shell commands when you have every parameter of data needed to run them from the configuration you collect would seem to be trivial, but would result in significant ease-of-use improvements. If you can't do this, at least explain why, and use dialog boxes to indicate to the user what they must do, so we don't have to guess or search your documentation for why this doesn't "just work".

dkocher commented 2 years ago

We should be able to implement this properly without the command line usage requirement with #13381.

michaelcox80-zz commented 2 years ago

We are hoping to move our Cyberduck users to SSO.

[profile main]
sso_start_url = https://xxx.awsapps.com/start#/
sso_region = eu-west-1
sso_account_id = xxxxxxx
sso_role_name = DevAccess
region = eu-west-1
output = json
[profile xxx_sandbox]
role_arn = arn:aws:iam::xxx:role/sbx_developer_limited_access
source_profile = main
[profile xxx_dev]
role_arn = arn:aws:iam::xxx:role/dev_developer_limited_access
source_profile = main

aws sso login --profile main
aws sts get-caller-identity --profile main

After authenticating via aws sso, all AWS commands work.

Cyberduck doesn't load with the profile named, we get the following in the logs:

DEBUG ch.cyberduck.core.sts.STSCredentialsConfigurator - Look for profile name xxx_sandbox in Local{path='/Users/xxx/.aws/config'} and Local{path='/Users/xxx/.aws/credentials'}
DEBUG ch.cyberduck.core.sts.STSCredentialsConfigurator - Reading AWS file Local{path='/Users/xxx/.aws/config'
DEBUG ch.cyberduck.core.sts.STSCredentialsConfigurator - Found matching profile xxx_sandbox
DEBUG ch.cyberduck.core.sts.STSCredentialsConfigurator - Configure credentials from role based profile xxx_sandbox
DEBUG ch.cyberduck.core.sts.STSCredentialsConfigurator - Request {RoleArn: arn:aws:iam::xxx:role/sbx_dev_limited_access,RoleSessionName: appwkddg,} from com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient@8333da
DEBUG com.amazonaws.request - Sending Request: POST https://sts.amazonaws.com / Parameters: ({"Action":["AssumeRole"],"Version":["2011-06-15"],"RoleArn":["arn:aws:iam::xxx:role/sbx_dev_limited_access"],"RoleSessionName":["appwkddg"]}Headers: (amz-sdk-invocation-id: ebdd27e8-dd3c-5e19-5d20-75a155dfe1f9, User-Agent: Cyberduck/8.4.3.38269 (Mac OS X/12.5.1) (x86_64), aws-sdk-java/1.12.135 Mac_OS_X/12.5.1 OpenJDK_64-Bit_Server_VM/17.0.1+12 java/17.0.1 kotlin/1.2.71 vendor/Eclipse_Adoptium cfg/retry-mode/legacy, ) 
WARN  ch.cyberduck.core.threading.BackgroundCallable - Failure java.lang.IllegalArgumentException: Access key cannot be null

dkocher commented 2 years ago

We are hoping to move our Cyberduck users to SSO.

In #13725.

blytheaw commented 1 year ago

We need this capability in order to use Cyberduck at our org

michaelcox80 commented 1 year ago

This has been resolved in the latest update (8.5.6), we tested this a couple of days ago.

dkocher commented 1 year ago

This has been resolved in the latest update (8-5-6), we tested this a couple of days ago.

I assume you refer to #13725.

dkocher commented 1 year ago

To allow connecting without the need for configuration using AWS CLI we are dependent on proper OIDC support in AWS SSO.

The IAM Identity Center OIDC service currently implements only the portions of the OAuth 2.0 Device Authorization Grant standard (https://tools.ietf.org/html/rfc8628) that are necessary to enable single sign-on authentication with the AWS CLI. Support for other OIDC flows frequently needed for native applications, such as Authorization Code Flow (+ PKCE), will be addressed in future releases. ^1

michael-crawford commented 1 year ago

David,

It’s been so long since I opened this, I’m forgetting the context now, and this answer isn’t very helpful on its own.

Your point about “without the need for configuration using AWS CLI” is what’s confusing me – I’d be FINE with Cyberduck being able to use my existing AWS CLI SSO configuration syntax, instead of having to collect and use another set of configuration data which would then go through OIDC.

I’ve been working mainly with Azure the last year, and before that, due to this issue with Cyberduck and our use of SSO for all credentials, I pretty much stopped using Cyberduck since I couldn’t use it without having to create additional IAM credentials I didn’t want to create strictly for that purpose.

What I want is simple: How can I use Cyberduck when I’m not using IAM credentials, but instead have configured SSO, and the AWS CLI to go through SSO?

You can assume I have a working AWS CLI configuration that works with SSO and gives me access to all the Accounts with appropriate S3 bucket permissions via the CLI – the question is, can Cyberduck use that AWS CLI configuration in a way similar to aws s3 … commands? Like I’d just reference --profile to reference the right profile?

What you’re describing, from an outsider’s perspective, seems like a red herring to me, but I’m not inside your code.

Mike

michaelcox80 commented 1 year ago

@michael-crawford, this is how we use Cyberduck.

We have multiple AWS accounts, our Cyberduck users have an AWS config files with the necessary role to assume in each account.

They run the following:

aws sso login --profile xxxx. 
aws sts get-caller-identity --profile xxxxx

Cyberduck is configured to look at the AWS config file.

dkocher commented 1 year ago

@michael-crawford, this is how we use Cyberduck.

We have multiple AWS accounts, our Cyberduck users have an AWS config files with the necessary role to assume in each account.

They run the following:
aws sso login --profile xxxx. 
aws sts get-caller-identity --profile xxxxx
Cyberduck is configured to look at the AWS config file.

This is supported using the S3 (Credentials from AWS Command Line Interface) profile ^1 connection profile but requires users to manually run aws sts get-caller-identity --profile … periodically in Terminal.

michaelcox80 commented 1 year ago

True, but session durations can be configured to last between 1-12 hours.

blytheaw commented 1 year ago

@michael-crawford, this is how we use Cyberduck. We have multiple AWS accounts, our Cyberduck users have an AWS config files with the necessary role to assume in each account. They run the following:
aws sso login --profile xxxx. 
aws sts get-caller-identity --profile xxxxx
Cyberduck is configured to look at the AWS config file.
This is supported using the S3 (Credentials from AWS Command Line Interface) profile 1 connection profile but requires users to manually run aws sts get-caller-identity --profile … periodically in Terminal.

Footnotes
1. https://docs.cyberduck.io/protocols/s3/#connecting-using-credentials-from-aws-command-line-interface [↩](#user-content-fnref-1-a52c168e113ad21aa283cb5e58d22f03)

This is what we have been doing for a while, and for technical users who understand AWS CLI, profiles, SSO, etc. it works well enough. But we are working with a growing number of internal users who are not technical and this is a lot to ask of them to do in order to browse files.

I don't fully understand the technical limitations in this project, but I do know many other tools using AWS SDK seem to be able to do this seamlessly without requiring a manual refresh of SSO sessions from the CLI.

I love the tool and it is 99% there for our use cases. This would be a huge win to be able to setup Cyberduck with AWS SSO once for all of our users and not have them bother with running AWS CLI commands or opening terminals.

DM1145 commented 1 year ago

I agree with this - if this is needed for these types of profile to work, then it should be a different type of S3 connection, and include that recurring necessity (token refresh via "aws sts get-caller-identity profile=" so users shouldn't have to separately do that.

I wouldn't consider this working until that is done. At present, this is a temporary [but annoying] workaround.

dkocher commented 1 year ago

I agree with this - if this is needed for these types of profile to work, then it should be a different type of S3 connection, and include that recurring necessity (token refresh via "aws sts get-caller-identity profile=" so users shouldn't have to separately do that.

I wouldn't consider this working until that is done. At present, this is a temporary [but annoying] workaround.

I assume one would need to find a way that we can obtain new temporary access tokens from AWS STS using cached SSO tokens from AWS CLI.

DM1145 commented 1 year ago

I don't think it makes sense for the application to spawn a login process (the user should make sure the credentials are active).

If the connection fails, couldn't the application internally execute this command shell script (once) as an un-windowed task? If it's configured right and the user has made sure the credentials are active, this should work. If it doesn't, then the application can then prompt that the credentials aren't active and the user needs to log in before using the app.

O a second, similar note, is the communication between the app and the storage provider visible anywhere (especially in case of failure)? The message received from the application before I ran the shell script "aws sts get-caller-identity profile=" didn't point me in the right direction. While I was lucky I found this ticket and was able to workaround, I think if I could see the transaction log I might have had a better idea how to correct this.

rub-arulprashanth commented 11 months ago

We have folks using S3 Browser in Windows and that works seamless and this is what the setup looks like

We don't need use of CLI, can Cyber do that similar to that. am using the CLI option but this will be lot simpler, whenever the session timesout the browser pops open asking to authenticate.

iterate-ch / cyberduck

AWS SSO based connections are too difficult to use #13377

Footnotes