Closed olgabot closed 5 years ago
kislyuk
commented
6 years ago @olgabot sorry I just saw this. Did you sort this out according to our discussion in Slack? In general, apps on EC2 instances are supposed to get their credentials from instance metadata (container hosts also provide this)
olgabot
commented
6 years ago thanks for your response! I haven't figured out how to deal with this yet, since the information I received was that aegea uses profiles but I don't know how to push the ~/.ssh/id_rsa.pub and ~/.aws/config files for every instance, outside of simply scping them
kislyuk
commented
6 years ago OK so you're dealing with two different sets of credentials here.
AWS credentials. When running on an EC2 instance, your application (reflow) should get its AWS credentials from EC2 instance metadata. I know Reflow is capable of ingesting these, but you may need to configure it properly to do so. I think @ryanking mentioned something in Slack about how Reflow fixed something about this recently. (Also, you need to launch the instance with an IAM role - if you don't specify an IAM role, the instance won't have AWS credentials in its metadata. But if you're launching with aegea, your instance should always have an IAM role.)
SSH credentials. What is reflow trying to use the credentials for? I'm not certain that they're required, but if you're sure they are, then my guess is that you'd want to use the AWS Secrets Manager to store the SSH keys and then retrieve them using the AWS credentials in step 1. Something like: cat ~/.ssh/id_rsa.pub | aegea secrets put reflow-ssh-pubkey (on the computer where you've generated the ssh key) followed by aws secretsmanager get-secret-value --secret-id reflow-ssh-pubkey > ~/.ssh/id_rsa.pub (on the instance that wants to use the key).
olgabot
commented
5 years ago Wow 4 months later and I'm still having the same problems... such is programming!
Here is how I launched my instance:
aegea launch --iam-role S3fromEC2 --ami-tags Name=czbiohub-reflow -t t2.micro olgabot-reflowFor AWS credentials, there's an option to export AWS_SDK_LOAD_CONFIG=1, but that still looks for the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.
The credentials are added to the EC2 instances created by Reflow so you can the log onto them and inspect the outputs. The aegea secrets command worked just fine but aws secretsmanager doesn't seem to be an option for me (details below)... How did the aws in /usr/bin/aws get there? Was that added by aegea?
olgabot
commented
5 years ago Here's the IAM role info:
Wed 21 Nov - 18:32 ~
aws iam get-role --role-name S3fromEC2
{
"Role": {
"Path": "/",
"RoleName": "S3fromEC2",
"RoleId": "redacted",
"Arn": "arn:aws:iam::redacted:role/S3fromEC2",
"CreateDate": "2017-12-06T00:22:48Z",
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
},
"Description": "Provides access to S3 buckets from an EC2 instance.",
"MaxSessionDuration": 3600
}
}@olgabot It sounds like perhaps reflow is unable to understand how to fetch credentials from instance metadata according to AWS conventions, or it's not being configured to do so.
For AWS credentials, there's an option to export AWS_SDK_LOAD_CONFIG=1, but that still looks for the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.
Requiring these variables to be present is generally incorrect behavior. In the absence of these environment variables, reflow should check the instance metadata (https://docs.aws.amazon.com/cli/latest/topic/config-vars.html#id1).
Regarding the version of the AWS CLI in your AMI: how did you build the czbiohub-reflow AMI? Did you use aegea image or something else? In either case, you are right to suspect that it's out of date.
kislyuk
commented
5 years ago @olgabot reading through the reflow docs, it appears reflow is basically not smart enough to integrate with AWS IAM roles, and instead requires long-lived static credentials (an AWS access key associated with an IAM user).
In this situation, you do need to set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables. I advise using aegea secrets to manage those, if you have to.
Hi @kislyuk! I'm running Reflow on aegea-launched EC2 instances and am having trouble getting Reflow to recognize the AWS credentials. I know they're there because I'm able to
aws s3 syncto the buckets I have access to.In the end, I
scp-d over the credentials to get it to work:But I'm wondering if I'm missing something and it's easier to do this already with Aegea.
Here's my home directory and environment variables: