NoCredentialProviders: no valid providers in chain

[Smana]

Hello guys,

I'm currently trying kops and i'm not able to use a profile listed in ~/.aws directory. I already had a look to a similar past issue https://github.com/kubernetes/kops/issues/855. But my credentials file is not a symlink.

running the aws cli works fine

aws s3 ls s3://kops-${mydomain} && echo $?
0

But not with kops

kops create cluster --cloud=aws --zones=eu-west1-a cluster0.${mydomain}
error reading cluster configuration "cluster0.${mydomain}": error reading configuration file s3://kops-${mydomain}/cluster0.${mydomain}/config: error getting location for S3 bucket "kops-${mydomain}": NoCredentialProviders: no valid providers in chain. Deprecated.

env | egrep '(AWS_PROFILE|KOPS)'
AWS_PROFILE=${myprofile}
KOPS_STATE_STORE=s3://kops-${domain}

How can i solve that please ?

[krisnova]

What does your aws config look like?

I would be willing wager that your aws config is default to some region, while kops defaults to us-east-1 #L41

More information on your aws config can be found here

There is a PR out to override this behavior https://github.com/kubernetes/kops/pull/961 with an env var, although I think we need to give the util/pkg/vfs/s3context.go a once over to correct this and https://github.com/kubernetes/kops/pull/961

I think a more robust patch is due here.. I will see about adding it...

Detection steps

Can you please add a bucket to us-east-1 called kops-${domain} and try again? Just wondering if my hunch here is correct.. If so the patch will be pretty straightforward

[Smana]

Hi @kris-nova,

Indeed my default region is eu-west-1. I tried to create a bucket in the US but i still get the same error.

Then i changed my default region to us-east-1 but it didn't help

aws configure get region
us-east-1

kops create cluster --cloud=aws --zones=eu-west1-a cluster0.${mydomain} -v=3
I1121 17:13:29.030311    4849 s3context.go:61] Querying S3 for bucket location for "kops-k8s-us-east-1"

error reading cluster configuration "cluster0.${mydomain}": error reading configuration file s3://kops-k8s-us-east-1/cluster0.${mydomain}/config: error getting location for S3 bucket "kops-k8s-us-east-1": NoCredentialProviders: no valid providers in chain. Deprecated. 
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors

[ghost]

I am having the same issue,

ubuntu@ip-private-ip:~$ kops delete cluster kubernetes1.in
error reading cluster configuration "kubernetes1.in": error reading configuration file s3://examplekubernetes.in/kubernetes1.in/config: error getting location for S3 bucket "examplekubernetes.in": NoCredentialProviders: no valid providers in chain. Deprecated. 
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors

[krisnova]

Sorry this issue must have gotten lost in my inbox @Smana

Could you and @voyalab mention what versions of kops you are running? Also were either of you able to get this working on your own?

[justinsb]

The default region shouldn't matter - the only time it's a problem is if you don't have access to us-east-1 (i.e. if you're running with a china/govcloud account)

Can you (both) confirm that you're setting AWS_PROFILE, but not AWS_DEFAULT_PROFILE, AWS_ACCESS_KEY_ID or AWS_SECRET_KEY etc. In short, AWS_PROFILE should be the only AWS env var set.

@voyalab can you confirm that aws ec2 --region us-east-1 describe-regions works there.

[bacongobbler]

Hey, just tacking on here that I am seeing this as well. Just migrating from kube-up (finally) and I'm seeing this as well.

><> kops create cluster --zones us-west-1a bacongobbler

error reading cluster configuration "bacongobbler": error reading configuration file s3://kops-bacongobbler/bacongobbler/config: error getting location for S3 bucket "kops-bacongobbler": NoCredentialProviders: no valid providers in chain. Deprecated. 
        For verbose messaging see aws.Config.CredentialsChainVerboseErrors

I can confirm that aws ec2 --region us-east-1 describe-regions works for me and I can see both us-east-1 and us-west-1. Dropping into aws ec2 describe-availability-zones --region us-west-1 I can see us-west-1a as available. The kops-bacongobbler bucket exists as well.

AWS_PROFILE is not set on my computer, only AWS_S3_BUCKET which I was using previously with kube-up:

><> env | grep AWS
AWS_S3_BUCKET=bacongobbler-k8s

Running with v1.4.4

EDIT: also, aws config:

[default]
region = us-west-1
aws_access_key_id = REDACTED
aws_secret_access_key = REDACTED
s3 =
    signature_version = s3v4

[bacongobbler]

Found out that it was somewhat of a dupe of https://github.com/kubernetes/kops/issues/855#issuecomment-259444852. Permissions on ~/.aws/config was 600 but was not a symlink. Everything works now after setting AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY!

[chrislovecnm]

Can we close?

[bacongobbler]

Not sure if this resolves the problem for anyone else but sure I guess

[chrislovecnm]

I am getting an error where kops will work with the env variables and not the config file. I am building off of master, and the account is running TFA.

[Smana]

I'll test it again and i'll keep you updated.

[Smana]

I still get the same error. Should i open a new issue ?

kops version
Version 1.4.4

kops create cluster --node-count 4 --zones eu-west-1a,eu-west-1b --dns-zone ${MYZONE} --node-size t2.medium --name ${CLUSTER_NAME}

error reading cluster configuration "${CLUSTER_NAME}": error reading configuration file s3://${S3_BUCKET}/${CLUSTER_NAME}/config: error getting location for S3 bucket "${S3_BUCKET}": NoCredentialProviders: no valid providers in chain. Deprecated. 
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors

Note: i'm using a MFA authentication

[Smana]

https://github.com/kubernetes/kops/issues/226

[ghost]

I was using kops 1.4.1 previously. I solved that problem by providing necessary IAM roles. Also I upgraded version of KOPS 1.4.4 . @justinsb @kris-nova Unfortunately , There is no documentation for IAM roles.

[mayurpatil11]

Can anybody help me withbelow error message: Since I am new to K8s deployment I might be doing silly mistake please bare me with it. (I am trying to install K8s cluster on AWS ubuntu server using Kops commands.

error reading cluster configuration "dev.k8scluster.com": error reading s3://storage.dev.k8ss.com/dev.k8scluster.com/config: Unable to list AWS regions: NoCredentialProviders: no valid providers in chain caused by: EnvAccessKeyNotFound: failed to find credentials in the environment. SharedCredsLoad: failed to load profile, . EC2RoleRequestError: no EC2 instance role found caused by: EC2MetadataError: failed to make EC2Metadata request caused by:

[debu99]

you must export the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, and it is not helpful even you already set it on ~/.aws/config

[fernandrone]

I'm having the same issue. Essentially, when using AWS_PROFILE I'm forced to set both AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY (as @debu99 said); even though my credentials are configured on ~/.aws/credentials, kops will fail.

Example, here's my config:

$ cat ~/.aws/config
[default]
output = json
region = us-east-1

[profile admin]
role_arn = <redacted>/Admin
source_profile = default
region = us-east-1

$ cat ~/.aws/credentials
[default]
aws_access_key_id=<redacted>
aws_secret_access_key=<redacted>

With this configuration and just AWS_PROFILE=admin set, aws ec2 --region us-east-1 describe-regions works fine. So the profile configuration is ok.

But kops fails:

$ AWS_PROFILE=admin kops get ig
error fetching cluster "<redacted>": error reading cluster configuration "<redacted>": error reading s3://<redacted>/config: Unable to list AWS regions: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, admin.
EC2RoleRequestError: no EC2 instance role found
caused by: RequestError: send request failed
caused by: Get http://169.254.169.254/latest/meta-data/iam/security-credentials/: dial tcp 169.254.169.254:80: connect: no route to host

However, if I set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, it works:

$ AWS_PROFILE=admin AWS_ACCESS_KEY_ID=<redacted> AWS_SECRET_ACCESS_KEY=<redacted> kops get ig

In case it's relevant:

$ kops version
Version 1.11.0 (git-2c2042465)

[superma58]

I think the SDK doesn't load all AWS shared config(~/.aws/config) as default, which causes error 'EnvAccessKeyNotFound: failed to find credentials in the environment'. So I fixed this issue by passing environment AWS_SDK_LOAD_CONFIG=true. This environment make SDK to load the all AWS shared config. Don't need to set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

[borelsaffo]

root@ip-172-31-85-125:/home/ubuntu# kops create cluster --name EKS3 --state=s3://borel1 --zones us-east-1

error reading cluster configuration "EKS3": error reading s3://borel1/EKS3/config: Unable to list AWS regions: NoCredentialProviders: no valid providers in chain caused by: EnvAccessKeyNotFound: failed to find credentials in the environment. SharedCredsLoad: failed to load profile, . EC2RoleRequestError: no EC2 instance role found caused by: EC2MetadataError: failed to make EC2Metadata request status code: 404, request id: caused by: <?xml version="1.0" encoding="iso-8859-1"?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

404 - Not Found

root@ip-172-31-85-125:/home/ubuntu# env | egrep '(AWS_PROFILE|KOPS)' KOPS_STATE_STORE=s3://borel1 root@ip-172-31-85-125:/home/ubuntu#

[olemarkus]

@borelsaffo please file a new bug following the template.