search

kubernetes / kops

Kubernetes Operations (kOps) - Production Grade k8s Installation, Upgrades and Management
https://kops.sigs.k8s.io/
Apache License 2.0
15.9k stars 4.65k forks source link

kops failed with Could not retrieve location for AWS bucket when do cross-account role profile in EC2 instance #4989

Closed wanghong230 closed 6 years ago

wanghong230 commented 6 years ago

------------- BUG REPORT TEMPLATE --------------------

  1. What kops version are you running? The command kops version, will display this information.

Version 1.8.1 (git-94ef202)

  1. What Kubernetes version are you running? kubectl version will print the version if a cluster is running or provide the Kubernetes version specified as a kops flag.

kubenetes 1.10. I am installing, so it is not releated here

  1. What cloud provider are you using?

AWS

  1. What commands did you run? What is the simplest way to reproduce this issue?

    kops create cluster --name=SOME_CLUSTER_NAME --state=s3://SOME_BUCKET --node-count=3 --authorization=RBAC --zones=us-west-2a,us-west-2b,us-west-2c --master-zones=us-west-2a,us-west-2b,us-west-2c --node-size=m5.large --master-size=r4.large --master-count=3 --networking=weave --topology=private --ssh-public-key=/root/.ssh/id_rsa.pub --kubernetes-version=1.10.0 --target=terraform -v 100

    We are runing it inside EC2 instance in a seperate AWS account to deploy cluster in another AWS account, the cross account is configured and tested via aws cli.

    cat ~/.aws/config
[default]
region = us-west-2
output = json
[profile test]
role_arn = arn:aws:iam::AWS_ACCOUNT:role/CROSS_ACCOUNT_ROLE
credential_source=Ec2InstanceMetadata

  2. What happened after the commands executed?

    
I0413 20:09:11.480862     177 s3context.go:145] unable to get bucket location from region "us-east-1"; scanning all regions: AccessDenied: Access Denied
status code: 403, request id: 3A063105A1E1FE56
I0413 20:09:11.893443     177 s3context.go:198] Querying S3 for bucket location for BUCKET
I0413 20:09:11.893866     177 s3context.go:203] Doing GetBucketLocation in "us-west-2"
I0413 20:09:11.894441     177 s3context.go:203] Doing GetBucketLocation in "ap-south-1"
I0413 20:09:11.894927     177 s3context.go:203] Doing GetBucketLocation in "eu-west-3"
I0413 20:09:11.895420     177 s3context.go:203] Doing GetBucketLocation in "eu-west-2"
I0413 20:09:11.895918     177 s3context.go:203] Doing GetBucketLocation in "eu-west-1"
I0413 20:09:11.896387     177 s3context.go:203] Doing GetBucketLocation in "ap-northeast-2"
I0413 20:09:11.896871     177 s3context.go:203] Doing GetBucketLocation in "ap-northeast-1"
I0413 20:09:11.897396     177 s3context.go:203] Doing GetBucketLocation in "sa-east-1"
I0413 20:09:11.897887     177 s3context.go:203] Doing GetBucketLocation in "ca-central-1"
I0413 20:09:11.898333     177 s3context.go:203] Doing GetBucketLocation in "ap-southeast-1"
I0413 20:09:11.898773     177 s3context.go:203] Doing GetBucketLocation in "ap-southeast-2"
I0413 20:09:11.899239     177 s3context.go:203] Doing GetBucketLocation in "eu-central-1"
I0413 20:09:11.899647     177 s3context.go:203] Doing GetBucketLocation in "us-east-1"
I0413 20:09:11.900090     177 s3context.go:203] Doing GetBucketLocation in "us-east-2"
I0413 20:09:11.900530     177 s3context.go:203] Doing GetBucketLocation in "us-west-1"

error reading cluster configuration "CLUSTER": error reading s3://BUCKET/CLUSTER/config: Could not retrieve location for AWS bucket BUCKET

6. What did you expect to happen?
It should works.

aws s3 ls can list the bucket and access the account buckets

  1. Please provide your cluster manifest. Execute kops get --name my.example.com -oyaml to display your cluster manifest. You may want to remove your cluster name and other sensitive information.

This issue is before having the configuration file.

  1. Please run the commands with most verbose logging by adding the -v 10 flag. Paste the logs into this report, or in a gist and provide the gist link here. See 5
  2. Anything else do we need to know? I set the environment.
AWS_SDK_LOAD_CONFIG=1
AWS_PROFILE=test

------------- FEATURE REQUEST TEMPLATE --------------------

  1. Describe IN DETAIL the feature/behavior/change you would like to see.

  2. Feel free to provide a design supporting your feature request.

wskinner commented 6 years ago

I am having a very similar issue.

neiltingley commented 6 years ago

double check the name of your bucket!

niroliyanage commented 6 years ago

+1 with the same issue...

sohel2020 commented 6 years ago

same issue i'm facing

niroliyanage commented 6 years ago

I got past it by adding a bucket policy just so it allows getBucketLocation

sohel2020 commented 6 years ago

@niroliyanage can you share the bucket policy please?

geojaz commented 6 years ago

/close see #5622

berniechiu commented 6 years ago

@niroliyanage I got this too, but I can't get it passed by adding getBucketLocation permission to policy

Still encountered something like Could not retrieve location for AWS bucket xxxx-xxxx-xxx

berniechiu commented 6 years ago

@niroliyanage Actually I resolved the above issue, but can't seem to export the config from the S3

encountered

kops export kubecfg --name=xxxxxxx --state s3://xxxxxxx

W0828 19:28:23.122657 43048 create_kubecfg.go:75] Did not find API endpoint for gossip hostname; may not be able to reach cluster

maybetonyfu commented 6 years ago

@berniechiu I ran into the exact error. I found my problem and fixed it. You can check if the root cause is the same.

Basically when I ask kops to generate a kubecfg for a gossip-based cluster, it will try to use AWS api to get your ELB public dns name. My problem is the user that runs kops export kubecfg does not have enough permission to do so.

I ended up put read/list permission for s3/ec2/r53/elb in am IAM policy. And attached the policy to the user who runs kops commands. The policy is not minimally permissive so it could be improved.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucketByTags",
                "s3:GetLifecycleConfiguration",
                "s3:GetBucketTagging",
                "s3:GetInventoryConfiguration",
                "s3:GetObjectVersionTagging",
                "s3:ListBucketVersions",
                "s3:GetBucketLogging",
                "s3:ListBucket",
                "s3:GetAccelerateConfiguration",
                "s3:GetBucketPolicy",
                "s3:GetObjectVersionTorrent",
                "s3:GetObjectAcl",
                "s3:GetEncryptionConfiguration",
                "s3:GetBucketRequestPayment",
                "s3:GetObjectVersionAcl",
                "s3:GetObjectTagging",
                "s3:GetMetricsConfiguration",
                "s3:GetIpConfiguration",
                "s3:ListBucketMultipartUploads",
                "s3:GetBucketWebsite",
                "s3:GetBucketVersioning",
                "s3:GetBucketAcl",
                "s3:GetBucketNotification",
                "s3:GetReplicationConfiguration",
                "s3:ListMultipartUploadParts",
                "s3:GetObject",
                "s3:GetObjectTorrent",
                "s3:GetBucketCORS",
                "s3:GetAnalyticsConfiguration",
                "s3:GetObjectVersionForReplication",
                "s3:GetBucketLocation",
                "s3:GetObjectVersion"
            ],
            "Resource": [
                "*",
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets",
                "s3:HeadBucket"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "ec2:DescribeVolumesModifications",
                "ec2:GetHostReservationPurchasePreview",
                "ec2:DescribeSnapshots",
                "ec2:DescribePlacementGroups",
                "ec2:GetConsoleScreenshot",
                "ec2:DescribeHostReservationOfferings",
                "ec2:DescribeInternetGateways",
                "ec2:GetLaunchTemplateData",
                "ec2:DescribeVolumeStatus",
                "ec2:DescribeScheduledInstanceAvailability",
                "ec2:DescribeSpotDatafeedSubscription",
                "ec2:DescribeVolumes",
                "ec2:DescribeFpgaImageAttribute",
                "ec2:DescribeExportTasks",
                "ec2:DescribeAccountAttributes",
                "ec2:DescribeNetworkInterfacePermissions",
                "ec2:DescribeReservedInstances",
                "ec2:DescribeKeyPairs",
                "ec2:DescribeNetworkAcls",
                "ec2:DescribeRouteTables",
                "ec2:DescribeReservedInstancesListings",
                "ec2:DescribeEgressOnlyInternetGateways",
                "ec2:DescribeSpotFleetRequestHistory",
                "ec2:DescribeLaunchTemplates",
                "ec2:DescribeVpcClassicLinkDnsSupport",
                "ec2:DescribeVpnConnections",
                "ec2:DescribeSnapshotAttribute",
                "ec2:DescribeVpcPeeringConnections",
                "ec2:DescribeReservedInstancesOfferings",
                "ec2:DescribeIdFormat",
                "ec2:DescribeVpcEndpointServiceConfigurations",
                "ec2:DescribePrefixLists",
                "ec2:GetReservedInstancesExchangeQuote",
                "ec2:DescribeVolumeAttribute",
                "ec2:DescribeInstanceCreditSpecifications",
                "ec2:DescribeVpcClassicLink",
                "ec2:DescribeImportSnapshotTasks",
                "ec2:DescribeVpcEndpointServicePermissions",
                "ec2:GetPasswordData",
                "ec2:DescribeScheduledInstances",
                "ec2:DescribeImageAttribute",
                "ec2:DescribeVpcEndpoints",
                "ec2:DescribeReservedInstancesModifications",
                "ec2:DescribeElasticGpus",
                "ec2:DescribeSubnets",
                "ec2:DescribeVpnGateways",
                "ec2:DescribeMovingAddresses",
                "ec2:DescribeAddresses",
                "ec2:DescribeInstanceAttribute",
                "ec2:DescribeRegions",
                "ec2:DescribeFlowLogs",
                "ec2:DescribeDhcpOptions",
                "ec2:DescribeVpcEndpointServices",
                "ec2:DescribeSpotInstanceRequests",
                "ec2:DescribeVpcAttribute",
                "ec2:GetConsoleOutput",
                "ec2:DescribeSpotPriceHistory",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DescribeAvailabilityZones",
                "ec2:DescribeNetworkInterfaceAttribute",
                "ec2:DescribeVpcEndpointConnections",
                "ec2:DescribeInstanceStatus",
                "ec2:DescribeHostReservations",
                "ec2:DescribeIamInstanceProfileAssociations",
                "ec2:DescribeTags",
                "ec2:DescribeLaunchTemplateVersions",
                "ec2:DescribeBundleTasks",
                "ec2:DescribeIdentityIdFormat",
                "ec2:DescribeImportImageTasks",
                "ec2:DescribeClassicLinkInstances",
                "ec2:DescribeNatGateways",
                "ec2:DescribeCustomerGateways",
                "ec2:DescribeVpcEndpointConnectionNotifications",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeSpotFleetRequests",
                "ec2:DescribeHosts",
                "ec2:DescribeImages",
                "ec2:DescribeFpgaImages",
                "ec2:DescribeSpotFleetInstances",
                "ec2:DescribeSecurityGroupReferences",
                "ec2:DescribeVpcs",
                "ec2:DescribeConversionTasks",
                "ec2:DescribeStaleSecurityGroups"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "route53:ListTrafficPolicyInstances",
                "route53:GetTrafficPolicyInstanceCount",
                "route53:GetChange",
                "route53:ListTrafficPolicyVersions",
                "route53:TestDNSAnswer",
                "route53:GetHostedZone",
                "route53:GetHealthCheck",
                "route53:ListHostedZonesByName",
                "route53:ListQueryLoggingConfigs",
                "route53:GetCheckerIpRanges",
                "route53:ListTrafficPolicies",
                "route53:ListResourceRecordSets",
                "route53:ListGeoLocations",
                "route53:GetTrafficPolicyInstance",
                "route53:GetHostedZoneCount",
                "route53:GetHealthCheckCount",
                "route53:GetQueryLoggingConfig",
                "route53:ListReusableDelegationSets",
                "route53:GetHealthCheckLastFailureReason",
                "route53:GetHealthCheckStatus",
                "route53:ListTrafficPolicyInstancesByHostedZone",
                "route53:ListHostedZones",
                "route53:ListVPCAssociationAuthorizations",
                "route53:GetReusableDelegationSetLimit",
                "route53:GetReusableDelegationSet",
                "route53:ListTagsForResource",
                "route53:ListTagsForResources",
                "route53:GetAccountLimit",
                "route53:ListTrafficPolicyInstancesByPolicy",
                "route53:ListHealthChecks",
                "route53:GetGeoLocation",
                "route53:GetHostedZoneLimit",
                "route53:GetTrafficPolicy"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:DescribeLoadBalancerAttributes",
                "elasticloadbalancing:DescribeSSLPolicies",
                "elasticloadbalancing:DescribeLoadBalancers",
                "elasticloadbalancing:DescribeTargetGroupAttributes",
                "elasticloadbalancing:DescribeListeners",
                "elasticloadbalancing:DescribeTags",
                "elasticloadbalancing:DescribeAccountLimits",
                "elasticloadbalancing:DescribeTargetHealth",
                "elasticloadbalancing:DescribeTargetGroups",
                "elasticloadbalancing:DescribeListenerCertificates",
                "elasticloadbalancing:DescribeRules",
                "elasticloadbalancing:DescribeLoadBalancerPolicies",
                "elasticloadbalancing:DescribeLoadBalancerPolicyTypes",
                "elasticloadbalancing:DescribeInstanceHealth"
            ],
            "Resource": "*"
        }
    ]
}
berniechiu commented 6 years ago

@tonysickpony Thx

I also came up my final solution for restricted role here ha!

https://github.com/kubernetes/kops/issues/1873#issuecomment-418012198

enavarre-cl commented 1 year ago

double check the name of your bucket! Thanks! XD