search

openshift / cloud-credential-operator

Manage cloud provider credentials as Kubernetes CRDs
Apache License 2.0
62 stars 145 forks source link

Running ccoctl with CloudFront second time generates incorrect manifests #684

Closed davtex closed 1 month ago

davtex commented 8 months ago

Hi,

I am currently working on implementing STS with Cloudfront in our in-house tool for deploying new clusters. I am currently using 4.13.

Everything works when I run ccoctl the first time:

./ccoctl aws create-all \
  --create-private-s3-bucket \
  --credentials-requests-dir /home/ec2-user/installer/redacted-test12/credentials-requests \
  --name redacted-redacted-test12-irsa-test \
  --region eu-central-1 \
  --output-dir /home/ec2-user/installer/redacted-test12/irsa-config

All manifests are created together with correct cluster-authentication-02-config.yaml :

apiVersion: config.openshift.io/v1
kind: Authentication
metadata:
  name: cluster
spec:
  serviceAccountIssuer: https://redacted.cloudfront.net

However when I run the same command again (e.g. because installation failed for whatever reason), following happens:

apiVersion: config.openshift.io/v1
kind: Authentication
metadata:
  name: cluster
spec:
  serviceAccountIssuer: https://redacted-redacted-test12-irsa-test-oidc.s3.eu-central-1.amazonaws.com

First run logs:

2024/03/06 00:18:28 Using existing RSA keypair found at /home/ec2-user/installer/redacted-test12/irsa-config/serviceaccount-signer.private
2024/03/06 00:18:28 Copying signing key for use by installer
2024/03/06 00:18:29 Bucket redacted-redacted-test12-irsa-test-oidc created
2024/03/06 00:18:31 CloudFront origin access identity created with ID REDACTED, waiting 30s for it to become active
2024/03/06 00:19:01 Update policy for bucket redacted-redacted-test12-irsa-test-oidc to allow access from CloudFront origin access identity with ID REDACTED
2024/03/06 00:19:02 Blocked public access for the bucket redacted-redacted-test12-irsa-test-oidc
2024/03/06 00:19:03 CloudFront distribution created with ID REDACTED
2024/03/06 00:19:03 Waiting 30s for CloudFront distribution with ID REDACTED to be deployed...
2024/03/06 00:19:33 Waiting 30s for CloudFront distribution with ID REDACTED to be deployed...
2024/03/06 00:20:04 Waiting 30s for CloudFront distribution with ID REDACTED to be deployed...
2024/03/06 00:20:34 Waiting 30s for CloudFront distribution with ID REDACTED to be deployed...
2024/03/06 00:21:05 Waiting 30s for CloudFront distribution with ID REDACTED to be deployed...
2024/03/06 00:21:35 Waiting 30s for CloudFront distribution with ID REDACTED to be deployed...
2024/03/06 00:22:06 Waiting 30s for CloudFront distribution with ID REDACTED to be deployed...
2024/03/06 00:22:36 Waiting 30s for CloudFront distribution with ID REDACTED to be deployed...
2024/03/06 00:23:07 CloudFront distribution with ID REDACTED is successfully deployed
2024/03/06 00:23:07 OpenID Connect discovery document in the S3 bucket redacted-redacted-test12-irsa-test-oidc at .well-known/openid-configuration updated
2024/03/06 00:23:07 Reading public key
2024/03/06 00:23:07 JSON web key set (JWKS) in the S3 bucket redacted-redacted-test12-irsa-test-oidc at keys.json updated
2024/03/06 00:23:08 Identity Provider created with ARN: arn:aws:iam::redacted:oidc-provider/d6j73lq0exp5g.cloudfront.net
2024/03/06 00:23:08 Ignoring CredentialsRequest openshift-cloud-credential-operator/openshift-cluster-api-aws with tech-preview annotation
2024/03/06 00:23:08 Role arn:aws:iam::redacted:role/redacted-redacted-test12-irsa-test-openshift-machine-api-aws-c created
2024/03/06 00:23:08 Saved credentials configuration to: /home/ec2-user/installer/redacted-test12/irsa-config/manifests/openshift-machine-api-aws-cloud-credentials-credentials.yaml
2024/03/06 00:23:08 Updated Role policy for Role redacted-redacted-test12-irsa-test-openshift-machine-api-aws-c
2024/03/06 00:23:09 Role arn:aws:iam::redacted:role/redacted-redacted-test12-irsa-test-openshift-cloud-credential- created
2024/03/06 00:23:09 Saved credentials configuration to: /home/ec2-user/installer/redacted-test12/irsa-config/manifests/openshift-cloud-credential-operator-cloud-credential-operator-iam-ro-creds-credentials.yaml
2024/03/06 00:23:09 Updated Role policy for Role redacted-redacted-test12-irsa-test-openshift-cloud-credential-
2024/03/06 00:23:09 Role arn:aws:iam::redacted:role/redacted-redacted-test12-irsa-test-openshift-image-registry-in created
2024/03/06 00:23:09 Saved credentials configuration to: /home/ec2-user/installer/redacted-test12/irsa-config/manifests/openshift-image-registry-installer-cloud-credentials-credentials.yaml
2024/03/06 00:23:09 Updated Role policy for Role redacted-redacted-test12-irsa-test-openshift-image-registry-in
2024/03/06 00:23:10 Role arn:aws:iam::redacted:role/redacted-redacted-test12-irsa-test-openshift-ingress-operator- created
2024/03/06 00:23:10 Saved credentials configuration to: /home/ec2-user/installer/redacted-test12/irsa-config/manifests/openshift-ingress-operator-cloud-credentials-credentials.yaml
2024/03/06 00:23:10 Updated Role policy for Role redacted-redacted-test12-irsa-test-openshift-ingress-operator-
2024/03/06 00:23:10 Role arn:aws:iam::redacted:role/redacted-redacted-test12-irsa-test-openshift-cloud-network-con created
2024/03/06 00:23:10 Saved credentials configuration to: /home/ec2-user/installer/redacted-test12/irsa-config/manifests/openshift-cloud-network-config-controller-cloud-credentials-credentials.yaml
2024/03/06 00:23:10 Updated Role policy for Role redacted-redacted-test12-irsa-test-openshift-cloud-network-con
2024/03/06 00:23:11 Role arn:aws:iam::redacted:role/redacted-redacted-test12-irsa-test-openshift-cluster-csi-drive created
2024/03/06 00:23:11 Saved credentials configuration to: /home/ec2-user/installer/redacted-test12/irsa-config/manifests/openshift-cluster-csi-drivers-ebs-cloud-credentials-credentials.yaml
2024/03/06 00:23:11 Updated Role policy for Role redacted-redacted-test12-irsa-test-openshift-cluster-csi-drive

Second run logs:

2024/03/06 00:23:47 Using existing RSA keypair found at /home/ec2-user/installer/redacted-test12/irsa-config/serviceaccount-signer.private
2024/03/06 00:23:47 Copying signing key for use by installer
2024/03/06 00:23:47 Bucket redacted-redacted-test12-irsa-test-oidc already exists and is owned by the user
2024/03/06 00:23:47 OpenID Connect discovery document in the S3 bucket redacted-redacted-test12-irsa-test-oidc at .well-known/openid-configuration updated
2024/03/06 00:23:47 Reading public key
2024/03/06 00:23:47 JSON web key set (JWKS) in the S3 bucket redacted-redacted-test12-irsa-test-oidc at keys.json updated
2024/03/06 00:23:48 Existing Identity Provider found with ARN: arn:aws:iam::redacted:oidc-provider/d6j73lq0exp5g.cloudfront.net
2024/03/06 00:23:48 Ignoring CredentialsRequest openshift-cloud-credential-operator/openshift-cluster-api-aws with tech-preview annotation
2024/03/06 00:23:48 Existing role arn:aws:iam::redacted:role/redacted-redacted-test12-irsa-test-openshift-machine-api-aws-c found
2024/03/06 00:23:49 Updated Role policy for Role redacted-redacted-test12-irsa-test-openshift-machine-api-aws-c
2024/03/06 00:23:49 Existing role arn:aws:iam::redacted:role/redacted-redacted-test12-irsa-test-openshift-cloud-credential- found
2024/03/06 00:23:49 Updated Role policy for Role redacted-redacted-test12-irsa-test-openshift-cloud-credential-
2024/03/06 00:23:49 Existing role arn:aws:iam::redacted:role/redacted-redacted-test12-irsa-test-openshift-image-registry-in found
2024/03/06 00:23:49 Updated Role policy for Role redacted-redacted-test12-irsa-test-openshift-image-registry-in
2024/03/06 00:23:49 Existing role arn:aws:iam::redacted:role/redacted-redacted-test12-irsa-test-openshift-ingress-operator- found
2024/03/06 00:23:49 Updated Role policy for Role redacted-redacted-test12-irsa-test-openshift-ingress-operator-
2024/03/06 00:23:50 Existing role arn:aws:iam::redacted:role/redacted-redacted-test12-irsa-test-openshift-cloud-network-con found
2024/03/06 00:23:50 Updated Role policy for Role redacted-redacted-test12-irsa-test-openshift-cloud-network-con
2024/03/06 00:23:50 Existing role arn:aws:iam::redacted:role/redacted-redacted-test12-irsa-test-openshift-cluster-csi-drive found
2024/03/06 00:23:50 Updated Role policy for Role redacted-redacted-test12-irsa-test-openshift-cluster-csi-drive

Problem: ccoctl seems to ignore "--create-private-s3-bucket" flag when S3 bucket already exists and generates incorrect manifest

Expectation: When "--create-private-s3-bucket" is set and S3 bucket exists, I would expect ccoctl to query AWS for associated Cloudfront instance and generate "cluster-authentication-02-config.yaml" manifest with correct value.

RCosta3 commented 6 months ago

CCOCTL fails to complete the OIDC configuration when creating setting S3 buckets with "--create-private-s3-bucket" on AWS where the AWS S3 policy setting at the Account level to prevent public access to S3 Buckets, and where buckets are created with public access blocked by default. This issue causes the OIDC deployment to terminate due to the attempt to set the S3 bucket policy to block public access as requested by this setting. Is there a possibility to have the CCOCTL code status the S3 Bucket public access setting prior to setting the policy so has not to terminated OIDC deployment if the bucket is already configure to deny public access?

openshift-bot commented 3 months ago

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

openshift-bot commented 2 months ago

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity. Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten /remove-lifecycle stale

openshift-bot commented 1 month ago

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen. Mark the issue as fresh by commenting /remove-lifecycle rotten. Exclude this issue from closing again by commenting /lifecycle frozen.

/close

openshift-ci[bot] commented 1 month ago

@openshift-bot: Closing this issue.

In response to [this](https://github.com/openshift/cloud-credential-operator/issues/684#issuecomment-2364765423): >Rotten issues close after 30d of inactivity. > >Reopen the issue by commenting `/reopen`. >Mark the issue as fresh by commenting `/remove-lifecycle rotten`. >Exclude this issue from closing again by commenting `/lifecycle frozen`. > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.