openshift-install create error getting S3 bucket (t7-6dlcq-bootstrap) policy: AccessDenied: Access Denied 4.12.0-0.okd-scos-2022-10-25-053756

[apmlima]

Having this error with OKD SCOS stable 4.12, other OKD versions work fine:

$ openshift-install create cluster --dir t7 level=info msg=Credentials loaded from default AWS environment variables level=info msg=Consuming Install Config from target directory level=info msg=Skipping quota checks level=info msg=Creating infrastructure resources... level=error level=error msg=Error: error getting S3 bucket (t7-6dlcq-bootstrap) policy: AccessDenied: Access Denied level=error msg= status code: 403, request id: AQZBRTNMN4VKFGRV, host id: GhQ4qI82IY659LOVbKl9pMHcuPkSFdaI0unlp84ineQltj8q799hESEWifxS3HrS55iN/QnFy4Q= level=error level=error msg= with aws_s3_bucket.ignition, level=error msg= on main.tf line 47, in resource "aws_s3_bucket" "ignition": level=error msg= 47: resource "aws_s3_bucket" "ignition" { level=error level=error msg=failed to fetch Cluster: failed to generate asset "Cluster": failure applying terraform for "bootstrap" stage: failed to create cluster: failed to apply Terraform: exit status 1 level=error level=error msg=Error: error getting S3 bucket (t7-6dlcq-bootstrap) policy: AccessDenied: Access Denied level=error msg= status code: 403, request id: AQZBRTNMN4VKFGRV, host id: GhQ4qI82IY659LOVbKl9pMHcuPkSFdaI0unlp84ineQltj8q799hESEWifxS3HrS55iN/QnFy4Q= level=error level=error msg= with aws_s3_bucket.ignition, level=error msg= on main.tf line 47, in resource "aws_s3_bucket" "ignition": level=error msg= 47: resource "aws_s3_bucket" "ignition" { level=error level=error

Thanks!

Pedro

[aleskandro]

Hello @apmlima can you send the entire content of t7/.openshift_install.log?

[LorbusChris]

@apmlima you'll have to ensure your AWS account has all the S3 permissions specified in https://docs.okd.io/latest/installing/installing_aws/installing-aws-account.html#installation-aws-permissions_installing-aws-account

[LorbusChris]

Also please double check your resource quotas/limits on the account that you're using, especially in case it's the same account that you're already using to deploy other OKD clusters.

[apmlima]

@aleskandro here it is .openshift_install.log: pedro_openshift_install.log.tar.gz

@LorbusChris , is creating 4.6 to 4.10 normal OKD clusters fine and the normal OKD version usually checks and reports missing permissions on create.

I appreciate your help guys!

[apmlima]

@LorbusChris , you Closed the ticket without point out what policy I am missing after I send the requested logs and send me to documentation full of mistakes: Invalid Action: The action s3:HeadBucket does not exist. Invalid Action: The action s3:GetBucketReplication does not exist. Invalid Action: The action s3.GetBucketPolicy does not exist.

This isn't the help we get on the original OKD project, and I installed OKD-SCOS to help you develop and improve this. I found an issue and reported. I work with OKD for more than 2 years and think instead of you being here dealing with issues and dealing with users might be more appropriate to be coding...

[LorbusChris]

@apmlima your interest in OKD/SCOS is appreciated, however toxic comments such as the above are not.

This is not an issue with the OKD installer, but with your AWS account permissions. OKD/SCOS have been installed dozens of times on AWS in CI and manual tests. Please triple check your account has sufficient permissions for these three s3 resources that you listed. They are also listed in the documentation I linked for you above. The permissions checking relies on simulation provided by the AWS API and is done on a best effort bases and does not guarantee to catch all problems.

[apmlima]

@LorbusChris , wasn't my intention to be toxic in any way and I appreciate all the OKD team's work. If you thought that I apologise. Enterprises for security reasons need to limit users access rights to only what are strictly required to do their jobs, we use an AWS account for cluster create purposes with the needed rights (policies) we expect to get from your documentation. If the policies you specify on your documentation don't exist there is an issue and I told you for you to fix and help me put openshift-install to work.

[LorbusChris]

All of these roles very much do exist, and what you’re seeing is the error you get if your account is not permitted to access them. Please refer to e.g. https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadBucket.html and talk to your organization‘s admin to add the required roles to your IAM account.

[apmlima]

@LorbusChris , Thanks for the fast reply. My real account is Admin and I am Admin. But the CI/CD pipeline that creates clusters uses an account (I created) only with the least needed policies that I expect to get from OKD docs. This account is creating 4.6-4.10 clusters with no permission issues. The s3:ListBucket on that AWS doc is added.

[LorbusChris]

Does your user have the s3:ListAllMyBuckets permission? (required per https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListBuckets.html)

[LorbusChris]

See also https://aws.amazon.com/premiumsupport/knowledge-center/s3-access-denied-bucket-policy/ https://aws.amazon.com/premiumsupport/knowledge-center/s3-troubleshoot-403/

[LorbusChris]

@apmlima I'm guessing s3:GetBucketPolicy will fix it for you. It was added to the docs recently

[apmlima]

@LorbusChris , Brilliant! Thanks for all this help. After adding s3:GetBucketPolicy finally created the bootstrap machine and worker/compute nodes but unfortunately has broken near the end, probably missing another permission...

time="2022-11-02T11:16:00Z" level=error msg="UnauthorizedOperation: You are not authorized to perform this operation. Encoded authorization failure message: jdKjBcZEAySjIK0A8mKNsduSnHmoPAf68VPLTt9EwBamIOImVyda6Agf8QLNF_2iX_65b92FYAHKvGto209Aw0Or6K2LyQtjYsosPcKjKWNfQvtL0fXTGAoR7OfNO0HyZ-thjYLsDth-Gn5Db2yQchm7jRcQKup4zhwmtaRL9DFi8WHJq2z8hs91drvHK0excOxrXCCv8ukHCSq_PQXj6P0mDm4UUhEFAsisU3mvLOgjaFKEkNEwx3t8jgTarynnU8hfKsNZMaVETzH50MFCp02kxfv6c-skOkl1B-Eu8BITMQ9weGDJGZGUaa8Up34ynmEPWy9QaCQKCVchu6ZP2mzTqCjcrenxawKgJOrNNVenPQvODUuS96h6FZjy16DVrftkcjNG2YzY4tbNkZ7_tJSjHrOCmfuY7LHVLABVqqBYVM54JbV2v4wrVx0UtFtY1up3TUSE0jOKteTFXKwSL2GTGqlBCBj3ioVD1UTAvznJr_FFjNyZhQQdZKhajF3dd8d8tJxsjX_inBH76-Ny5LuZep_QkM6wPygRDKbkbiC_l6Lb-BEj5VuE_BySHgoJCZZcVHLPJkTFuunoM2vJIE1w8BCCweDhNLDrt1_9zQDNAVKQC9tOy82RWErbQUJOhu_bFGOPKl8S5Rz_1QSvNltZpLFj3RT8guWQFYo_hx71I6qhhkVE4g5lt_pFV5jh2CzMqs4gk5T_l5cw7qTkkCxzvTICo7pFI0Qx3jWyKRv47QjeDutg34GJg6XXrXX-0COQD9bTRQfwovjWuQa6b41ozjcen5LNfVNmGSi7cgVhEbvmzgke2d-9WnZ7D2uyIQ6pqSImWjA5uc8CaqYi2jDquoFsqa4N2_HmGOAkC1Mls2Puj3ilmSbCuj0YL43dq3sKjtccFsf0VpHqHq2w-0M8Xjyc6-Q3Uxk5gZB-KuKZ8vlh3X__zmb6pGbcX6CJqN-9RIPSe9AWzXf6FKHL62h9rwoFLNWxKYQzk6SzeoaaS2vutCDjJCrOwQhWNnr-e7jJCTrDOrz8af5-VVCZTr-idatx3kiSZ-0Vgc8HVeSBZbtvMcvtHimOWje53ARVoJbH0WVhRLreeItCpS2VKkXUZHJAhPbVUtTDQ16Mj3RCLlpaRPAoOaT2QWcR11C2iw\n\tstatus code: 403, request id: 8d20618e-67b7-4395-9b05-7f570ffefe62" Instance=i-0077a4f11eb235886

Please where can I check what permissions added to the docs recently? I am sending the latest logs: https://drive.google.com/file/d/176p2hpga6fmUgUeiOoFtP7cnd_OLG3vC/view?usp=sharing https://drive.google.com/file/d/1HPgdeLW04Oqyp2AzHwj_Xka_Afd4cEcO/view?usp=sharing

[LorbusChris]

https://github.com/openshift/openshift-docs/commits/main/modules/installation-aws-permissions.adoc

[apmlima]

@LorbusChris , thanks for that doc and I tried the missing perm but still get the error at the end. Can you please give me an idea with the logs I provided what is the missing perms?

[LorbusChris]

Please do the legwork yourself: First thing you should always do is to do a search for the error message you received yourself. Also never paste hundreds of lines of logs here forcing everybody to scroll endlessly.

How far did you go back in the commit history of the doc site? There was more than just one additional permission added/changed over the last months/years. You're still missing something that might've been added to the docs some more time ago.

Now, please decode that error message to see what the error is (giyf: https://aws.amazon.com/premiumsupport/knowledge-center/ec2-not-auth-launch/)