search

org-formation / aws-resource-providers

A community driven repository where you can find AWS Resource Type Providers for different purposes (including org-formation ones).
MIT License
85 stars 21 forks source link

020-secure-defaults: Error validating schemaHandlerPackage #145

Open msaavedra-earnd opened 2 months ago

msaavedra-earnd commented 2 months ago

When doing the integration of the secure defaults, I am getting the following error message on the register types I am integrating:

ERROR: Workload EbsEncryptionDefaultsRP in 123456789021/us-east-2 updated failed. reason: Error validating schemaHandlerPackage. Check the permissions on the bucket and object in your account and try again. (123456789021 = LogArchiveAccount)

This happens with the types that I am integrating, which are:

I am doing the integration according to the reference. Additionally I was checking the providers and I don't see anything different.

Am I missing something?

OlafConijn commented 2 months ago

there most common reasons for this issue: the role you are using doesnt have permissions to read from the community-resource-provider-catalog bucket.

If not the permissions: I believe i have seen this issue once or twice returned from CloudFormation and then go away by itself. That typically took a couple of hours.

craighurley commented 2 months ago

I'm seeing this same issue. I got it yesterday and again today; nearly 24hours between attempts.

I was using the old version of resource types (0.x.y) and wanted to upgrade to the latest version (1.0.0).

The OrganizationAccountAccessRole role being used is very permissive:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "*",
            "Resource": "*"
        }
    ]
}

Is there any other info I can gather to help troubleshoot?

craighurley commented 2 months ago

For this task file:

Parameters:
  <<: !Include "../../parameters.yaml"

  catalogBucket:
    Type: String
    Default: community-resource-provider-catalog

# Repository for public CloudFormation resource types: https://github.com/org-formation/aws-resource-providers

OrganizationsPolicyRp:
  Type: register-type
  ResourceType: "Community::Organizations::Policy"
  SchemaHandlerPackage: !Sub "s3://${catalogBucket}/community-organizations-policy-1.0.0.zip"
  MaxConcurrentTasks: !Ref MaxConcurrentTasks
  OrganizationBinding:
    Region: us-east-1 # Only compatible to us-east-1 region
    IncludeMasterAccount: true

CommunityIamPasswordPolicyRP:
  Type: register-type
  ResourceType: 'Community::IAM::PasswordPolicy'
  SchemaHandlerPackage: !Sub "s3://${catalogBucket}/community-iam-passwordpolicy-1.0.0.zip"
  MaxConcurrentTasks: !Ref MaxConcurrentTasks
  OrganizationBinding:
    Region: us-east-1 # IAM is a global service that operates out of us-east-1
    IncludeMasterAccount: true
    Account: '*'

I get these errors:

INFO: Executing: update-organization organization.yaml.
INFO: organization up to date, no work to be done.
INFO: Task OrganizationUpdate execute successful.
INFO: Executing: include templates/010-types/_tasks.yaml.
INFO: Executing: register-type OrganizationsPolicyRp.
INFO: Executing: register-type CommunityIamPasswordPolicyRP.
INFO: register-type workload OrganizationsPolicyRp already up to date.
INFO: Task OrganizationsPolicyRp execute successful.
ERROR: Workload CommunityIamPasswordPolicyRP in 111111111111/us-east-1 update failed. reason: Error validating schemaHandlerPackage. Check the permissions on the bucket and object in your account and try again. (111111111111 = DevAccount)
Error validating schemaHandlerPackage. Check the permissions on the bucket and object in your account and try again. (use option --print-stack to print stack)
ERROR: Workload CommunityIamPasswordPolicyRP in 222222222222/us-east-1 update failed. reason: Error validating schemaHandlerPackage. Check the permissions on the bucket and object in your account and try again. (222222222222 = SecurityToolingAccount)
Error validating schemaHandlerPackage. Check the permissions on the bucket and object in your account and try again. (use option --print-stack to print stack)
ERROR: Workload CommunityIamPasswordPolicyRP in 333333333333/us-east-1 update failed. reason: Error validating schemaHandlerPackage. Check the permissions on the bucket and object in your account and try again. (333333333333 = BackupAccount)
Error validating schemaHandlerPackage. Check the permissions on the bucket and object in your account and try again. (use option --print-stack to print stack)
ERROR: Workload CommunityIamPasswordPolicyRP in 444444444444/us-east-1 update failed. reason: Error validating schemaHandlerPackage. Check the permissions on the bucket and object in your account and try again. (444444444444 = ProdAccount)
Error validating schemaHandlerPackage. Check the permissions on the bucket and object in your account and try again. (use option --print-stack to print stack)
ERROR: Workload CommunityIamPasswordPolicyRP in 555555555555/us-east-1 update failed. reason: Error validating schemaHandlerPackage. Check the permissions on the bucket and object in your account and try again. (555555555555 = SandboxAccount)
Error validating schemaHandlerPackage. Check the permissions on the bucket and object in your account and try again. (use option --print-stack to print stack)
ERROR: Workload CommunityIamPasswordPolicyRP in 666666666666/us-east-1 update failed. reason: Error validating schemaHandlerPackage. Check the permissions on the bucket and object in your account and try again. (666666666666 = SharedServicesAccount)
Error validating schemaHandlerPackage. Check the permissions on the bucket and object in your account and try again. (use option --print-stack to print stack)
ERROR: Workload CommunityIamPasswordPolicyRP in 777777777777/us-east-1 update failed. reason: Error validating schemaHandlerPackage. Check the permissions on the bucket and object in your account and try again. (777777777777 = LogArchiveAccount)
Error validating schemaHandlerPackage. Check the permissions on the bucket and object in your account and try again. (use option --print-stack to print stack)
ERROR: Workload CommunityIamPasswordPolicyRP in 888888888888/us-east-1 update failed. reason: Error validating schemaHandlerPackage. Check the permissions on the bucket and object in your account and try again. (888888888888 = IdentityAccount)
Error validating schemaHandlerPackage. Check the permissions on the bucket and object in your account and try again. (use option --print-stack to print stack)
INFO: Workload CommunityIamPasswordPolicyRP in 000000000000/us-east-1 update successful. (000000000000 = MasterAccount)
ERROR: 
ERROR: ==========================
ERROR: Stopped performing task(s)
ERROR: Following tasks completed: 
ERROR:  - Workload CommunityIamPasswordPolicyRP in 000000000000/us-east-1 (000000000000 = MasterAccount)
ERROR: Following tasks failed: 
ERROR:  - Workload CommunityIamPasswordPolicyRP in 888888888888/us-east-1 (888888888888 = IdentityAccount)
ERROR:  - Workload CommunityIamPasswordPolicyRP in 777777777777/us-east-1 (777777777777 = LogArchiveAccount)
ERROR:  - Workload CommunityIamPasswordPolicyRP in 222222222222/us-east-1 (222222222222 = SecurityToolingAccount)
ERROR:  - Workload CommunityIamPasswordPolicyRP in 333333333333/us-east-1 (333333333333 = BackupAccount)
ERROR:  - Workload CommunityIamPasswordPolicyRP in 666666666666/us-east-1 (666666666666 = SharedServicesAccount)
ERROR:  - Workload CommunityIamPasswordPolicyRP in 555555555555/us-east-1 (555555555555 = SandboxAccount)
ERROR:  - Workload CommunityIamPasswordPolicyRP in 111111111111/us-east-1 (111111111111 = DevAccount)
ERROR:  - Workload CommunityIamPasswordPolicyRP in 444444444444/us-east-1 (444444444444 = ProdAccount)
ERROR: ==========================
ERROR: 
ERROR: Task CommunityIamPasswordPolicyRP execute failed. reason: Number of failed tasks 8 exceeded tolerance for failed tasks 0.
ERROR:

Note that the CommunityIamPasswordPolicyRP update in the Master account succeeded, but not in the member accounts.

msaavedra-earnd commented 2 months ago

Yes, the same thing happened to me. Only one account works and the rest of them fail. I tried many times and I always get the same error. Also it is not the first register type that I add, I have added others and it works. Particularly with these secure-defaults is that I am having problems :(

I tried adding one by one each register type of the secure-defaults and they all fail.

Tried with versions 0.0.x and 1.0.0 and it fails

msaavedra-earnd commented 2 months ago

@OlafConijn @craighurley I was able to run a perform with a --verbose and found this:

INFO: Executing: register-type EbsEncryptionDefaultsRP.
DEBG: Setting build action on register-type / EbsEncryptionDefaultsRP for 12345678900/us-east-2 to UpdateOrCreate - hash from state did not match. (12345678900 = SecurityAccount)
DEBG: Setting build action on register-type / EbsEncryptionDefaultsRP for 12345678900/us-east-2 to UpdateOrCreate - no existing target was found in state. (12345678900 = LogArchiveAccount)
DEBG: Setting build action on register-type / EbsEncryptionDefaultsRP for 12345678900/us-east-2 to UpdateOrCreate - no existing target was found in state. (12345678900 = IdentityAccount)
DEBG: Setting build action on register-type / EbsEncryptionDefaultsRP for 12345678900/us-east-2 to UpdateOrCreate - no existing target was found in state. (12345678900 = EarndDevelopmentAccount)
DEBG: Setting build action on register-type / EbsEncryptionDefaultsRP for 12345678900/us-east-2 to UpdateOrCreate - no existing target was found in state. (12345678900 = EarndTestAccount)
DEBG: Setting build action on register-type / EbsEncryptionDefaultsRP for 12345678900/us-east-2 to UpdateOrCreate - no existing target was found in state. (12345678900 = EarndStagingAccount)
DEBG: Setting build action on register-type / EbsEncryptionDefaultsRP for 12345678900/us-east-2 to UpdateOrCreate - no existing target was found in state. (12345678900 = EarndProductionAccount)
DEBG: Setting build action on register-type / EbsEncryptionDefaultsRP for 12345678900/us-east-2 to UpdateOrCreate - no existing target was found in state. (12345678900 = EarndDeploymentsAccount)
DEBG: Setting build action on register-type / EbsEncryptionDefaultsRP for 12345678900/us-east-2 to UpdateOrCreate - no existing target was found in state. (12345678900 = StagingSCPAccount)
DEBG: Setting build action on register-type / EbsEncryptionDefaultsRP for 12345678900/us-east-2 to UpdateOrCreate - hash from state did not match. (12345678900 = ManagementAccount)

no existing target was found in state for severel accounts and hash from state did not match for a couple of accounts. The strange thing is that this only happens to me with secure-defaults. I have other register types and they work fine.

stefan-karlsson commented 2 weeks ago

Having the same issue, have the permissions on the bucket changed recently? image

OlafConijn commented 2 weeks ago

the permissions have not changed. you indeed do not have permissions to do a list-objects (if you need this, happy to help out!).

you can test your permissions by:

aws s3api head-object --bucket community-resource-provider-catalog --key community-accessanalyzer-analyzer-0.1.0.zip
stefan-karlsson commented 2 weeks ago

the permissions have not changed. you indeed do not have permissions to do a list-objects (if you need this, happy to help out!).

you can test your permissions by:

aws s3api head-object --bucket community-resource-provider-catalog --key community-accessanalyzer-analyzer-0.1.0.zip

Thanks, the command aws s3api head-object --bucket community-resource-provider-catalog --key community-organizations-nodefaultvpc-1.0.0.zip returns the file metadata, so there is some access.

AcceptRanges: bytes
ContentLength: 73649381
ContentType: application/zip
ETag: '"c290b92c266f19ab47f2666d5af6a7d7-9"'
LastModified: '2024-01-23T11:19:01+00:00'
Metadata: {}
ServerSideEncryption: AES256

This is my output from when running npm run perform-tasks as a Administrator on my master account:

ERROR: Workload EbsEncryptionDefaultsRP in 891376944747/eu-north-1 update failed. reason: Error validating schemaHandlerPackage. Check the permissions on the bucket and object in your account and try again. (891376944747 = SecurityAccount)
Error validating schemaHandlerPackage. Check the permissions on the bucket and object in your account and try again. (use option --print-stack to print stack)
ERROR: 
ERROR: ==========================
ERROR: Stopped performing task(s)
ERROR: Following tasks failed: 
ERROR:  - Workload EbsEncryptionDefaultsRP in 891376944747/eu-north-1 (891376944747 = SecurityAccount)
ERROR: Following tasks were not executed: 
ERROR:  - Workload EbsEncryptionDefaultsRP in 211125683270/eu-north-1 (211125683270 = LogArchiveAccount)
ERROR:  - Workload EbsEncryptionDefaultsRP in 654654225256/eu-north-1 (654654225256 = OrgBuildAccount)
ERROR:  - Workload EbsEncryptionDefaultsRP in 637423492960/eu-north-1 (637423492960 = ManagementAccount)
ERROR: ==========================
ERROR: 
ERROR: Task EbsEncryptionDefaultsRP execute failed. reason: Number of failed tasks 1 exceeded tolerance for failed tasks 0.
ERROR: Workload S3PublicAccessBlockRP in 891376944747/eu-north-1 update failed. reason: Error validating schemaHandlerPackage. Check the permissions on the bucket and object in your account and try again. (891376944747 = SecurityAccount)
Error validating schemaHandlerPackage. Check the permissions on the bucket and object in your account and try again. (use option --print-stack to print stack)
ERROR: 
ERROR: ==========================
ERROR: Stopped performing task(s)
ERROR: Following tasks failed: 
ERROR:  - Workload S3PublicAccessBlockRP in 891376944747/eu-north-1 (891376944747 = SecurityAccount)
ERROR: Following tasks were not executed: 
ERROR:  - Workload S3PublicAccessBlockRP in 211125683270/eu-north-1 (211125683270 = LogArchiveAccount)
ERROR:  - Workload S3PublicAccessBlockRP in 654654225256/eu-north-1 (654654225256 = OrgBuildAccount)
ERROR:  - Workload S3PublicAccessBlockRP in 637423492960/eu-north-1 (637423492960 = ManagementAccount)
ERROR: ==========================
ERROR: 
ERROR: Task S3PublicAccessBlockRP execute failed. reason: Number of failed tasks 1 exceeded tolerance for failed tasks 0.
ERROR: Workload NoDefaultVpcRP in 891376944747/eu-north-1 update failed. reason: Error validating schemaHandlerPackage. Check the permissions on the bucket and object in your account and try again. (891376944747 = SecurityAccount)
Error validating schemaHandlerPackage. Check the permissions on the bucket and object in your account and try again. (use option --print-stack to print stack)
ERROR: Workload NoDefaultVpcRP in 211125683270/eu-north-1 update failed. reason: Error validating schemaHandlerPackage. Check the permissions on the bucket and object in your account and try again. (211125683270 = LogArchiveAccount)
Error validating schemaHandlerPackage. Check the permissions on the bucket and object in your account and try again. (use option --print-stack to print stack)
ERROR: Workload NoDefaultVpcRP in 654654225256/eu-north-1 update failed. reason: Error validating schemaHandlerPackage. Check the permissions on the bucket and object in your account and try again. (654654225256 = OrgBuildAccount)
Error validating schemaHandlerPackage. Check the permissions on the bucket and object in your account and try again. (use option --print-stack to print stack)
ERROR: 
ERROR: ==========================
ERROR: Stopped performing task(s)
ERROR: Following tasks failed: 
ERROR:  - Workload NoDefaultVpcRP in 891376944747/eu-north-1 (891376944747 = SecurityAccount)
ERROR:  - Workload NoDefaultVpcRP in 211125683270/eu-north-1 (211125683270 = LogArchiveAccount)
ERROR:  - Workload NoDefaultVpcRP in 654654225256/eu-north-1 (654654225256 = OrgBuildAccount)
ERROR: ==========================
ERROR: 
ERROR: Task NoDefaultVpcRP execute failed. reason: Number of failed tasks 3 exceeded tolerance for failed tasks 0.
ERROR: 
ERROR: ==========================
ERROR: Stopped performing task(s)
ERROR: Following tasks failed: 
ERROR:  - Task EbsEncryptionDefaultsRP
ERROR:  - Task S3PublicAccessBlockRP
ERROR:  - Task NoDefaultVpcRP
ERROR: ==========================
ERROR: 
ERROR: Task RegisterTypes execute failed. reason: Number of failed tasks 3 exceeded tolerance for failed tasks 0.
ERROR: 
ERROR: ==========================
ERROR: Stopped performing task(s)
ERROR: Following tasks failed: 
ERROR:  - Task RegisterTypes
ERROR: Following tasks were not executed: 
ERROR:  - Task PasswordPolicy
ERROR:  - Task SecureDefaults
ERROR:  - Task NoDefaultVpc
ERROR:  - Task OrganizationPolicies
ERROR:  - Task AccessAnalyzer
ERROR: ==========================
ERROR: 
ERROR: Task SecureDefaults execute failed. reason: Number of failed tasks 1 exceeded tolerance for failed tasks 0.
ERROR: 
ERROR: ==========================
ERROR: Stopped performing task(s)
ERROR: Following tasks completed: 
ERROR:  - Task OrganizationUpdate
ERROR:  - Task OrganizationBuild
ERROR:  - Task Types
ERROR:  - Task SeviceControlPolicies
ERROR: Following tasks failed: 
ERROR:  - Task SecureDefaults
ERROR: Following tasks were not executed: 
ERROR:  - Task ServiceQuotas
ERROR:  - Task Budgets
ERROR:  - Task AccountCreation
ERROR:  - Task CloudTrail
ERROR:  - Task GuardDuty
ERROR:  - Task AwsConfigInventory
ERROR:  - Task AwsSso
ERROR: ==========================
ERROR: 
ERROR: Number of failed tasks 1 exceeded tolerance for failed tasks 0.
msaavedra-earnd commented 3 days ago

@stefan-karlsson Did you solve your issue? I got a mail some time ago where it appeared that you did but in this thread I don't see the answer. So I don't know if it really worked for you or not

stefan-karlsson commented 3 days ago

@stefan-karlsson Did you solve your issue? I got a mail some time ago where it appeared that you did but in this thread I don't see the answer. So I don't know if it really worked for you or not

Did not solve it :(

OlafConijn commented 3 days ago

my guess is that you are using a version of the resource providers that does not exist? could you post the task including the value for SchemaHandlerPackage?

thanks

msaavedra-earnd commented 3 days ago

@OlafConijn I am using the same as the template from org-formation-reference


EbsEncryptionDefaultsRP:
  Type: register-type
  SchemaHandlerPackage: !Sub "s3://${catalogBucket}/community-organizations-ebsencryptiondefaults-1.0.0.zip"
  ResourceType: "Community::Organizations::EbsEncryptionDefaults"
  OrganizationBinding:
    IncludeMasterAccount: true
    Account: '*'
    Region: !Ref allRegions # Ebs Encryption Defaults need to be set in all regions.

S3PublicAccessBlockRP:
  Type: register-type
  SchemaHandlerPackage: !Sub "s3://${catalogBucket}/community-s3-publicaccessblock-1.0.0.zip"
  ResourceType: "Community::S3::PublicAccessBlock"
  OrganizationBinding:
    IncludeMasterAccount: true
    Account: '*'
    Region: !Ref allRegions

NoDefaultVpcRP:
  Type: register-type
  SchemaHandlerPackage: !Sub "s3://${catalogBucket}/community-organizations-nodefaultvpc-1.0.0.zip"
  ResourceType: "Community::Organizations::NoDefaultVPC"
  MaxConcurrentTasks: 1000
  OrganizationBinding:
    Region: !Ref allRegions
    Account: "*"
    IncludeMasterAccount: true
OlafConijn commented 3 days ago

thanks interesting.... if i look at the template i see the bucket-name as part of the SchemaHandlerPackage.

in your snippet i see ${catalogBucket}, are you sure this value is getting replaced correctly? e.g. does it work if you change the expression with "community-resource-provider-catalog"

or add a parameters section to the top of your file, like so:

Parameters:
  <<: !Include "../../_parameters.yml"

  catalogBucket:
    Type: String
    Default: community-resource-provider-catalog
msaavedra-earnd commented 3 days ago

Yes, initially I have the bucket name parameterized.

I just tried without parameter (copying the name directly in the SchemaHandlerPackage) and same error. It does not work.


EbsEncryptionDefaultsRP:
  Type: register-type
  SchemaHandlerPackage: s3://community-resource-provider-catalog/community-organizations-ebsencryptiondefaults-1.0.0.zip
  ResourceType: "Community::Organizations::EbsEncryptionDefaults"
  OrganizationBinding:
    IncludeMasterAccount: true
    Account: '*'
    Region: !Ref allRegions # Ebs Encryption Defaults need to be set in all regions.

S3PublicAccessBlockRP:
  Type: register-type
  SchemaHandlerPackage: s3://community-resource-provider-catalog/community-s3-publicaccessblock-1.0.0.zip
  ResourceType: "Community::S3::PublicAccessBlock"
  OrganizationBinding:
    IncludeMasterAccount: true
    Account: '*'
    Region: !Ref allRegions

NoDefaultVpcRP:
  Type: register-type
  SchemaHandlerPackage: s3://community-resource-provider-catalog/community-organizations-nodefaultvpc-1.0.0.zip
  ResourceType: "Community::Organizations::NoDefaultVPC"
  MaxConcurrentTasks: 1000
  OrganizationBinding:
    Region: !Ref allRegions
    Account: "*"
    IncludeMasterAccount: true

Same error:

ERROR: Workload EbsEncryptionDefaultsRP in 123456789012/us-east-2 updated failed. reason: Error validating schemaHandlerPackage. Check the permissions on the bucket and object in your account and try again. (123456789012 = LogArchiveAccount)
Error validating schemaHandlerPackage. Check the permissions on the bucket and object in your account and try again.
CFNRegistryException: Error validating schemaHandlerPackage. Check the permissions on the bucket and object in your account and try again.
OlafConijn commented 2 days ago

right!

(i went back and copied the exact version from the reference into a new project, this did work for me)

what i would suspect at this point is a permission issues: the error seems to happen in 891376944747/eu-north-1 (SecurityAccount).

in the previous comment you ran the following command to check access in the Management account:

aws s3api head-object --bucket community-resource-provider-catalog --key community-organizations-ebsencryptiondefaults-1.0.0.zip

could you do this once more in the SecurityAccount (eu-north-1 region)?
ideally using the "OrganizationFormationBuildAccessRole" in that account, otherwise, perhaps review this role and paste it in the comments?

if that doesnt work, i would be happy to get on a call and try to get to the bottom of this. this error is not very descriptive and happens from time to time, typically doesnt take too long to diagnose!