Support registry resource schemas

softprops commented 4 years ago

My company uses this tool to ensure correctness of our CloudFormation templates and would like to explore CloudFormation's new registry for third party resources. In particular, we'd are trialing those provided by datadog.

Understandably cfn-lint is not currently aware of the schema's these declare for custom resources but I was wondering if adding support for them would possible given the current schema definition formation cfn-lint relies on? The problem for us is that cfn-lint will always fail for templates containing these resources. The current work around is to put them into separate files and use a naming convention that cfn-lint avoids but I don't think that's an ideal approach for the way CloudFormation registry resources are intended to be used.

Strawman idea. Add a new multi value parameter the s3 schema uris these resources are required to publish to cf-lint and have it incorporate those in its validation. This allows for organizations to change resource schema versions of these resources and to update s3 uris provided to cfn lint in sync

kddejong commented 4 years ago

@softprops how would you feel about a cfn-lint command that would update a patch file from that registry? I'm not sure I would like to do it with --update-specs at this point because we have tried to make that command to not require authentication to AWS. @PatMyron any thoughts here?

softprops commented 4 years ago

I don't know enough implementation details to know what that does. Rather than focus on implementation I'll try to make clearer the goal. I would like to have a way to continue to use cfn-lint in ci and not fail when it encounters a cfn template for a custom resource.

A common property of all third party resources is that they all require hosting of a schema file describing that resource. What would be great if there was a way for cfn to leverage that type to schema mapping to adapt aws third party resource schema model to what ever schema model cfn lint uses internally.

How does the patch file system work?

azec-pdx commented 4 years ago

Would like to contribute on this as another user of this feature. Our case is same, as I described in https://github.com/aws-cloudformation/cfn-python-lint/issues/1401

azec-pdx commented 4 years ago

Did a reading on comment and it seems like use of Customized Specifications with --override-spec is not helpful for our cases. It ignored my Third-Party schema as it probably couldn't find existing to merge it with and linting resulted in more errors.

PatMyron commented 4 years ago

There are now publicly accessible CloudFormation Registry resource provider schemas for all of the PUBLIC resource types in Amazon's namespaces

The Linter has a lot of Amazon resource type information that is not available in these schemas that we don't want to lose:

public resource types not yet fully migrated to the CloudFormation Registry are lacking most schema information. primaryIdentifier may even be incorrect for these types
constantly evolving enums like instance types, IAM actions, lambda runtimes, partitions, regions, availability zones, etc. don't make sense to maintain in CloudFormation resource schemas as blocking validations, especially with the significant time and effort to deploy changes to AWS' most regulated regions
some schema information doesn't match up exactly as previously expected due to technical limitations: writeOnlyProperties, readOnlyProperties, createOnlyProperties
etc.

Resource schemas definitely also contain information the Linter does not currently check, so it's worth figuring out how to start utilizing them as well. In order to not lose any existing information checked not available in these schemas, it probably makes sense to start utilizing resource schemas writing additional new rule(s)

The trickiest part will be handling CloudFormation template syntax like dynamic references that cannot be resolved. Some things like combined JSON schemas will make it particularly tricky to properly ignore not yet resolvable template syntax because one property utilizing unresolvable template syntax may also hinder our ability to easily validate other properties of that resource as well. In order to reduce false positive noise rates, we should err on the side of caution and stop evaluating in circumstances like those where schema validation may fail simply due to unresolved template syntax

Once the initial framework utilizing these resource schemas is in place, expanding support for this rule beyond Amazon's PUBLIC resource type namespaces should be trivial: we should able to provide the ability for customers to provide their own directory of resource schemas or something similar easily. Even including some of the most popular non-PUBLIC resource types by default might make sense as well

As far as current errors from non-PUBLIC CloudFormation Registry type usage, I believe it should be limited to this existing rule, E3001, that we need to re-evaluate:

https://github.com/aws-cloudformation/cfn-lint/blob/93659fba5cf86ee34177c55b28df0aa9140b7dc9/src/cfnlint/rules/resources/Configuration.py#L105

Resource-level ignores should mitigate these errors until then:

  Resource:
    Type: Datadog::IAM::User
    Metadata:
      cfn-lint:
        config:
          ignore_checks:
          - E3001

kddejong commented 4 years ago

Here are a few options we have as we continue to think about this issue.

One solution would be to create a way to convert Resource Provider Schema into CloudFormation Resource Spec format. Our current rules will continue to work as needed. I don't feel like this is a sustainable long term answer as the Schemas continue to evolve we will continually run into issues as things that won't convert correctly to the CloudFormation Resource Spec format. The Registry Schema has additional data that we could leverage for checks that we would be missing.
Rewrite the existing rules to leverage the the new Resource Provider Schema. This is going to take some work as some of the rules need to be rewritten. Ideally we could use a JSON Schema validation package to do most of that heavy lifting and providing additional logic on top of that package based on our needs. This is probably the most ideal but will take the most time. Additionally we may have to think about how people write custom rules and the impact that will result from PropertyTypes not existing any more. Those names may not match the property names so there is a chance things will break as we lose those names.

PatMyron commented 3 years ago

With Release v0.43.0, one or more directories of CloudFormation Registry Resource Schemas can be input with:

-s / --registry-schemas

aws-cloudformation / cfn-lint

Support registry resource schemas #1277