search

aws-cloudformation / cfn-lint

CloudFormation Linter
MIT No Attribution
2.41k stars 577 forks source link

cfnlint.exceptions.DuplicateRuleError: Rule already included: E2531 #3300

Closed isuftin closed 1 month ago

isuftin commented 1 month ago

CloudFormation Lint Version

cfn-lint 0.87.7

What operating system are you using?

Alpine Linux

Describe the bug

When attempting to validate a JSON CloudFormation template, we are seeing the following error:

Traceback (most recent call last):
  File "/usr/local/bin/cfn-lint", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/site-packages/cfnlint/__main__.py", line 40, in main
    matches = list(cfnlint.core.get_matches(filenames, args))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/cfnlint/core.py", line 175, in get_matches
    (template, rules, errors) = get_template_rules(filename, args)
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/cfnlint/core.py", line 319, in get_template_rules
    _build_rule_cache(args)
  File "/usr/local/lib/python3.11/site-packages/cfnlint/core.py", line 274, in _build_rule_cache
    __CACHED_RULES = cfnlint.core.get_rules(
                     ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/cfnlint/core.py", line 160, in get_rules
    rules.create_from_directory(rules_path)
  File "/usr/local/lib/python3.11/site-packages/cfnlint/rules/__init__.py", line 595, in create_from_directory
    self.extend(result)
  File "/usr/local/lib/python3.11/site-packages/cfnlint/rules/__init__.py", line 304, in extend
    self.register(rule)
  File "/usr/local/lib/python3.11/site-packages/cfnlint/rules/__init__.py", line 291, in register
    raise DuplicateRuleError(rule_id=rule.id)
cfnlint.exceptions.DuplicateRuleError: Rule already included: E2531

The command being run: cfn-lint my-template.json -i "W1011 E7001" -f json

Happens when just running cfn-lint my-template.json

Expected behavior

Template validates without issue.

Reproduction template

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Description": "EBS Volume supporting SonarQube ECS",
  "Resources": {
    "EBS": {
      "Type": "AWS::EC2::Volume",
      "Properties": {
        "AutoEnableIO": false,
        "AvailabilityZone": { "Ref": "AvailabilityZone" },
        "Encrypted": true,
        "KmsKeyId": { "Fn::FindInMap": ["EnvironmentMapping", "kms", "key"] },
        "VolumeType": "st1",
        "Size": 512
      },
      "DeletionPolicy": "Snapshot",
      "UpdateReplacePolicy" : "Retain"
    }
  },
  "Mappings": {
    "EnvironmentMapping": {
      "Fn::Transform": {
        "Name": "AWS::Include",
        "Parameters": {
          "Location": {
            "Fn::Sub": "s3://my-bucket/snippets/${Environment}.json"
          }
        }
      }
    }
  },
  "Outputs": {
    "EBSID": {
      "Description": "Physical ID of EBS Volume",
      "Value": { "Ref": "EBS" }
    }
  },
  "Parameters": {
    "AvailabilityZone": {
      "Description": "Availability Zone for EBS volume",
      "Type": "String",
      "AllowedValues": ["us-west-2a", "us-west-2b", "us-west-2c"],
      "Default": "us-west-2a"
    },
    "Environment": {
      "Description": "Defines the environment that this stack lives in",
      "Type": "String",
      "AllowedValues": [
        "development",
        "test",
        "qa",
        "prod-internal",
        "management"
      ],
      "Default": "test"
    }
  }
}
isuftin commented 1 month ago

Note this also happens when we just try to list rules.

cfn-lint -l
Traceback (most recent call last):
  File "/usr/local/bin/cfn-lint", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/site-packages/cfnlint/__main__.py", line 39, in main
    (args, filenames, formatter) = cfnlint.core.get_args_filenames(sys.argv[1:])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/cfnlint/core.py", line 232, in get_args_filenames
    rules = cfnlint.core.get_rules(
            ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/cfnlint/core.py", line 160, in get_rules
    rules.create_from_directory(rules_path)
  File "/usr/local/lib/python3.11/site-packages/cfnlint/rules/__init__.py", line 595, in create_from_directory
    self.extend(result)
  File "/usr/local/lib/python3.11/site-packages/cfnlint/rules/__init__.py", line 304, in extend
    self.register(rule)
  File "/usr/local/lib/python3.11/site-packages/cfnlint/rules/__init__.py", line 291, in register
    raise DuplicateRuleError(rule_id=rule.id)
cfnlint.exceptions.DuplicateRuleError: Rule already included: E2531
kddejong commented 1 month ago

My guess is somehow we have a cached rule as part of an upgrade process.

I'm curious what files you have in /usr/local/lib/python3.11/site-packages/cfnlint/rules/resources/lmbd/. There was a file rename in the last release. The old file should have been removed during the upgrade process. If it is still there I would be curious about your install and upgrade process.

looks like you installed via pip so lets try pip uninstall cfn-lint The assumption here is that /usr/local/lib/python3.11/site-packages/cfnlint/ should be emptied out. Then run pip install cfn-lint

Another option may be to use pip install cfn-lint --force-reinstall

We actually test for this to in our unit tests but those operate on fresh installs.

isuftin commented 1 month ago

Uninstalling cfn-lint and reinstalling it worked. pip install cfn-lint --force-reinstall did not.

We build our utilities image which includes cfn-lint on top of a bunch of other in-house images - aws-utils -> aws-cli -> python 3.11 -> alpine 3.19

We do install cfn-lint in our aws-cli image because it's required by aws-sam-cli and cloudformation-cli, both of which are installed on that image. Often we just rebuild the aws-utils image when new dependencies (like cfn-lint) are released but we don't always require the upstream aws-cli image to be rebuilt. I think what happened here was the aws-cli had a previous version of cfn-lint in it, then building the utlities image on top installed a newer version of cfn-lint. I would assume that it would just replace all the rules files as well. We do also run "cfn-lint -u" at the tail end of our utilities image build.

Here's what was in /usr/local/lib/python3.11/site-packages/cfnlint/rules/resources/lmbd/ 

/ # ls -al /usr/local/lib/python3.11/site-packages/cfnlint/rules/resources/lmbd/
total 56
drwxr-xr-x    1 root     root          4096 Jun 13 14:20 .
drwxr-xr-x    1 root     root          4096 Jun 13 14:20 ..
-rw-r--r--    1 root     root          2341 Jun 13 14:20 DeprecatedRuntime.py
-rw-r--r--    1 root     root          1782 Jun 13 14:20 DeprecatedRuntimeCreate.py
-rw-r--r--    1 root     root          1431 Jun 13 14:07 DeprecatedRuntimeEnd.py
-rw-r--r--    1 root     root          1873 Jun 13 14:20 DeprecatedRuntimeEol.py
-rw-r--r--    1 root     root          1598 Jun 13 14:20 DeprecatedRuntimeUpdate.py
-rw-r--r--    1 root     root          2097 Jun 13 14:20 EventsLogGroupName.py
-rw-r--r--    1 root     root          2284 Jun 13 14:20 SnapStart.py
-rw-r--r--    1 root     root          1245 Jun 13 14:20 SnapStartEnabled.py
-rw-r--r--    1 root     root          3231 Jun 13 14:20 SnapStartSupported.py
-rw-r--r--    1 root     root          2729 Jun 13 14:20 ZipPackageRequiredProperties.py
-rw-r--r--    1 root     root           106 Jun 13 14:20 __init__.py
drwxr-xr-x    1 root     root          4096 Jun 13 14:20 __pycache__

What's also interesting is that when I run the utilities image, I need to uninstall cfn-lint twice since apparently python has two versions installed?..

/ # pip uninstall cfn-lint -y
Found existing installation: cfn-lint 0.87.7
Uninstalling cfn-lint-0.87.7:
  Successfully uninstalled cfn-lint-0.87.7
/ # pip uninstall cfn-lint -y
Found existing installation: cfn-lint 0.87.6
Uninstalling cfn-lint-0.87.6:
  Successfully uninstalled cfn-lint-0.87.6
/ # pip uninstall cfn-lint -y
WARNING: Skipping cfn-lint as it is not installed.

..assuming 0.87.6 was what was in the upstream image. But I've never seen a python install with two versions of a lib installed at the same time?

isuftin commented 1 month ago

My guess as to why to versions may be installed is because in the aws-cli image which we installed an earlier version of cfn-lint, we're doing so in a slightly weird way to keep the image slightly smaller..

FROM python:3.11 AS builder

COPY requirements.txt /svc/requirements.txt # This had cfn-lint 0.87.6 in it

pip wheel -r /svc/requirements.txt --wheel-dir=/svc/wheels

FROM python:3.11

RUN --mount=type=cache,target=/root/.cache
    --mount=type=cache,from=builder,source=/svc,target=/svc \
    pip install --no-cache-dir --no-index --find-links=/svc/wheels -r /svc/requirements.txt

Let's say we tag that image as aws-cli:latest

So then in our downstream Docker image build we do:

FROM aws-cli:latest

RUN pip install -U cfn-lint && \
     cfn-lint --update-specs

So it may be because in the first image, we're installing from compiled wheels from the local filesystem and in the downstream image we're just using regular pip-install. It could be that pip allows for the same package with two versions being installed in that way.

Not sure, but it does seem like you've identified the proper workaround. Thank you.