Redundant alarms for existing Inspector findings

[achave11-ucsc]

We just got hundreds of alarm messages for an image (kibana/kibana-oss) that we didn't touch. It's a lot of work to triage. Ideally, we would only want to be notify on newly discovered vulnerabilities on existing images and all vulnerabilities for new version of an images.

• [ ] Security design review completed; the Resolution of this issue does not …
  • [ ] … affect authentication; for example:
  • OAuth 2.0 with the application (API or Swagger UI)
  • Authentication of developers with Google Cloud APIs
  • Authentication of developers with AWS APIs
  • Authentication with a GitLab instance in the system
  • Password and 2FA authentication with GitHub
  • API access token authentication with GitHub
  • Authentication with
  • [ ] … affect the permissions of internal users like access to
  • Cloud resources on AWS and GCP
  • GitLab repositories, projects and groups, administration
  • an EC2 instance via SSH
  • GitHub issues, pull requests, commits, commit statuses, wikis, repositories, organizations
  • [ ] … affect the permissions of external users like access to
  • TDR snapshots
  • [ ] … affect permissions of service or bot accounts
  • Cloud resources on AWS and GCP
  • [ ] … affect audit logging in the system, like
  • adding, removing or changing a log message that represents an auditable event
  • changing the routing of log messages through the system
  • [ ] … affect monitoring of the system
  • [ ] … introduce a new software dependency like
  • Python packages on PYPI
  • Command-line utilities
  • Docker images
  • Terraform providers
  • [ ] … add an interface that exposes sensitive or confidential data at the security boundary
  • [ ] … affect the encryption of data at rest
  • [ ] … require persistence of sensitive or confidential data that might require encryption at rest
  • [ ] … require unencrypted transmission of data within the security boundary
  • [ ] … affect the network security layer; for example by
  • modifying, adding or removing firewall rules
  • modifying, adding or removing security groups
  • changing or adding a port a service, proxy or load balancer listens on
• [ ] Documentation on any unchecked boxes is provided in comments below

[dsotirho-ucsc]

I found two notifications for the same finding (matched by the findingArn), posted to our SNS topic about 15 hours apart.

Finding updated Oct 13, 2023, 9:43:50 AM

``` { "deployment": "dev", "event": { "version": "0", "id": "41f4792e-e860-8666-6dd7-a15742fdcf0b", "detail-type": "Inspector2 Finding", "source": "aws.inspector2", "account": "122796619775", "time": "2023-10-13T09:43:50Z", "region": "us-east-1", "resources": [ "arn:aws:ecr:us-east-1:122796619775:repository/docker.elastic.co/kibana/kibana-oss/sha256:b6ce7dc7ff122a4fc14b33552c95800fd71bb659339278a51bca731c2442a861" ], "detail": { "awsAccountId": "122796619775", "description": "addBinding in xmlparse.c in Expat (aka libexpat) before 2.4.3 has an integer overflow.", "exploitAvailable": "YES", "exploitabilityDetails": { "lastKnownExploitAt": "Oct 11, 2023, 5:13:49 AM" }, "findingArn": "arn:aws:inspector2:us-east-1:122796619775:finding/e77225e47ddc9c50e2c30420d273d1f1", "firstObservedAt": "Jun 4, 2023, 8:18:32 PM", "fixAvailable": "YES", "inspectorScore": 9.8, "inspectorScoreDetails": { "adjustedCvss": { "adjustments": [], "cvssSource": "NVD", "score": 9.8, "scoreSource": "NVD", "scoringVector": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H", "version": "3.1" } }, "lastObservedAt": "Oct 13, 2023, 9:43:50 AM", "packageVulnerabilityDetails": { "cvss": [ { "baseScore": 7.5, "scoringVector": "AV:N/AC:L/Au:N/C:P/I:P/A:P", "source": "NVD", "version": "2.0" }, { "baseScore": 9.8, "scoringVector": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H", "source": "NVD", "version": "3.1" } ], "referenceUrls": [ "https://security.gentoo.org/glsa/202209-24", "https://cert-portal.siemens.com/productcert/pdf/ssa-484086.pdf", "https://www.debian.org/security/2022/dsa-5073" ], "relatedVulnerabilities": [], "source": "NVD", "sourceUrl": "https://nvd.nist.gov/vuln/detail/CVE-2022-22822", "vendorCreatedAt": "Jan 10, 2022, 2:12:00 PM", "vendorSeverity": "CRITICAL", "vendorUpdatedAt": "Oct 6, 2022, 3:29:00 PM", "vulnerabilityId": "CVE-2022-22822", "vulnerablePackages": [ { "arch": "X86_64", "epoch": 0, "fixedInVersion": "0:2.2.5-4.el8_5.3", "name": "expat", "packageManager": "OS", "release": "4.el8", "remediation": "dnf update expat", "sourceLayerHash": "sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621", "version": "2.2.5" } ] }, "remediation": { "recommendation": { "text": "None Provided" } }, "resources": [ { "details": { "awsEcrContainerImage": { "architecture": "amd64", "imageHash": "sha256:b6ce7dc7ff122a4fc14b33552c95800fd71bb659339278a51bca731c2442a861", "imageTags": [ "7.10.2-linux-amd64" ], "platform": "CENTOS_8", "pushedAt": "Jun 4, 2023, 8:18:14 PM", "registry": "122796619775", "repositoryName": "docker.elastic.co/kibana/kibana-oss" } }, "id": "arn:aws:ecr:us-east-1:122796619775:repository/docker.elastic.co/kibana/kibana-oss/sha256:b6ce7dc7ff122a4fc14b33552c95800fd71bb659339278a51bca731c2442a861", "partition": "aws", "region": "us-east-1", "type": "AWS_ECR_CONTAINER_IMAGE" } ], "severity": "CRITICAL", "status": "ACTIVE", "title": "CVE-2022-22822 - expat", "type": "PACKAGE_VULNERABILITY", "updatedAt": "Oct 13, 2023, 9:43:50 AM" } } } ```

Finding updated Oct 14, 2023, 1:01:52 AM

``` { "deployment": "dev", "event": { "version": "0", "id": "eec1bcd1-91b3-7b34-cd5a-d14142589a30", "detail-type": "Inspector2 Finding", "source": "aws.inspector2", "account": "122796619775", "time": "2023-10-14T01:01:52Z", "region": "us-east-1", "resources": [ "arn:aws:ecr:us-east-1:122796619775:repository/docker.elastic.co/kibana/kibana-oss/sha256:b6ce7dc7ff122a4fc14b33552c95800fd71bb659339278a51bca731c2442a861" ], "detail": { "awsAccountId": "122796619775", "description": "addBinding in xmlparse.c in Expat (aka libexpat) before 2.4.3 has an integer overflow.", "exploitAvailable": "YES", "exploitabilityDetails": { "lastKnownExploitAt": "Oct 11, 2023, 5:13:49 AM" }, "findingArn": "arn:aws:inspector2:us-east-1:122796619775:finding/e77225e47ddc9c50e2c30420d273d1f1", "firstObservedAt": "Jun 4, 2023, 8:18:32 PM", "fixAvailable": "YES", "inspectorScore": 9.8, "inspectorScoreDetails": { "adjustedCvss": { "adjustments": [], "cvssSource": "NVD", "score": 9.8, "scoreSource": "NVD", "scoringVector": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H", "version": "3.1" } }, "lastObservedAt": "Oct 14, 2023, 1:01:52 AM", "packageVulnerabilityDetails": { "cvss": [ { "baseScore": 7.5, "scoringVector": "AV:N/AC:L/Au:N/C:P/I:P/A:P", "source": "NVD", "version": "2.0" }, { "baseScore": 9.8, "scoringVector": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H", "source": "NVD", "version": "3.1" } ], "referenceUrls": [ "https://security.gentoo.org/glsa/202209-24", "https://cert-portal.siemens.com/productcert/pdf/ssa-484086.pdf", "https://www.debian.org/security/2022/dsa-5073" ], "relatedVulnerabilities": [], "source": "NVD", "sourceUrl": "https://nvd.nist.gov/vuln/detail/CVE-2022-22822", "vendorCreatedAt": "Jan 10, 2022, 2:12:56 PM", "vendorSeverity": "CRITICAL", "vendorUpdatedAt": "Oct 6, 2022, 3:29:48 PM", "vulnerabilityId": "CVE-2022-22822", "vulnerablePackages": [ { "arch": "X86_64", "epoch": 0, "fixedInVersion": "0:2.2.5-4.el8_5.3", "name": "expat", "packageManager": "OS", "release": "4.el8", "remediation": "dnf update expat", "sourceLayerHash": "sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621", "version": "2.2.5" } ] }, "remediation": { "recommendation": { "text": "None Provided" } }, "resources": [ { "details": { "awsEcrContainerImage": { "architecture": "amd64", "imageHash": "sha256:b6ce7dc7ff122a4fc14b33552c95800fd71bb659339278a51bca731c2442a861", "imageTags": [ "7.10.2-linux-amd64" ], "platform": "CENTOS_8", "pushedAt": "Jun 4, 2023, 8:18:14 PM", "registry": "122796619775", "repositoryName": "docker.elastic.co/kibana/kibana-oss" } }, "id": "arn:aws:ecr:us-east-1:122796619775:repository/docker.elastic.co/kibana/kibana-oss/sha256:b6ce7dc7ff122a4fc14b33552c95800fd71bb659339278a51bca731c2442a861", "partition": "aws", "region": "us-east-1", "type": "AWS_ECR_CONTAINER_IMAGE" } ], "severity": "CRITICAL", "status": "ACTIVE", "title": "CVE-2022-22822 - expat", "type": "PACKAGE_VULNERABILITY", "updatedAt": "Oct 14, 2023, 1:01:52 AM" } } } ```

The only differences found between the two JSON are the event ID and date related fields:

(.venv) daniel@Crispin ~/Library/Application Support/JetBrains/PyCharm2023.1/scratches $ diff scratch_96.json scratch_97.json 
5c5
<         "id": "41f4792e-e860-8666-6dd7-a15742fdcf0b",
---
>         "id": "eec1bcd1-91b3-7b34-cd5a-d14142589a30",
9c9
<         "time": "2023-10-13T09:43:50Z",
---
>         "time": "2023-10-14T01:01:52Z",
35c35
<             "lastObservedAt": "Oct 13, 2023, 9:43:50 AM",
---
>             "lastObservedAt": "Oct 14, 2023, 1:01:52 AM",
59c59
<                 "vendorCreatedAt": "Jan 10, 2022, 2:12:00 PM",
---
>                 "vendorCreatedAt": "Jan 10, 2022, 2:12:56 PM",
61c61
<                 "vendorUpdatedAt": "Oct 6, 2022, 3:29:00 PM",
---
>                 "vendorUpdatedAt": "Oct 6, 2022, 3:29:48 PM",
107c107
<             "updatedAt": "Oct 13, 2023, 9:43:50 AM"
---
>             "updatedAt": "Oct 14, 2023, 1:01:52 AM"
(.venv) daniel@Crispin ~/Library/Application Support/JetBrains/PyCharm2023.1/scratches $ 

From AWS documentation, it appears that the vendorCreatedAt and vendorUpdatedAt differences caused the re-notification about this finding.

https://docs.aws.amazon.com/inspector/latest/user/findings-managing-automating-responses.html

Amazon Inspector creates an event for Amazon EventBridge for newly generated findings, newly aggregated findings, and changes in the state of findings. Anything other than a change to the updatedAt and lastObservedAt fields will publish a new event. This means new events for a finding are generated when you take actions such as restarting a resource or changing the tags associated with a resource. However, the finding ID in the id field remains the same. Events are emitted on a best-effort basis.

From the Inspector documentation I do not see a way to configure Inspector so that only new findings are sent to EventBridge.

One possibility I can think of is to filter out events when firstObservedAt and updatedAt do not have the same value, however EventBridge would have to send the event to a Lambda to accomplish this.

Note: We are using an EventBridge event pattern to filter the events sent to our SNS (severity in ['CRITICAL', 'HIGH'] and status == 'ACTIVE'), however the comparison operations for use in event patterns don't allow checking if two field values equal each other.

[achave11-ucsc]

@hannes-ucsc: "During standup, @dsotirho-ucsc proposed a strategy to detect genuinely new findings. I'd like to ask him to formulate it here and use the inspector API to collect examples of findings that would be detected by his strategy."

[dsotirho-ucsc]

Inspector findings contain a few date properties, including firstObservedAt, lastObservedAt, and updatedAt. In newly created findings these three properties have the same value, and in updated findings the firstObservedAt property retains its original value while updatedAt is updated.

Once https://github.com/DataBiosphere/azul/issues/5246 (Route SNS notifications through a Lambda function) is in place, we can use the Lambda to distinguish new findings from updated findings by comparing a finding's firstObservedAt to updatedAt value.

Alternatively, the Lambda could compare the updatedAt date to the current time (using a reasonable buffer window) to determine if the finding is new.

(.venv) daniel@Crispin ~/repo/azul3 $ _select
lrwxr-xr-x  1 daniel  staff  9 Oct 17 14:32 .active -> anvilprod
(.venv) daniel@Crispin ~/repo/azul3 $
(.venv) daniel@Crispin ~/repo/azul3 $ aws inspector2 list-findings --filter-criteria '{"findingStatus" : [{"comparison" : "EQUALS", "value": "ACTIVE"}]}'  | jq '[.findings[] | {"findingArn": .findingArn, "firstObservedAt": .firstObservedAt, "lastObservedAt": .lastObservedAt, "updatedAt": .updatedAt}]' > findings.json
(.venv) daniel@Crispin ~/repo/azul3 $
(.venv) daniel@Crispin ~/repo/azul3 $ head -n 13 findings.json
[
  {
    "findingArn": "arn:aws:inspector2:us-east-1:465330168186:finding/006cd9b8758d2a97e34b4dff5bba7d57",
    "firstObservedAt": 1696873676.504,
    "lastObservedAt": 1696873676.504,
    "updatedAt": 1696873676.504
  },
  {
    "findingArn": "arn:aws:inspector2:us-east-1:465330168186:finding/008f61cda1c49ca0c36b21a4eecb7ef3",
    "firstObservedAt": 1683271917.184,
    "lastObservedAt": 1697242916.251,
    "updatedAt": 1697242916.251
  },
(.venv) daniel@Crispin ~/repo/azul3 $
(.venv) daniel@Crispin ~/repo/azul3 $ grep "findingArn" findings.json | wc -l
     580
(.venv) daniel@Crispin ~/repo/azul3 $
(.venv) daniel@Crispin ~/repo/azul3 $ cat findings.json | jq '.[] | select(.firstObservedAt == .updatedAt)' | grep "findingArn" | wc -l
     217
(.venv) daniel@Crispin ~/repo/azul3 $
(.venv) daniel@Crispin ~/repo/azul3 $ cat findings.json | jq '.[] | select(.firstObservedAt != .updatedAt)' | grep "findingArn" | wc -l
     363
(.venv) daniel@Crispin ~/repo/azul3 $

[dsotirho-ucsc]

@hannes-ucsc: "Sounds good. Let's tackle this once the blocker is resolved."