SlackPublisher: Requests Exceed Rate Limit

msymons commented 3 years ago

Current Behavior:

After setting up Slack notifications on DT 4.3.1 I have seen 269 occurrences of HTTP 429 ERROR in the space of 24 hours. I cannot say that the errors did not occur in earlier versions of DT, but my logs go back only 3 months... to two days after I disabled notifications because DT 3.8.0 had problems with CycloneDX 1.2 BOMs.

10:51:54.488 INFO [OssIndexAnalysisTask] Analyzing 82 component(s)
10:51:55.724 ERROR [SlackPublisher] An error was encountered publishing notification to Slack
10:51:55.725 ERROR [SlackPublisher] HTTP Status : 429 Too Many Requests
10:51:55.725 ERROR [SlackPublisher] Destination: https://hooks.slack.com/services/XXXXXXXXX/YYYYYYYYY/ZZZZZZZZZZZZZZZZZZZZZZZZ
10:51:55.732 ERROR [SlackPublisher] An error was encountered publishing notification to Slack
10:51:55.732 ERROR [SlackPublisher] HTTP Status : 429 Too Many Requests
10:51:55.732 ERROR [SlackPublisher] Destination: https://hooks.slack.com/services/XXXXXXXXX/YYYYYYYYY/ZZZZZZZZZZZZZZZZZZZZZZZZ
10:51:55.796 ERROR [SlackPublisher] An error was encountered publishing notification to Slack
10:51:55.796 ERROR [SlackPublisher] HTTP Status : 429 Too Many Requests
10:51:55.796 ERROR [SlackPublisher] Destination: https://hooks.slack.com/services/XXXXXXXXX/YYYYYYYYY/ZZZZZZZZZZZZZZZZZZZZZZZZ
10:51:56.076 ERROR [SlackPublisher] An error was encountered publishing notification to Slack
10:51:56.076 ERROR [SlackPublisher] HTTP Status : 429 Too Many Requests
10:51:56.076 ERROR [SlackPublisher] Destination: https://hooks.slack.com/services/XXXXXXXXX/YYYYYYYYY/ZZZZZZZZZZZZZZZZZZZZZZZZ
10:51:56.421 INFO [OssIndexAnalysisTask] Analyzing 50 component(s)
10:51:56.421 INFO [OssIndexAnalysisTask] Sonatype OSS Index analysis complete
10:51:56.425 INFO [PolicyEngine] Evaluating 202 component(s) against applicable policies
10:51:56.498 ERROR [SlackPublisher] An error was encountered publishing notification to Slack
10:51:56.498 ERROR [SlackPublisher] HTTP Status : 429 Too Many Requests
10:51:56.498 ERROR [SlackPublisher] Destination: https://hooks.slack.com/services/XXXXXXXXX/YYYYYYYYY/ZZZZZZZZZZZZZZZZZZZZZZZZ

In #322 (logged long ago against DT 3.4.0), it was stated:

In its current form, notification are a stream of in-memory events. Once the events have been consumed by any event listeners that care, they are quickly GCed. They are not persistent and reside only in memory.

There is nothing logged to state that the notifications have been retried and thus it would appear that they have poofed, never to be delivered. This is problematic because, without notifications, it can be hard to know what new vulnerabilities have been detected most recently in the portfolio. This is something that I was attempting to deal with via a different route when suggesting enhancement #1137.

Steps to Reproduce:

Create a standard slack notification (mine used NEW_VULNERABILITY only)
Create a new project
Upload a BOM that is designed to generate a lot of alerts. My one came from a maven project containing nothing but two dependencies:

    <dependencies>
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
            <version>2.7.3</version>
        </dependency>
        <dependency>
            <groupId>org.keycloak</groupId>
            <artifactId>keycloak-core</artifactId>
            <version>1.8.0.Final</version>
        </dependency>
    </dependencies>

Check the log. When I uploaded this test BOM (which had 9 components in total including transitive dependencies) it generated an additional 45 Slack Too Many Requests events in the space of 2 seconds. (DT reported 100 vulnerabilities).

Expected Behavior:

Basic expectation is that the number of slack notifications received should match the number that should have been received

I know that this did not happen because Slack reported...

Environment:

Dependency-Track Version: 4.3.1
Distribution: [ Docker ]
BOM Format & Version: CycloneDX 1.2 generated using cyclonedx-maven-plugin v2.5.2
Database Server: [ PostgreSQL ]
Browser: Firefox

Additional Details:

The link in the screenshot above is to Slack documentation on Rate Limits.

This makes me wonder if maybe the way to approach things is perhaps to think about switching to use Slack Web API as suggested in #776

msymons commented 2 years ago

Testing in v4.4.1 shows that the rate throttling happens more often now... but only when GHSA integration is enabled: it means that far more individual vulnerabilities are creating two separate alerts.

msymons commented 1 year ago

Issue is still occurring in v4.9.0 SNAPSHOT. Log entry has changed:

2023-09-06 19:57:19,420 ERROR [SlackPublisher] An error was encountered publishing notification to Slackwith HTTP Status : 429 Too Many Requests Destination: https://hooks.slack.com/services/xxxx/yyyy/zzzz Response: {"retry_after":1,"ok":false,"error":"rate_limited"}

Note that the "retry_after" is 1 second. This seems to be more strictly limited than when this issue was logged in 2021.

DependencyTrack / dependency-track