[DigitalOcean] Notice of SSH bruteforce activities

[dduportal]

Service(s)

ci.jenkins.io, DigitalOcean

Summary

The 21 of April 2023, the Jenkins infrastructure tem received an email from DigitalOcean support about a report of SSH bruteforce attack reported from one of our IPs.

The report is public and visible here: https://digitalocean.abusehq.net/share/binuPbZxK6x1APoWH4Zdrqb9oiD0j9j_0AD4IqVIk4jKhX9FEBsv2pjxFtuj45Zo

The team immediatelly audited the infrastructure:

• The application pods running in the machine pointed by the abuse report are all part of the unique build https://github.com/jenkinsci/bom/pull/1990. The changes, the logs on ci.jenkins.io and the build log does not show anything that could point to a problem with ci.jenkins.io or its agents
• The (exhaustive) analyzis of datadog logs during the 15:29 GMT+2 -> 15:32 GMT+2 period shows only normal logs (the build agent, the associated services collecting data or managing network, etc.)
• The (exhaustive) analyzis of datadog network flows during the same period does not show ANY usage of the SSH protocol on this machine (neither from its lifecycle). It shows the usual HTTPS outbound traffic to JFrog, ACP and the outbound traffic to ci.jenkins.io.
• Metrics does not show any unusual CPU/Disk/Network/Memory usage.
• This machine was automatically removed once the build was finished (ephemeral worker node) and no data is persisted (except the datadog log collected data)
• The only unexpected "monitoring" issue we saw is that our Datadog "process monitoring" integration on Kubernetes is not setup correctly so no data is collected: we can only rely on the default datadog agent process history which is restricted (by default) to only pods/container and kubelet agents: nothing unusual on this matter.

=> We were not able to confirm this abuse report was run from this machine.

We tried to apply a compensating measure by adding firewall rule to the Kubernetes nodes. Alas, the DigitalOcean managed Kuberentes is already managing its own firewall ruleset which allow ALL outbound traffic (defeating any custom restriction) as per https://docs.digitalocean.com/products/kubernetes/details/managed/#worker-node-firewalls

So we decided to disable the cluster on ci.jenkins.io until we hear back from Digital Ocean. Ref. https://github.com/jenkins-infra/jenkins-infra/pull/2780.

We answered back to DigitalOcean support to get more info and support from them.

Reproduction steps

No response

[dduportal]

A few metrics/traces/logs captured during the audit:

[dduportal]

Closing as we received today an email confirming our analysis: Jenkins infrastructure was not used for an SSH bruteforce attack:

Hey there,

Thanks for the update. I'm seeing what happened here. It looks like the reporter shared two different times, causing a bit of confusion on our automation. I've actioned the appropriate account and I'll close this ticket out. Apologies for any trouble it's caused.

Swimmingly,

Security Operations DigitalOcean Security