[Rule Tuning] SSH (Secure Shell) from the Internet

[swiftbird07]

Link to rule

https://github.com/elastic/detection-rules/blob/2fe48e32254c5591470e026f5b2922de5a5f580f/rules/network/command_and_control_ssh_secure_shell_from_the_internet.toml

Description

For me, on Ubuntu 20.04.2 LTS (Focal Fossa) this rule triggers exactly every time I log in via SSH locally and then type sudo -s and my password. The source IP of the alerted event is always a random public Ip X.X.0.0 where X are sometimes the same, but not consistently. The 0.0. is always the same (which makes an attack very unlikely). Also I don't have any respective ports open to even allow SSH from the internet, so this must be a false positive.

Example Data

[threat-punter]

I see the public IP address in the source.address field is registered to Telefonica Brasil S.a. Are you logging in via from another host on your local network or are you logging in via localhost or 127.0.0.1? It looks like your connection request might be getting routed via a gateway with a public IP address.

Source: https://whois.domaintools.com/201.1.0.0

[swiftbird07]

@threat-punter I log in from my Mac in the same network (at home). To be precise I use ssh username@192.168.123.45 -p 2278 This will connect me either directly or if I use the WireGuard tunnel through my OPNsense firewall (but which is in Germany and not open to any ssh stuff as I said)

[threat-punter]

The rule seems to be working as expected and triggering based on the data that is being logged in Elasticsearch (a source IP address that is not in a private IP address range).

Re the routing that is happening/configuration of your network: Have you tried SSH'ing from another host on your LAN to rule out the Mac that you're using? Have you tried disconnecting your firewall/router from the Internet and observing what source.address is logged when you try and SSH across your LAN between the hosts again?

[swiftbird07]

Re the routing that is happening/configuration of your network: Have you tried SSH'ing from another host on your LAN to rule out the Mac that you're using?

Yes I tried it from my Windows Desktop with the same result (just another random IP - now from Egypt, ending again with 0.0)

Have you tried disconnecting your firewall/router from the Internet and observing what source.address is logged when you try and SSH across your LAN between the hosts again?

I tried the above with the normal Wireguard connection and with direct LAN only connection (so no Firewall involved). It just gave me two alerts with the same (public) IP address. I found no mention in the alert of the 192.168.... IP I actually used to connect. I checked the ssh auth.log: Apr 4 01:54:20 pc-s2ubntLT sshd[93202]: Accepted password for martin from 192.168.178.12 port 50659 ssh2 Apr 4 01:54:20 pc-s2ubntLT sshd[93202]: pam_unix(sshd:session): session opened for user martin by (uid=0) Apr 4 01:55:16 pc-s2ubntLT sudo: pam_unix(sudo:auth): Couldn't open /etc/securetty: No such file or directory Apr 4 01:55:22 pc-s2ubntLT sudo: pam_unix(sudo:auth): Couldn't open /etc/securetty: No such file or directory Apr 4 01:55:22 pc-s2ubntLT sudo: martin : TTY=pts/3 ; PWD=/home/martin ; USER=root ; COMMAND=/bin/bash Apr 4 01:55:22 pc-s2ubntLT sudo: pam_unix(sudo:session): session opened for user root by martin(uid=0)

The relevant alert JSON (note there is also the same alert at the same time with "disconnect via" instead of "accepted connection via"):

{
  "_id": "f5f70140041dc670ea876d11e2476f2e7fa52bf9fe379c5f74eefeea1b4c6e99",
  "_index": ".siem-signals-default-000002",
  "_score": "1",
  "_type": "_doc",
  "@timestamp": "2021-04-03T23:59:03.617Z",
  "agent": {
    "id": "399e60ec-06d8-5c84-5ecc-b8c1e26x",
    "type": "endpoint",
    "version": "7.12.0"
  },
  "data_stream": {
    "dataset": "endpoint.events.network",
    "namespace": "default",
    "type": "logs"
  },
  "destination": {
    "address": "192.168.178.83",
    "ip": "192.168.178.83",
    "port": "2278"
  },
  "ecs": {
    "version": "1.6.0"
  },
  "elastic": {
    "agent": {
      "id": "0f68c360-914f-11eb-ad46-6b387fe22345"
    }
  },
  "event": {
    "action": "connection_accepted",
    "category": "network",
    "created": "2021-04-03T23:54:02.216Z",
    "dataset": "endpoint.events.network",
    "id": "M4Kn0KPNmhaBmueR+++6pvpg",
    "ingested": "2021-04-03T23:54:37.241Z",
    "kind": "signal",
    "module": "endpoint",
    "outcome": "success",
    "sequence": "35218229",
    "type": "start"
  },
  "group": {
    "Ext": {
      "real": {
        "id": "0",
        "name": "root"
      }
    },
    "id": "0",
    "name": "root"
  },
  "host": {
    "architecture": "x86_64",
    "hostname": "pc-s2ubntLT",
    "id": "09ae28af385043ddb8372052471bb27e",
    "ip": "10.3.0.5,127.0.0.1,::1,192.168.178.83,fe80::ccfd:681d:9d43:bbc8",
    "mac": "cc:2f:71:47:40:ff,d4:5d:df:01:6c:09",
    "name": "pc-s2ubntLT",
    "os": {
      "Ext": {
        "variant": "Ubuntu"
      },
      "family": "ubuntu",
      "full": "Ubuntu 20.04.2",
      "kernel": "5.8.0-48-generic #54~20.04.1-Ubuntu SMP Sat Mar 20 13:40:25 UTC 2021",
      "name": "Linux",
      "platform": "ubuntu",
      "version": "20.04.2"
    }
  },
  "message": "Endpoint network event",
  "network": {
    "transport": "tcp",
    "type": "ipv4"
  },
  "process": {
    "entity_id": "Mzk5ZTYwZWMtMDZkOC01Yzg0LTVlY2MtYjhjMWUyNjM2NTk2LTI5MDgtMTMyNjE3NTE1NzIuNjM1OTY3MDMA=",
    "executable": "/usr/sbin/sshd",
    "Ext": {
      "ancestry": "Mzk5ZTYwZWMtMDZkOC01Yzg0LTVlY2MtYjhjMWUyNjM2NTk2LTI5MDgtMTMyNjE3NTE1NzIuNjIxODQ5MDMA=,Mzk5ZTYwZWMtMDZkOC01Yzg0LTVlY2MtYjhjMWUyNjM2NTk2LTEtMTMyNjE3NTEyOTcuMA=="
    },
    "name": "sshd",
    "pid": "2908"
  },
  "signal": {
    "_meta": {
      "version": "25"
    },
    "ancestors": "{\"id\":\"5IQomngB5okLWPji7wBT\",\"type\":\"event\",\"index\":\".ds-logs-endpoint.events.network-default-2021.03.27-000001\",\"depth\":0}",
    "depth": "1",
    "original_event": {
      "action": "connection_accepted",
      "category": "network",
      "created": "2021-04-03T23:54:02.216Z",
      "dataset": "endpoint.events.network",
      "id": "M4Kn0KPNmhaBmueR+++6pvpg",
      "ingested": "2021-04-03T23:54:37.241782849Z",
      "kind": "event",
      "module": "endpoint",
      "outcome": "success",
      "sequence": "35218229",
      "type": "start"
    },
    "original_time": "2021-04-03T23:54:02.216Z",
    "parent": {
      "depth": "0",
      "id": "5IQomngB5okLWPji7wBT",
      "index": ".ds-logs-endpoint.events.network-default-2021.03.27-000001",
      "type": "event"
    },
    "parents": "{\"id\":\"5IQomngB5okLWPji7wBT\",\"type\":\"event\",\"index\":\".ds-logs-endpoint.events.network-default-2021.03.27-000001\",\"depth\":0}",
    "rule": {
      "actions": "",
      "author": "Elastic",
      "created_at": "2021-03-30T19:11:03.080Z",
      "created_by": "Martin",
      "description": "This rule detects network events that may indicate the use of SSH traffic from the Internet TO THE NEW PORT. SSH is commonly used by system administrators to remotely control a system using the command line shell. If it is exposed to the Internet, it should be done with strong security controls as it is frequently targeted and exploited by threat actors as an initial access or back-door vector.",
      "enabled": "true",
      "exceptions_list": "{\"id\":\"a6416520-8f3e-11eb-ad46-6b387fe22345\",\"list_id\":\"b8ad2be4-c8c9-40f1-a027-ee6fff8d9d8a\",\"type\":\"detection\",\"namespace_type\":\"single\"}",
      "false_positives": "Some network security policies allow SSH directly from the Internet but usage that is unfamiliar to server or network owners can be unexpected and suspicious. SSH services may be exposed directly to the Internet in some networks such as cloud environments. In such cases, only SSH gateways, bastions or jump servers may be expected expose SSH directly to the Internet and can be exempted from this rule. SSH may be required by some work-flows such as remote access and support for specialized software products and servers. Such work-flows are usually known and not unexpected.",
      "filters": "",
      "from": "now-540s",
      "id": "ab17afb0-918b-11eb-ad46-6b387fe22345",
      "immutable": "false",
      "index": "filebeat-*,packetbeat-*,logs-endpoint.events.*",
      "interval": "5m",
      "language": "kuery",
      "license": "Elastic License v2",
      "max_signals": "100",
      "meta": {
        "from": "4m"
      },
      "name": "SSH (Secure Shell) from the Internet [New Port]",
      "output_index": ".siem-signals-default",
      "query": "event.category:(network or network_traffic) and network.transport:tcp and (destination.port:2278 or event.dataset:zeek.ssh) and not source.ip:( 10.0.0.0/8 or 127.0.0.0/8 or 169.254.0.0/16 or 172.16.0.0/12 or 192.168.0.0/16 or 224.0.0.0/4 or \"::1\" or \"FE80::/10\" or \"FF00::/8\" ) and destination.ip:( 10.0.0.0/8 or 172.16.0.0/12 or 192.168.0.0/16 )",
      "references": "",
      "risk_score": "92",
      "risk_score_mapping": "",
      "rule_id": "03b2ca5a-bf77-4be0-9ac2-556cb8afb075",
      "severity": "critical",
      "severity_mapping": "",
      "tags": "Elastic,Host,Network,Threat Detection,Command and Control",
      "threat": "{\"framework\":\"MITRE ATT&CK\",\"tactic\":{\"id\":\"TA0011\",\"name\":\"Command and Control\",\"reference\":\"https://attack.mitre.org/tactics/TA0011/\"},\"technique\":[]},{\"framework\":\"MITRE ATT&CK\",\"tactic\":{\"id\":\"TA0008\",\"name\":\"Lateral Movement\",\"reference\":\"https://attack.mitre.org/tactics/TA0008/\"},\"technique\":[{\"id\":\"T1021\",\"name\":\"Remote Services\",\"reference\":\"https://attack.mitre.org/techniques/T1021/\",\"subtechnique\":[]}]},{\"framework\":\"MITRE ATT&CK\",\"tactic\":{\"id\":\"TA0001\",\"name\":\"Initial Access\",\"reference\":\"https://attack.mitre.org/tactics/TA0001/\"},\"technique\":[{\"id\":\"T1190\",\"name\":\"Exploit Public-Facing Application\",\"reference\":\"https://attack.mitre.org/techniques/T1190/\",\"subtechnique\":[]}]}",
      "timestamp_override": "event.ingested",
      "to": "now",
      "type": "query",
      "updated_at": "2021-04-03T23:54:02.074Z",
      "updated_by": "Martin",
      "version": "10"
    },
    "status": "open"
  },
  "source": {
    "address": "156.219.0.0",
    "geo": {
      "city_name": "Cairo",
      "continent_name": "Africa",
      "country_iso_code": "EG",
      "country_name": "Egypt",
      "location": "{\"lon\":31.2852,\"lat\":30.0778}",
      "region_iso_code": "EG-C",
      "region_name": "Cairo Governorate"
    },
    "ip": "156.219.0.0",
    "port": "49393"
  },
  "user": {
    "Ext": {
      "real": {
        "id": "0",
        "name": "root"
      }
    },
    "id": "0",
    "name": "root"
  }
}

[peasead]

The rule seems to be working as expected and triggering based on the data that is being logged in Elasticsearch (a source IP address that is not in a private IP address range).

I agree with @threat-punter that the rule is working as intended, I think the question is around how your connections may be getting routed. I'm wondering if WireGuard is doing a hairpin?

@maof97 have you by any chance tried running tcpdump on the server you're connecting to and see what that's telling you?

sudo tcpdump -i [interface] 'port 2278'?

What about disabling WireGuard, OPNsense, etc. and just doing a straight point-to-point connection?

[swiftbird07]

@peasead as I wrote in the last post, I also tried it with point-to-point already with the same result. I will post tcpdump today if I get home from work.

[swiftbird07]

I will post tcpdump today if I get home from work.

Here it is: dump_ssh.log

Btw. I tested and the alert really only triggers if I sudo -i to root after the ssh login. Just using ssh doesn't trigger any alert.

[peasead]

Thanks for the log file. Looking at this, is this the same thing you're doing that's triggering the alert? I ask because it looks like you're sshing to yourself?

The screenshot at the beginning of the Issue shows you sshing to a 192.168. address and the log shows you sshing from 192.168.178.64 to localhost.

23:37:32.552930 IP 192.168.178.64.49751 > localhost.2278:

[swiftbird07]

Thanks for the log file. Looking at this, is this the same thing you're doing that's triggering the alert? I ask because it looks like you're sshing to yourself?

The screenshot at the beginning of the Issue shows you sshing to a 192.168. address and the log shows you sshing from 192.168.178.64 to localhost.

23:37:32.552930 IP 192.168.178.64.49751 > localhost.2278:

Yes I ssh from my Mac in this case (192.168.178.64) to my Server (192.168.178.83) from which I used tcpdump. The server (or more like a laptop that is now deemed to be a server) is in my local home network and also physically at home. Sorry if that was confusing 😅

[dcode]

@maof97 I tried to recreate your environment using Vagrant and some provisioning scripts in this repo, but when SSH'ing from my host box to a VM, or from VM to VM using Wireguard, I wasn't able to replicate the public ip source addresses that you're seeing.

That said, there's plenty of variables that could be different between my setup and yours. Which leads me to some questions:

1. In the tcpdump output, I don't understand why you have packets going from a interface address (i.e. 192.168.178.64) to loopback (i.e. localhost).. that doesn't usually happen. Can you share your routing table on your system named pc-s2ubntLT using ip route?
2. Is your wireguard setup similar to mine in the provisioning script? Namely, are you using the Ubuntu packages and kernel module? There are some user-space only implementations that don't require kernel modules at all. This could make the packets take weird routes.
3. If you run my Vagrantfile, do you get the same results as you noted in this issue? If not, can you tweak the environment to be representative of your configuration so we can reproduce it?

All those things said, as @peasead and @threat-punter noted, the detection is working correctly. Instead what we want to determine is: why is your endpoint logging the event that way?

Thanks!

[swiftbird07]

1. In the tcpdump output, I don't understand why you have packets going from a interface address (i.e. 192.168.178.64) to loopback (i.e. localhost).. that doesn't usually happen. Can you share your routing table on your system named pc-s2ubntLT using ip route?

This is the ip-route that is in use currently (with Wireguard activated). I added some routes in the wg-config file to actually make it work.

default via 192.168.178.1 dev enp3s0 proto dhcp metric 100 10.3.0.0/24 dev wg0 scope link 169.254.0.0/16 dev enp3s0 scope link metric 1000 192.168.178.0/24 dev enp3s0 proto kernel scope link src 192.168.178.83 metric 100

I will come back to the other questions tomorrow.

[swiftbird07]

@dcode

Sorry for the late reply.

I installed Wireguard with the normal apt-get install wireguard command. But as I said, the problem exist if I use ssh without Wireguard even activated.

[peasead]

Hey @maof97, thanks for your patience and I hope you had a good weekend.

I think this may be a bug in the endpoint. We're still collecting some info, but there may be someone else with a similar issue.

What I'm doing to do and recommend:

1. I'm going to close this Issue as the rule is working as intended
2. I'm going to recommend you open a bug Issue here.
3. I'm going to link this Issue with your bug Issue
4. I'll continue to watch this and provide any commentary the Endpoint team may need.

If you disagree, please feel free to re-open this and discuss.

[swiftbird07]

@peasead Thanks for the response. The provided link 404s for me. Instead I created the issue manually here: https://github.com/elastic/endpoint/issues/6

[peasead]

Sorry about that 404 @maof97. I was trying to be too fancy, I guess.

Can you move that Issue here- https://github.com/elastic/endpoint-dev/issues/new/choose

[swiftbird07]

Sorry about that 404 @maof97. I was trying to be too fancy, I guess.

Can you move that Issue here- https://github.com/elastic/endpoint-dev/issues/new/choose

Still 404s sorry. I guess I don't have access to elastic/endpoint-dev/ (can't find that in search either)

[peasead]

Still 404s sorry. I guess I don't have access to elastic/endpoint-dev/ (can't find that in search either)

Ah, got it. I'll take care of it.