Issue receiving Syslog from Palo Alto

josephtillman11 commented 3 years ago

Describe the problem

I have configured my Palo Alto firewall to forward traffic logs via syslog to the HELK ubuntu server. I've tried UDP and TCP port 8516 as noted in the syslog input file located in /helk-logstash/pipeline. I created an output file 9960-syslog-output.conf based off of an answer to a previous issue on here. I send the syslogs but nothing ever shows up in Kibana.

Provide the output of the following commands

Get operating system and version for linux (except Mac) use:
cat /etc/os-release
for Mac/OSX use:
sw_vers
Get disk space, memory, processor cores, and docker storage
echo -e "\nDocker Space:" && df -h /var/lib/docker; echo -e "\nMemory:" && free -g; echo -e "\nCores:" && getconf _NPROCESSORS_ONLN
Get output of the HELK docker containers:
docker ps --filter "name=helk"

NAME="Ubuntu"
VERSION="18.04.3 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.3 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

Docker Space:
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda2       1.4T   24G  1.3T   2% /

Memory:
              total        used        free      shared  buff/cache   available
Mem:             31          22           0           0           8           7
Swap:             7           0           7

Cores:
8

CONTAINER ID   IMAGE                                                 COMMAND                  CREATED        STATUS        PORTS                                                                                                                                                                                                  NAMES
54248d9b8e25   confluentinc/cp-ksql-cli:5.1.3                        "/bin/sh"                29 hours ago   Up 29 hours                                                                                                                                                                                                          helk-ksql-cli
7d52c6057170   confluentinc/cp-ksql-server:5.1.3                     "/etc/confluent/dock…"   29 hours ago   Up 29 hours   0.0.0.0:8088->8088/tcp                                                                                                                                                                                 helk-ksql-server
af4ac26b7e81   otrf/helk-kafka-broker:2.4.0                          "./kafka-entrypoint.…"   29 hours ago   Up 29 hours   0.0.0.0:9092->9092/tcp                                                                                                                                                                                 helk-kafka-broker
e36ecaaabb77   otrf/helk-zookeeper:2.4.0                             "./zookeeper-entrypo…"   29 hours ago   Up 29 hours   2181/tcp, 2888/tcp, 3888/tcp                                                                                                                                                                           helk-zookeeper
731aa1ade1c2   otrf/helk-elastalert:latest                           "./elastalert-entryp…"   29 hours ago   Up 29 hours                                                                                                                                                                                                          helk-elastalert
4a552f660092   otrf/helk-nginx:0.3.0                                 "/opt/helk/scripts/n…"   29 hours ago   Up 29 hours   0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp                                                                                                                                                               helk-nginx
6492e012589d   otrf/helk-logstash:7.6.2.1                            "/usr/share/logstash…"   29 hours ago   Up 29 hours   0.0.0.0:3515->3515/tcp, 0.0.0.0:5044->5044/tcp, 0.0.0.0:5514->5514/tcp, 0.0.0.0:5514->5514/udp, 0.0.0.0:8515-8516->8515-8516/tcp, 0.0.0.0:8531->8531/tcp, 0.0.0.0:8515-8516->8515-8516/udp, 9600/tcp   helk-logstash
b0d5d9737375   docker.elastic.co/kibana/kibana:7.6.2                 "/usr/share/kibana/s…"   29 hours ago   Up 29 hours   5601/tcp                                                                                                                                                                                               helk-kibana
ea8b1bfaa203   docker.elastic.co/elasticsearch/elasticsearch:7.6.2   "/usr/share/elastics…"   29 hours ago   Up 29 hours   9200/tcp, 9300/tcp                                                                                                                                                                                     helk-elasticsearch

What version of HELK are you using

run the command from within the HELK root directory cat .git/refs/heads/master
and include what date you cloned the HELK repo

8f5643e5457b0b903107670b167ce39a8e6cd713

Cloned repo 12/22/2020

What steps did you take trying to fix the issue

Googling, tried what was recommended on an earlier issue request, tried UDP and TCP, tried various ports.

priamai commented 3 years ago

Yes syslog PaloAlto should be supported:

  tcp {
    port => 8516
    type => "syslog-paloalto"
    add_field => {
      "etl_input_port" => 8516
      "etl_input_protocol" => "tcp"
      "etl_input_application_protocol" => "syslog"
      "etl_pipeline" => "syslog-tcp-input-paloalto"
    }
    #id => "syslog-tcp-8516-0001"
  }

Can you show us how did you configure your palo alto firewall to forward the syslogs? I suggest 2 things:

first use netcat to see if you can receive events (without running HELK): nc -l 0.0.0.0 8516 > palo-alto.syslog
second use tcpdump when running HELK: sudo tcpdump -i eth0 -n tcp port 8516 -vvv -w palo-alto.pcap Make sure tcpdump is listening to the right interface. Share your outputs here.

Cheers!

josephtillman11 commented 3 years ago

For the Palo Alto configuration, we already had logs being sent to another server. So I just changed the IP and port to match the HELK server.

When I try to run netcat I get the following:

root@bigbrother:/# nc -l 0.0.0.0 8516 > palo-alto.syslog
nc: Address already in use

When I do the tcpdump, packets are captured:

root@bigbrother:/# sudo tcpdump -i ens3 -n tcp port 8516 -vvv -w palo-alto.pcap
tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
^C85 packets captured
113 packets received by filter
0 packets dropped by kernel
root@bigbrother:/# sudo tcpdump -ttttnnr palo-alto.pcap
reading from file palo-alto.pcap, link-type EN10MB (Ethernet)
2020-12-29 13:06:41.060385 IP 10.0.14.254.42137 > 10.0.14.207.8516: Flags [P.], seq 3888362345:3888362886, ack 2312911279, win 115, options [nop,nop,TS val 1476304996 ecr 3452743587], length 541
2020-12-29 13:06:41.060426 IP 10.0.14.254.42137 > 10.0.14.207.8516: Flags [.], seq 541:1989, ack 1, win 115, options [nop,nop,TS val 1476304996 ecr 3452743587], length 1448
2020-12-29 13:06:41.060481 IP 10.0.14.207.8516 > 10.0.14.254.42137: Flags [.], ack 541, win 2474, options [nop,nop,TS val 3452744498 ecr 1476304996], length 0
2020-12-29 13:06:41.060529 IP 10.0.14.207.8516 > 10.0.14.254.42137: Flags [.], ack 1989, win 2466, options [nop,nop,TS val 3452744498 ecr 1476304996], length 0
2020-12-29 13:06:41.060632 IP 10.0.14.254.42137 > 10.0.14.207.8516: Flags [P.], seq 1989:2896, ack 1, win 115, options [nop,nop,TS val 1476304996 ecr 3452744498], length 907
2020-12-29 13:06:41.060677 IP 10.0.14.207.8516 > 10.0.14.254.42137: Flags [.], ack 2896, win 2474, options [nop,nop,TS val 3452744499 ecr 1476304996], length 0
2020-12-29 13:06:41.114675 IP 10.0.14.254.42137 > 10.0.14.207.8516: Flags [P.], seq 2896:3435, ack 1, win 115, options [nop,nop,TS val 1476305002 ecr 3452744499], length 539
2020-12-29 13:06:41.114758 IP 10.0.14.207.8516 > 10.0.14.254.42137: Flags [.], ack 3435, win 2474, options [nop,nop,TS val 3452744553 ecr 1476305002], length 0
...
...

priamai commented 3 years ago

Great, the netcat was a a test to check if the syslogs were coming when HELK was switched off, this is why it gives you an error. Now this is good it shows that there are indeed packets coming from your firewall. I want to show you know a better way to debug Logstash. Login into HELK and follow the monitoring pipeline

You will then see your inputs:

Sadly the inputs have no sequence id so it is confusing (we should improve this).

If you scroll down we can see the filters:

Please let the syslog run and see where the messages are flowing in the filter panel. This will let us understand if there is a parsing error at some stage and gets dropped.

Let me know and post the screenshots here. Cheers.

josephtillman11 commented 3 years ago

There are so many (hundreds) of filters listed here I am not quite sure what I am looking for. I have several servers using winlogbeats so I don't know what is from them and what isnt. I dont see any filters about palo alto or syslog?

josephtillman11 commented 3 years ago

I found them! I had to change the index on the left to "indexme*" what does this mean? Also it didn't really parse the fields. No IPs, Ports, etc.

priamai commented 3 years ago

When debugging Logstash pipelines I usually turn off all the other sources and leave only the one under observation. Otherwise as you said can be quite challenging!

Now that is good news however I would expect to see them in this index indexme-syslog-*:

    # Unparsed syslog
    else if [@metadata][helk_parsed] != "yes" and [etl_input_application_name] =~ "^syslog" {
      elasticsearch {
        hosts => ["helk-elasticsearch:9200"]
        index => "indexme-syslog-%{etl_input_port}-%{+YYYY.MM.dd}"
        document_id => "%{[@metadata][log_hash]}"
        user => 'elastic'
        #password => 'elasticpassword'
      }
    }

Could you check in the drop-down whether they show up?

Also I found an interesting project here. The logstash filter can be easily integrated into HELK pipeline: https://github.com/shadow-box/Palo-Alto-Networks-ELK-Stack/blob/master/PAN-OS.conf

It seems like you can just enable CSV output into your PAN firewall and then apply that filter. Happy to help but I will need your syslog to replay and test. Cheers!

priamai commented 3 years ago

There is also a more recent tutorial here where they parse the log structure (without CSV conversion) via simple regexes.

Cyb3rWard0g commented 3 years ago

Thank you very much @priamai ! I appreciate you taking the time to help troubleshoot. 👍

priamai commented 3 years ago

Hey @Cyb3rWard0g my pleasure, you know I was thinking to make the Logstash files more easy to debug and visualize we should convert everything to include the pipeline directive. That way we don't have a huge filter list block which will be a challenge as we expand the project. There is a nice example here: https://github.com/priamai/fortinet-2-elasticsearch/tree/master/logstash

PS: @priamai and @robomotic are both me.

Cheers.

Cyb3rWard0g / HELK