search

patrickjennings / logstash-pfsense

Logstash configuration for pfSense syslog events.
93 stars 41 forks source link

Not populating all values for tcp_flags #7

Open jvedman opened 4 years ago

jvedman commented 4 years ago

On multiple instances, I only see tcp_flags values of RA or FA. When the field has A, FPA, or PA (the only ones I can reliably find on search) tcp_flags doesn't populate at all in Elasticsearch. An example search result file (csv) is attached based on the following search: tags:dcSense AND direction:in AND src_ip:(192.168.0. OR 192.168.1. OR 192.168.10. OR 192.168.11. OR 192.168.14. OR 192.168.15.) AND !(src_ip: (fe80 OR 127.0.0.1)) AND !(dest_ip:( 192.168.0. OR 192.168.1.* OR 192.168.0.255 OR 192.168.1.255 OR 192.168.10.255 OR 192.168.11.255 OR 192.168.14.255 OR 192.168.15.255 OR 255.255.255.255)) AND !(tcp_flags:(RA OR FA)) Outbound-Blocked-Traffic.txt

I am using your pfSense grok and conf.d files (though I've made some minor modifications along the way, including to GeoIP tag outgoing traffic). All of that is working fine, I just am not getting the tcp_flags field to populate on values other than RA or FA.

patrickjennings commented 4 years ago

The match on tcp_flags is on the word boundaries. So it should match all of RA, FA, FPA, etc. The fact that you are seeing RA and FA, to me, seems like it's not an issue on the ELK side but on the pfSense side.

Are you sending all syslog events in pfSense or just a subset? Are you seeing any _grokparsefailure tagged messages in your indices?

jvedman commented 4 years ago

So, I am sending all of my traffic to logs, and I am not seeing _grokparsefailure tags on any of the traffic in question. I see the data (in the attached file, they are all PA or FPA) in the data message where it is expected to be, it just doesn't get tagged. But I'm with you in wondering if this is something weird in how pfSense is translating the actual data into human-readable format (maybe there's an ascii issue?) since the grok is for a pretty straight-forward word regex. I'll pursue with them as well. All Clients - Outbound Blocked Traffic (pfSense).txt

Thank you, Patrick. Appreciate your attention!

jvedman commented 4 years ago

Oh, and because I'm curious, I'm wondering if you're seeing the same thing. If you want to check your own logs, look specifically for blocked outbound TCP traffic on your LAN interface(s).

I'm trying to parse out the causes for outbound traffic being blocked on the firewalls I manage. There are lots of possible reasons and it would be a lot easier if I could weed out based on tcp_flags... but... ;-)

patrickjennings commented 4 years ago

The data I have access to right now is from over a year ago. But I do see some documents for FA, RA, FPA, and SA.

Screenshot_20200206_232105

I would be curious to see whether you can capture some raw syslog traffic from pfSense? Either by setting up a secondary output in logstash or tcpdump could be an option.

jvedman commented 4 years ago

Yes, I'm sure I can do that. It might take me a bit, but I'll get it as quickly as I can manage.