search

philhagen / sof-elk

Configuration files for the SOF-ELK VM
GNU General Public License v3.0
1.46k stars 272 forks source link

NetFlow UDP 9995 not listening #320

Closed stijnos1991 closed 5 months ago

stijnos1991 commented 5 months ago

Hello! I recently installed sof-elk to capture some NetFlow data transmitted from an OPNsense router. Via fw_modify.sh -a open -p 9995 -r udp as root I opened the port on the local firewall. But netstat -tulpn does not show a listener on port 9995. I verified the logstash config file is present (/usr/local/sof-elk/configfiles/0200-input-netflow.conf) and seems correct which defines the input. Am I missing something here?

Kind regards,

philhagen commented 5 months ago

Hm, that certainly doesn't sound right. I just tested on one here:

[elk_user@sof-elk ~]$ sudo netstat -tulpn | grep 9995
udp        0      0 0.0.0.0:9995            0.0.0.0:*                           2128/java

FWI, that should be reported (with a different PID, of course) even if the firewall is not open. I can't think of a reason that port would not be listening unless the entire logstash pipeline was crashed. Do you have anything in /var/log/logstash/logstash-plain.log that may provide insight? You should be able to search for 9995 to get close.

I have the following on the same system as the netstat command was run:

[2023-10-27T20:18:49,410][INFO ][logstash.inputs.udp      ][main][945ed0156c99a0f45b4236702e814c70f307cd0e60221a10d3f8b12aa3bc11b0] Starting UDP listener {:address=>"0.0.0.0:9995"}
philhagen commented 5 months ago

Also, the next release of the VM will use a new NetFlow ingest via filebeat. Not a help now, but that is running reliably in dev at this time.

stijnos1991 commented 5 months ago

Thanks for your help. I see some warnings about a cache file not being present in the logstash-plain.log. I'll do a reinstall first to exclude some weird user error on my side. Will report back when done to see if this issue was fixed for me.

philhagen commented 5 months ago

sure thing. if you're seeing a template cache error, that's expected.

[2023-10-27T20:17:39,274][WARN ][logstash.codecs.netflow  ] Template Cache does not exist {:file_path=>"/var/lib/logstash//netflow_templates.cache"}

let me know what you find!

stijnos1991 commented 5 months ago

On a clean install the listener is coming up just fine. Opened the port and the NetFlow data is received! No idea what was wrong on the previous install, but... fixed! Thanks for your pointers :)

philhagen commented 5 months ago

OK great! Glad to hear. That input has historically been quite stable, but has become a little flaky recently. I suspect that's somewhat related to Elastic's move from the native input to Filebeat - which we'll have enabled instead for the future version (along with proper NetFlow v9 handling and much more!)

Glad it's working!