search

crowdsecurity / hub

Main repository for crowdsec scenarios/parsers
https://hub.crowdsec.net
144 stars 140 forks source link

Unifi OS 3 aren't being parsed #940

Open GNU-Plus-Windows-User opened 5 months ago

GNU-Plus-Windows-User commented 5 months ago

Describe the bug Unifi OS 3 and newer logs are not being parsed correctly, resulting in detection scenarios such as port scanning not working correctly.

To Reproduce

  1. Install the unifi collection cscli collections install crowdsecurity/unifi and reload crowdsec
  2. Setup a syslog endpoint via acquis.yaml using the following yaml: 
    source: syslog
listen_addr: 0.0.0.0
listen_port: 514
labels:
type: unifi
  3. Configure a Unifi OS 3 console or newer to log to the syslog endpoint
  4. check crowdsec logs and see time="03-11-2023 04:48:35" level=error msg="could not parse message: version must be 1" client=0.0.0.0 type=syslog
  5. check cscli metrics and see no logs are being parsed 
    ╭────────────────────┬────────────┬──────────────┬────────────────┬────────────────────────╮
│       Source       │ Lines read │ Lines parsed │ Lines unparsed │ Lines poured to bucket │
├────────────────────┼────────────┼──────────────┼────────────────┼────────────────────────┤
│ syslog:0.0.0.0     │ 295        │ -            │ 295            │ -                      │
╰────────────────────┴────────────┴──────────────┴────────────────┴────────────────────────╯

Expected behavior Logs should be parsed

Screenshots N/A

Additional context This issue was originally reported within the CrowdSec Discord

LaurenceJJones commented 5 months ago

check crowdsec logs and see time="03-11-2023 04:48:35" level=error msg="could not parse message: version must be 1" client=0.0.0.0 type=syslog

so the error is happening within syslog acquisition itself

https://github.com/crowdsecurity/crowdsec/blame/4e724f6c0a54ad1c67eeab6ca3be62f00ee0cf20/pkg/acquisition/modules/syslog/syslog.go#L209

It not even hitting the parsers at all..... so what format is it if its not RFC3164 or RFC5424

Can you post some example lines?

GNU-Plus-Windows-User commented 5 months ago

@LaurenceJJones I'm not sure what to look for, so let me know if you are missing some specific logs.

I didn't remove the MAC address from the last log line, that's how it was sent.

Feb  8 18:19:32 Unifi-Dream-Machine [LAN_LOCAL-RET-2147483647] DESCR="no rule description" IN=br0 OUT= MAC=fake-mac-address SRC=0.0.0.0 DST=0.0.0.0 LEN=40 TOS=00 PREC=0x00 TTL=128 ID=31307 DF PROTO=TCP SPT=54649 DPT=443 SEQ=578136041 ACK=657436146 WINDOW=8195 ACK URGP=0 UID=125 GID=132 MARK=1a0000
Feb  8 18:19:31 Unifi-Dream-Machine [WAN_LOCAL-D-2147483647] DESCR="[WAN_LOCAL]Drop All Other Traf" IN=eth4 OUT= MAC=fake-mac-address SRC=0.0.0.0 DST=0.0.0.0 LEN=40 TOS=00 PREC=0x00 TTL=239 ID=13706 PROTO=TCP SPT=45584 DPT=29552 SEQ=2451790175 ACK=0 WINDOW=1024 SYN URGP=0 MARK=1a0000
Feb  8 18:19:30 Unifi-Dream-Machine [LAN_IN-D-20038] DESCR="Default Implicit Deny" IN=br0 OUT=eth4 MAC=fake-mac-address SRC=0.0.0.0 DST=0.0.0.0 LEN=243 TOS=00 PREC=0x00 TTL=63 ID=3558 DF PROTO=UDP SPT=6537 DPT=6537 LEN=223 MARK=1a0000
Feb  8 18:23:33 Unifi-Dream-Machine [PREROUTING-DNAT-13] DESCR="PortForward DNAT [Reverse Proxy 44" IN=br5 OUT= MAC=fake-mac-address SRC=0.0.0.0 DST=0.0.0.0 LEN=60 TOS=00 PREC=0x00 TTL=64 ID=42142 DF PROTO=TCP SPT=50118 DPT=443 SEQ=746590349 ACK=0 WINDOW=64240 SYN URGP=0 MARK=1a0000
Feb  8 18:23:33 Unifi-Dream-Machine [POSTROUTING-MASQUERADE-14] DESCR="PortForward MASQUERADE [Rev" IN= OUT=br5 MAC= SRC=0.0.0.0 DST=0.0.0.0 LEN=60 TOS=00 PREC=0x00 TTL=63 ID=42142 DF PROTO=TCP SPT=50118 DPT=443 SEQ=746590349 ACK=0 WINDOW=64240 SYN URGP=0 MARK=1a0000
LaurenceJJones commented 5 months ago

Can you capture the raw syslog packet?

The issue at the moment is the syslog acquisition NOT the parser. If you used rsyslog to a file it would work fine.

WhyAydan commented 4 months ago

Hey @LaurenceJJones,

SYSLOG on CrowdSec Node: https://drive.proton.me/urls/SCXVG17A2R#0wuY9TIDGhzc Local TCPDump from UDM SE: https://drive.proton.me/urls/F0SZV6Z4W0#GtsV6AZflD13

Lemme know if you need it in a dif format.

GNU-Plus-Windows-User commented 4 months ago

@WhyAydan Thank you for providing these, I didn't have the time to run a packet capture.

WhyAydan commented 4 months ago

Tbh, no idea if thats what Laurence needs but who knows lol

LaurenceJJones commented 4 months ago

Hmmm it seems to be RFC compliant on my end and within @WhyAydan pcap also

<45>Feb 12 09:52:07 ToonDreamMachine ToonDreamMachine syslog-ng[3459965]: Syslog connection established; fd='28', server='AF_INET(10.72.1.222:514)', local='AF_INET(0.0.0.0:0)'

Still would like a pcap from @GNU-Plus-Windows-User just incase it something we are not seeing

I will do some more testing

WhyAydan commented 4 months ago

Hey, if it helps I also get the same error that @GNU-Plus-Windows-User gets from crowdsec

LaurenceJJones commented 4 months ago

Hey, if it helps I also get the same error that @GNU-Plus-Windows-User gets from crowdsec

Okay, then I try to see if I can reply the packet the syslog endpoint.

LaurenceJJones commented 4 months ago

Also if you get chance can you put the acquisition into debug log level as it should log the reason why the first RFC parser fails

WhyAydan commented 4 months ago 
time="2024-02-12T13:12:46Z" level=debug msg="could not parse as RFC5424 (version must be 1) : <15>Bedroom HIDDEN,USW_FLEX_MINI-2.0.0.704: INFORM: Send notify [setparam] inform to [http://192.168.1.1:8080/inform] Time 1974750" client=192.168.1.230 type=syslog
time="2024-02-12T13:12:47Z" level=debug msg="could not parse as RFC3164 (timestamp is not valid)" client=192.168.1.230 type=syslog
time="2024-02-12T13:12:47Z" level=error msg="could not parse message: version must be 1" client=192.168.1.230 type=syslog
time="2024-02-12T13:12:47Z" level=debug msg="could not parse as RFC5424 (version must be 1) : <15>Bedroom HIDDEN,USW_FLEX_MINI-2.0.0.704: INFORM: Send normal inform to [http://192.168.1.1:8080/inform] Time 1974751" client=192.168.1.230 type=syslog
time="2024-02-12T13:13:06Z" level=debug msg="could not parse as RFC3164 (timestamp is not valid)" client=192.168.1.230 type=syslog
time="2024-02-12T13:13:06Z" level=error msg="could not parse message: version must be 1" client=192.168.1.230 type=syslog
time="2024-02-12T13:13:06Z" level=debug msg="could not parse as RFC5424 (version must be 1) : <15>Bedroom HIDDEN,USW_FLEX_MINI-2.0.0.704: INFORM: Send normal inform to [http://192.168.1.1:8080/inform] Time 1974771" client=192.168.1.230 type=syslog
time="2024-02-12T13:13:24Z" level=debug msg="could not parse as RFC3164 (timestamp is not valid)" client=192.168.1.230 type=syslog
time="2024-02-12T13:13:24Z" level=error msg="could not parse message: version must be 1" client=192.168.1.230 type=syslog
time="2024-02-12T13:13:24Z" level=debug msg="could not parse as RFC5424 (version must be 1) : <15>Bedroom HIDDEN,USW_FLEX_MINI-2.0.0.704: INFORM: Send normal inform to [http://192.168.1.1:8080/inform] Time 1974788" client=192.168.1.230 type=syslog
time="2024-02-12T13:13:44Z" level=debug msg="could not parse as RFC3164 (timestamp is not valid)" client=192.168.1.230 type=syslog
time="2024-02-12T13:13:44Z" level=error msg="could not parse message: version must be 1" client=192.168.1.230 type=syslog
time="2024-02-12T13:13:44Z" level=debug msg="could not parse as RFC5424 (version must be 1) : <15>Bedroom HIDDEN,USW_FLEX_MINI-2.0.0.704: INFORM: Send normal inform to [http://192.168.1.1:8080/inform] Time 1974808" client=192.168.1.230 type=syslog