search

elastic / integrations

Elastic Integrations
https://www.elastic.co/integrations
Other
186 stars 391 forks source link

[Citrix ADC] Syslog messages are not according to documentation #10153

Open fgierlinger opened 2 weeks ago

fgierlinger commented 2 weeks ago

As @eriroley mentioned in #9592 , the Citrix ADC messages received over syslog are missing the fields

even though the documentation of citrix adc clearly states, that the logs are to be expected in this format. Therefore the grok pattern fails. We double checked all the settings, but the log message format is not configurable for adc logs.

As of our understanding, the logfile format with the timestamp, level and client ip is only valid for logs saved to the /var/log folder.

https://github.com/elastic/integrations/blob/7cdf5800aed7b680040eaba311f997d54407726b/packages/citrix_adc/data_stream/log/elasticsearch/ingest_pipeline/default.yml#L26-L37

Here are some example messages:

<134> 06/14/2024:18:29:23 GMT host1 0-PPE-1 : default REWRITE Message 209051714 0 :  "Source: 1.1.1.1 Destination: 192.168.0.1 Request URL: site.example.com/webcam/1.jpg HTTP Status: 200"
<134> 06/14/2024:16:02:01 GMT host1 0-PPE-0 : default TCP CONN_TERMINATE 225601598 0 :  Source 192.168.0.1:443 - Destination 192.168.0.2:15567 - Start Time 06/14/2024:16:02:01 GMT - End Time 06/14/2024:16:02:01 GMT - Total_bytes_send 0 - Total_bytes_recv 1
<134> 06/14/2024:16:01:40 GMT host1 0-PPE-0 : default TCP CONN_DELINK 220659229 0 :  Source 1.1.1.1:48277 - Vserver 10.0.0.1:443 - NatIP 192.168.0.1:12248 - Destination 192.168.0.100:443 - Delink Time 06/14/2024:16:01:40 GMT - Total_bytes_send 0 - Total_bytes_recv 619
<135> 06/14/2024:16:01:39 GMT host1 0-PPE-0 : default SSLLOG SSL_HANDSHAKE_SUCCESS 220659228 0 :  SPCBId 27719517 - ClientIP 1.1.1.1 - ClientPort 38797 - VserverServiceIP 192.168.0.1 - VserverServicePort 443 - ClientVersion TLSv1.2 - CipherSuite "TLS1.2-ECDHE-RSA-AES256-GCM-SHA384" - Session New  - HandshakeTime 120 ms
jamiehynds commented 2 weeks ago

@piyush-elastic could you please look at this one.

eriroley commented 2 weeks ago

I got an email notification that I'm not seeing in the thread here - but, to answer the question - yes, our ADC/Netscaler has been in production for quite a while (we keep it up-to-date), it is a virtual appliance that was migrated from a legacy MPX back in the day.

Are you thinking that there is a legacy setting on the ADC that needs to be adjusted?

jvalente-salemstate commented 2 weeks ago

I got an email notification that I'm not seeing in the thread here - but, to answer the question - yes, our ADC/Netscaler has been in production for quite a while (we keep it up-to-date), it is a virtual appliance that was migrated from a legacy MPX back in the day.

Are you thinking that there is a legacy setting on the ADC that needs to be adjusted?

that was me. I had a mistake in there but wasn't near a computer to correct it.

But for anyone who missed it, I'd asked the device was from before Citrix rebranded the Netscaler to ADC. The non-working format in the OP does match older parsers from before that, and our logs also have this format (device purchased prior to 2020, but up to date in terms of version).

I do think that it may be a scenario where the format isn't updated (maybe to not break existing logging ?) on older devices. I don't know how to verify or change that though as I'm not our Citrix admin, or I'd be able to test this before waiting for our new unit to be up.

jvalente-salemstate commented 2 weeks ago

So looking closer at our configuration, I think what we're seeing is the use of classic policies. These were deprecated before 13.0. The logs in /var/log/ appear as expected since the export policies don't apply to them. I am also going to guess using syslog.conf to specify an agent with the integration as the syslog server would work. If it is the classic policies, we should probably be moving off those 😀

However, I don't think that is the issue here. Looking at the pipeline, it does fail to match that first grok pattern but the entire event.original is sent to the native pipeline or CEF pipeline after being matched and stored in another field with the second pattern. There's a grok pattern in the Native pipeline that should be catching this format and I can't determine why it's not working. There's a space before the PRI in the pattern. Could that be it?

jvalente-salemstate commented 6 days ago

@fgierlinger @eriroley

Could either of you share a few more (redacted) sample logs or the error message?

I applied the following to our new, and old, devices 

add audit syslogAction elastic_agent_vip x.x.x.x -serverPort 10610 -logLevel EMERGENCY ALERT CRITICAL ERROR WARNING NOTICE INFORMATIONAL -tcp ALL -acl ENABLED -userDefinedAuditlog YES -appflowExport ENABLED -lsn ENABLED -alg ENABLED -subscriberLog ENABLED -dns ENABLED -ContentInspectionLog ENABLED -sslInterception ENABLED -urlFiltering ENABLED -timeZone LOCAL_TIME
add audit syslogPolicy elastic_agent_syslog TRUE elastic_agent_vip
bind audit syslogGlobal -policyName elastic_agent_syslog -priority 10

The format still differs but in my case I am missing the time zone (need to try not setting it in the config I guess) but after I edited the native pipeline to make that an optional match it works.

But the sample logs in the OP also worked without any edits via the grok debugger.

edit: looking at logs that have come in, it does look like additional parsers for things like Citrix Gateway / Storefront events are needed.