Could not index event to Elasticsearch - logs-pfelk-openvpn not created

[jakes670]

Describe the bug User login on pFsense Firewall with OpenVPN Authentication is with FreeRadius and 2fa

To Reproduce Steps to reproduce the behavior:

1. Login with OpenVPN to a pFsense server
2. Index logs-pfelk-openvpn is not created.
3. view output of logstash
4. One of 20 errors all the same
5. [WARN ] 2023-04-29 14:15:06.589 [[pfelk]>worker14] elasticsearch - Could not index event to Elasticsearch. status: 400, action: ["create", {:_id=>nil, :_index=>"logs-pfelk-openvpn", :routing=>nil}, {"event"=>{"original"=>"<29>1 2023-04-29T16:15:06.777622+02:00 pfsense-firewall openvpn 13109 - - 1.2.3.4:17023 Expected Remote Options String (VER=V4): 'V4,dev-type tun,link-mtu 1569,tun-mtu 1500,proto UDPv4,keydir 1,cipher AES-128-CBC,auth SHA256,keysize 128,tls-auth,key-method 2,tls-client'", "dataset"=>"pfelk.openvpn", "created"=>2023-04-29T14:15:06.777Z}, "openvpn"=>{"user"=>{"ip"=>"1.2.3.4", "port"=>"17023"}}, "@version"=>"1", "log"=>{"syslog"=>{"appname"=>"openvpn", "hostname"=>"pfsense-firewall", "version"=>"1", "severity"=>{"code"=>5, "name"=>"Notice"}, "priority"=>29, "facility"=>{"code"=>3, "name"=>"system"}, "procid"=>"13109"}}, "host"=>{"ip"=>"172.20.0.1"}, "service"=>{"type"=>"system"}, "tags"=>["pfelk", "openvpn"], "data_stream"=>{"namespace"=>"openvpn", "type"=>"logs", "dataset"=>"pfelk"}, "type"=>"firewall", "@timestamp"=>2023-04-29T14:15:06.419208765Z}], response: {"create"=>{"_index"=>".ds-logs-pfelk-openvpn-2023.04.29-000002", "_id"=>"7ABezYcBhJIWzwjzEOJF", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [openvpn.user] of type [keyword] in document with id '7ABezYcBhJIWzwjzEOJF'. Preview of field's value: '{port=17023, ip=1.2.3.4}'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:398"}}}}

Firewall System (please complete the following information):

• pfSense
• Version 2.6.0

Operating System (please complete the following information):

• OS

• Linux 5.15.0-60-generic x86_64 PRETTY_NAME="Ubuntu 22.04 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04 (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=jammy

  Installation method (docker & script):

Elasticsearch, Logstash, Kibana (please complete the following information):

• Version of ELK components 8.6.2

  Elasticsearch, Logstash, Kibana logs:

• Logstash logs [WARN ] 2023-04-29 14:15:06.589 [[pfelk]>worker14] elasticsearch - Could not index event to Elasticsearch. status: 400, action: ["create", {:_id=>nil, :_index=>"logs-pfelk-openvpn", :routing=>nil}, {"event"=>{"original"=>"<29>1 2023-04-29T16:15:06.777622+02:00 pfsense-firewall openvpn 13109 - - 1.2.3.4:17023 Expected Remote Options String (VER=V4): 'V4,dev-type tun,link-mtu 1569,tun-mtu 1500,proto UDPv4,keydir 1,cipher AES-128-CBC,auth SHA256,keysize 128,tls-auth,key-method 2,tls-client'", "dataset"=>"pfelk.openvpn", "created"=>2023-04-29T14:15:06.777Z}, "openvpn"=>{"user"=>{"ip"=>"1.2.3.4", "port"=>"17023"}}, "@version"=>"1", "log"=>{"syslog"=>{"appname"=>"openvpn", "hostname"=>"pfsense-firewall", "version"=>"1", "severity"=>{"code"=>5, "name"=>"Notice"}, "priority"=>29, "facility"=>{"code"=>3, "name"=>"system"}, "procid"=>"13109"}}, "host"=>{"ip"=>"172.20.0.1"}, "service"=>{"type"=>"system"}, "tags"=>["pfelk", "openvpn"], "data_stream"=>{"namespace"=>"openvpn", "type"=>"logs", "dataset"=>"pfelk"}, "type"=>"firewall", "@timestamp"=>2023-04-29T14:15:06.419208765Z}], response: {"create"=>{"_index"=>".ds-logs-pfelk-openvpn-2023.04.29-000002", "_id"=>"7ABezYcBhJIWzwjzEOJF", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [openvpn.user] of type [keyword] in document with id '7ABezYcBhJIWzwjzEOJF'. Preview of field's value: '{port=17023, ip=1.2.3.4}'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:398"}}}}

Additional context Add any other context about the problem here.

**Attach the pfELK Error Log (error.pfelk), for Better Assistance***

• Do not copy/paste log; attach as a file

[a3ilson]

@jakes670 thanks for providing sufficient details to evaluate this matter. Unfortunately, I’m traveling for work and won’t be able to correct this until after 5 May.

The error appears to be specific to the openvpn.user value/field and the data type. I’ll take the log that you provided for further testing and reply next week with a possible solution if that works and your sale to validate. Alternatively, if you or someone else devises a solution submit a pull request.

the “openvpn.user” needs to be defined within the template or 05 file. However, that field also contains nested items (eg “openvpn.user.name”)

[a3ilson]

Closing as this will be addressed with issue #376 which has been a very long work-in progress