search

wazuh / wazuh

Wazuh - The Open Source Security Platform. Unified XDR and SIEM protection for endpoints and cloud workloads.
https://wazuh.com/
Other
10.75k stars 1.65k forks source link

Wazuh manager not able to fetch/parse zeek logs from wazuh agent #9662

Closed dipenpatel235 closed 3 years ago

dipenpatel235 commented 3 years ago

I just created a wazuh manager and wazuh agent. Now I want to send a zeek log to wazuh manager through wazuh agent. Wazuh agent collecting the log files but wazuh manager not showing any response or logs.

Wazuh agent running on ubuntu 18.04 and wazuh manager is running on centos 7 in docker. Wazuh manager : 4.1.5, Wazuh agent : 4.1.5 Here is my local.zeek file

2021-08-11_17-01

all the logs converted in JSON and files extension is (.log)

owlh.zeek

2021-08-11_17-02

Wazuh agent ossec.conf file I also configured wazuh manager IP address, port, and all the required configuration

2021-08-11_17-03

juliancnn commented 3 years ago

Hi @dipenpatel235,

Could you share with us some entries from the files you collect with logcollector? By analyzing the /user/local/zeek/logs/current/*.log files we can see if there is a problem with the logs getting.

Regards, Julian

dipenpatel235 commented 3 years ago

2021-08-12_20-13

==> capture_loss.log <== {"ts":1628776930.738566,"ts_delta":900.0008020401001,"peer":"zeek","gaps":0,"acks":0,"percent_lost":0.0} {"ts":1628777830.739055,"ts_delta":900.0004889965057,"peer":"zeek","gaps":0,"acks":0,"percent_lost":0.0} {"ts":1628778730.74023,"ts_delta":900.0011751651764,"peer":"zeek","gaps":0,"acks":0,"percent_lost":0.0}

==> conn.log <== {"ts":1628778667.893483,"uid":"CuYzZ63phQJYrsgAXa","id.orig_h":"221.131.165.23","id.orig_p":23573,"id.resp_h":"172.31.21.152","id.resp_p":22,"proto":"tcp","conn_state":"OTH","local_orig":false,"local_resp":true,"missed_bytes":0,"history":"^c","orig_pkts":0,"orig_ip_bytes":0,"resp_pkts":0,"resp_ip_bytes":0,"bro_engine":"CONN"} {"ts":1628778678.165472,"uid":"CsazKF2UIhalnOC7Xj","id.orig_h":"221.131.165.23","id.orig_p":23573,"id.resp_h":"172.31.21.152","id.resp_p":22,"proto":"tcp","conn_state":"OTH","local_orig":false,"local_resp":true,"missed_bytes":0,"history":"^c","orig_pkts":0,"orig_ip_bytes":0,"resp_pkts":0,"resp_ip_bytes":0,"bro_engine":"CONN"} {"ts":1628778691.733475,"uid":"Coz08k2V18DFXoXKaj","id.orig_h":"221.131.165.23","id.orig_p":23573,"id.resp_h":"172.31.21.152","id.resp_p":22,"proto":"tcp","conn_state":"OTH","local_orig":false,"local_resp":true,"missed_bytes":0,"history":"^c","orig_pkts":0,"orig_ip_bytes":0,"resp_pkts":0,"resp_ip_bytes":0,"bro_engine":"CONN"} {"ts":1628778719.637492,"uid":"CKlBtpdsmZJteTbBg","id.orig_h":"221.131.165.23","id.orig_p":23573,"id.resp_h":"172.31.21.152","id.resp_p":22,"proto":"tcp","conn_state":"OTH","local_orig":false,"local_resp":true,"missed_bytes":0,"history":"^c","orig_pkts":0,"orig_ip_bytes":0,"resp_pkts":0,"resp_ip_bytes":0,"bro_engine":"CONN"}

==> dhcp.log <== {"ts":1628777861.988888,"uids":["CSJDBR2idU96bMOroa"],"client_addr":"172.31.21.152","server_addr":"172.31.16.1","mac":"06:a5:91:bd:b1:88","domain":"us-east-2.compute.internal","assigned_addr":"172.31.21.152","lease_time":3600.0,"msg_types":["ACK"],"duration":0.0}

==> dns.log <== {"ts":1628776909.719421,"uid":"C0RT022veBeHZLIF9j","id.orig_h":"172.31.21.152","id.orig_p":59157,"id.resp_h":"172.31.0.2","id.resp_p":53,"proto":"udp","trans_id":31402,"query":"2.0.17.172.in-addr.arpa","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":false,"RA":true,"Z":0,"answers":["ip-172-17-0-2.us-east-2.compute.internal"],"TTLs":[300.0],"rejected":false,"bro_engine":"DNS"} {"ts":1628778026.248051,"uid":"CW3tWP3j162D1fzwNc","id.orig_h":"172.31.21.152","id.orig_p":36192,"id.resp_h":"172.31.0.2","id.resp_p":53,"proto":"udp","trans_id":33798,"rcode":3,"rcode_name":"NXDOMAIN","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false,"bro_engine":"DNS"} {"ts":1628778953.665514,"uid":"CUqJrBwNUMfdzwVu9","id.orig_h":"172.31.21.152","id.orig_p":52520,"id.resp_h":"172.31.0.2","id.resp_p":53,"proto":"udp","trans_id":34369,"rcode":3,"rcode_name":"NXDOMAIN","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false,"bro_engine":"DNS"} {"ts":1628778953.665972,"uid":"CUqJrBwNUMfdzwVu9","id.orig_h":"172.31.21.152","id.orig_p":52520,"id.resp_h":"172.31.0.2","id.resp_p":53,"proto":"udp","trans_id":34369,"rcode":3,"rcode_name":"NXDOMAIN","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false,"bro_engine":"DNS"}

==> files.log <== {"ts":1628777110.882924,"fuid":"FUvLXl3dUfiEHVIeHf","tx_hosts":["209.141.54.8"],"rx_hosts":["172.31.21.152"],"conn_uids":["CuL3Wv4cUhT8w3PUj8"],"source":"HTTP","depth":0,"analyzers":["SHA1","MD5"],"mime_type":"text/plain","duration":0.0,"local_orig":false,"is_orig":true,"seen_bytes":29,"total_bytes":29,"missing_bytes":0,"overflow_bytes":0,"timedout":false,"md5":"eb7255406bc93cf7814114fabb05f46b","sha1":"e263d5e8d9a0337657800a3a269630202248268e"}

==> http.log <== {"ts":1628777110.882924,"uid":"CuL3Wv4cUhT8w3PUj8","id.orig_h":"209.141.54.8","id.orig_p":34794,"id.resp_h":"172.31.21.152","id.resp_p":80,"trans_depth":1,"method":"POST","host":"18.221.202.188","uri":"/boaform/admin/formLogin","referrer":"http://18.221.202.188:80/admin/login.asp","user_agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:71.0) Gecko/20100101 Firefox/71.0","origin":"http://18.221.202.188:80","request_body_len":29,"response_body_len":0,"tags":[],"orig_fuids":["FUvLXl3dUfiEHVIeHf"],"orig_mime_types":["text/plain"]} {"ts":1628778370.048578,"uid":"CYoitv4teckFCQ2x52","id.orig_h":"58.121.115.81","id.orig_p":60279,"id.resp_h":"172.31.21.152","id.resp_p":80,"trans_depth":1,"method":"GET","uri":"/","request_body_len":0,"response_body_len":0,"tags":[]}

==> notice.log <== {"ts":1628776930.738566,"note":"CaptureLoss::Too_Little_Traffic","msg":"Only observed 0 TCP ACKs and was expecting at least 1.","actions":["Notice::ACTION_LOG"],"suppress_for":3600.0} {"ts":1628777830.739055,"note":"CaptureLoss::Too_Little_Traffic","msg":"Only observed 0 TCP ACKs and was expecting at least 1.","actions":["Notice::ACTION_LOG"],"suppress_for":3600.0} {"ts":1628778730.74023,"note":"CaptureLoss::Too_Little_Traffic","msg":"Only observed 0 TCP ACKs and was expecting at least 1.","actions":["Notice::ACTION_LOG"],"suppress_for":3600.0}

==> ntp.log <== {"ts":1628777862.078427,"uid":"C807FY3xluHnyP9eS6","id.orig_h":"172.31.21.152","id.orig_p":60337,"id.resp_h":"91.189.94.4","id.resp_p":123,"version":4,"mode":4,"stratum":2,"poll":8.0,"precision":1.1920928955078126e-7,"root_delay":0.00836181640625,"root_disp":0.0334625244140625,"ref_id":"17.253.108.253","ref_time":1628777197.8882094,"org_time":1628777861.2312878,"rec_time":1628777862.035259,"xmt_time":1628777862.0352975,"num_exts":0}

==> ssh.log <== {"ts":1628777425.722496,"uid":"CmBDM5L9FoatNrYj4","id.orig_h":"78.196.138.44","id.orig_p":59212,"id.resp_h":"172.31.21.152","id.resp_p":22,"auth_attempts":0,"direction":"INBOUND","client":"SSH-2.0-libssh_0.9.5","bro_engine":"SSH"} {"ts":1628777503.472943,"uid":"CjOc7z1uEhopfBvVze","id.orig_h":"106.75.57.20","id.orig_p":60084,"id.resp_h":"172.31.21.152","id.resp_p":22,"auth_attempts":0,"direction":"INBOUND","client":"SSH-2.0-libssh-0.6.3","bro_engine":"SSH"} {"ts":1628777508.415323,"uid":"C2Khf72ioNgtg3ZFcl","id.orig_h":"159.65.152.148","id.orig_p":39460,"id.resp_h":"172.31.21.152","id.resp_p":22,"auth_attempts":0,"direction":"INBOUND","client":"SSH-2.0-libssh_0.9.5","bro_engine":"SSH"} {"ts":1628777519.432458,"uid":"CsMJJg2gbTFRH2sCh7","id.orig_h":"78.130.225.212","id.orig_p":39770,"id.resp_h":"172.31.21.152","id.resp_p":22,"auth_attempts":0,"direction":"INBOUND","client":"SSH-2.0-libssh-0.6.3","bro_engine":"SSH"} {"ts":1628777595.651692,"uid":"CdHJuB4vAFf3hnZnu4","id.orig_h":"221.181.185.220","id.orig_p":18803,"id.resp_h":"172.31.21.152","id.resp_p":22,"auth_attempts":0,"direction":"INBOUND","client":"SSH-2.0-PUTTY","bro_engine":"SSH"} {"ts":1628777672.717758,"uid":"CsitZl3GFSeJgKDZy3","id.orig_h":"181.52.209.199","id.orig_p":36774,"id.resp_h":"172.31.21.152","id.resp_p":22,"auth_attempts":0,"direction":"INBOUND","client":"SSH-2.0-libssh-0.6.3","bro_engine":"SSH"} {"ts":1628777804.165845,"uid":"CQhrkI1NYXIpszBIPc","id.orig_h":"167.99.107.57","id.orig_p":36096,"id.resp_h":"172.31.21.152","id.resp_p":22,"auth_attempts":0,"direction":"INBOUND","client":"SSH-2.0-Go","bro_engine":"SSH"} {"ts":1628778014.31724,"uid":"CqUTZa4LsUQbo8Q3z8","id.orig_h":"219.91.190.148","id.orig_p":5349,"id.resp_h":"172.31.21.152","id.resp_p":22,"auth_attempts":0,"direction":"INBOUND","client":"SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.2","bro_engine":"SSH"} {"ts":1628778664.553603,"uid":"Ck1vnWsRrsd1gFSj5","id.orig_h":"221.131.165.23","id.orig_p":23573,"id.resp_h":"172.31.21.152","id.resp_p":22,"auth_attempts":0,"direction":"INBOUND","client":"SSH-2.0-PUTTY","bro_engine":"SSH"} {"ts":1628778937.184541,"uid":"CVSxUj1g1bbrPRsW3h","id.orig_h":"219.91.190.148","id.orig_p":6065,"id.resp_h":"172.31.21.152","id.resp_p":22,"auth_attempts":0,"direction":"INBOUND","client":"SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.2","bro_engine":"SSH"}

==> ssl.log <== {"ts":1628777194.067219,"uid":"CHlXmP3mk1ednhIZNc","id.orig_h":"192.241.211.186","id.orig_p":50830,"id.resp_h":"172.31.21.152","id.resp_p":443,"resumed":false,"established":false,"bro_engine":"SSL"} {"ts":1628777812.194845,"uid":"CZkGA9jKQmlo85ezj","id.orig_h":"192.241.215.32","id.orig_p":43218,"id.resp_h":"172.31.21.152","id.resp_p":443,"resumed":false,"established":false,"bro_engine":"SSL"} {"ts":1628778982.786137,"uid":"C499s24eUSmDxnwfrb","id.orig_h":"198.74.56.99","id.orig_p":61000,"id.resp_h":"172.31.21.152","id.resp_p":443,"resumed":false,"established":false,"bro_engine":"SSL"}

==> stats.log <== {"ts":1628776870.755296,"peer":"zeek","mem":215,"pkts_proc":845,"bytes_recv":128928,"pkts_dropped":0,"pkts_link":845,"pkt_lag":0.000492095947265625,"events_proc":662,"events_queued":660,"active_tcp_conns":3,"active_udp_conns":0,"active_icmp_conns":1,"tcp_conns":27,"udp_conns":18,"icmp_conns":1,"timers":1838,"active_timers":64,"files":0,"active_files":0,"dns_requests":0,"active_dns_requests":0,"reassem_tcp_size":53214,"reassem_file_size":0,"reassem_frag_size":0,"reassem_unknown_size":0} {"ts":1628777170.755621,"peer":"zeek","mem":215,"pkts_proc":4352,"bytes_recv":920077,"pkts_dropped":0,"pkts_link":4352,"pkt_lag":0.0014491081237792969,"events_proc":487,"events_queued":490,"active_tcp_conns":5,"active_udp_conns":0,"active_icmp_conns":0,"tcp_conns":12,"udp_conns":3,"icmp_conns":0,"timers":1801,"active_timers":66,"files":1,"active_files":0,"dns_requests":0,"active_dns_requests":0,"reassem_tcp_size":17617,"reassem_file_size":0,"reassem_frag_size":0,"reassem_unknown_size":0} {"ts":1628777470.756588,"peer":"zeek","mem":215,"pkts_proc":383,"bytes_recv":73969,"pkts_dropped":0,"pkts_link":383,"pkt_lag":0.0004990100860595703,"events_proc":492,"events_queued":489,"active_tcp_conns":3,"active_udp_conns":0,"active_icmp_conns":0,"tcp_conns":9,"udp_conns":0,"icmp_conns":0,"timers":1795,"active_timers":65,"files":0,"active_files":0,"dns_requests":0,"active_dns_requests":0,"reassem_tcp_size":20138,"reassem_file_size":0,"reassem_frag_size":0,"reassem_unknown_size":0} {"ts":1628777770.757216,"peer":"zeek","mem":215,"pkts_proc":373,"bytes_recv":67249,"pkts_dropped":0,"pkts_link":373,"pkt_lag":0.00046706199645996094,"events_proc":483,"events_queued":485,"active_tcp_conns":2,"active_udp_conns":0,"active_icmp_conns":0,"tcp_conns":7,"udp_conns":0,"icmp_conns":0,"timers":1671,"active_timers":59,"files":0,"active_files":0,"dns_requests":0,"active_dns_requests":0,"reassem_tcp_size":23573,"reassem_file_size":0,"reassem_frag_size":0,"reassem_unknown_size":0} {"ts":1628778070.757496,"peer":"zeek","mem":215,"pkts_proc":3397,"bytes_recv":739699,"pkts_dropped":0,"pkts_link":3397,"pkt_lag":0.00045299530029296875,"events_proc":534,"events_queued":534,"active_tcp_conns":3,"active_udp_conns":0,"active_icmp_conns":0,"tcp_conns":8,"udp_conns":7,"icmp_conns":0,"timers":1769,"active_timers":63,"files":0,"active_files":0,"dns_requests":0,"active_dns_requests":0,"reassem_tcp_size":12090,"reassem_file_size":0,"reassem_frag_size":0,"reassem_unknown_size":0} {"ts":1628778370.758151,"peer":"zeek","mem":215,"pkts_proc":967,"bytes_recv":183562,"pkts_dropped":0,"pkts_link":967,"pkt_lag":0.0005118846893310547,"events_proc":344,"events_queued":342,"active_tcp_conns":4,"active_udp_conns":0,"active_icmp_conns":0,"tcp_conns":4,"udp_conns":0,"icmp_conns":0,"timers":1673,"active_timers":65,"files":0,"active_files":0,"dns_requests":0,"active_dns_requests":0,"reassem_tcp_size":15545,"reassem_file_size":0,"reassem_frag_size":0,"reassem_unknown_size":0} {"ts":1628778670.758412,"peer":"zeek","mem":215,"pkts_proc":200,"bytes_recv":40239,"pkts_dropped":0,"pkts_link":200,"pkt_lag":0.0003941059112548828,"events_proc":383,"events_queued":387,"active_tcp_conns":3,"active_udp_conns":0,"active_icmp_conns":0,"tcp_conns":8,"udp_conns":0,"icmp_conns":0,"timers":1648,"active_timers":58,"files":0,"active_files":0,"dns_requests":0,"active_dns_requests":0,"reassem_tcp_size":13130,"reassem_file_size":0,"reassem_frag_size":0,"reassem_unknown_size":0} {"ts":1628778970.763602,"peer":"zeek","mem":215,"pkts_proc":332,"bytes_recv":60709,"pkts_dropped":0,"pkts_link":332,"pkt_lag":0.00045800209045410156,"events_proc":402,"events_queued":401,"active_tcp_conns":3,"active_udp_conns":0,"active_icmp_conns":0,"tcp_conns":7,"udp_conns":2,"icmp_conns":0,"timers":1657,"active_timers":60,"files":0,"active_files":0,"dns_requests":0,"active_dns_requests":0,"reassem_tcp_size":18326,"reassem_file_size":0,"reassem_frag_size":0,"reassem_unknown_size":0}

==> stderr.log <== listening on eth0

send-mail: /usr/sbin/sendmail not found send-mail: /usr/sbin/sendmail not found send-mail: /usr/sbin/sendmail not found send-mail: /usr/sbin/sendmail not found

==> stdout.log <== max memory size (kbytes, -m) unlimited data seg size (kbytes, -d) unlimited virtual memory (kbytes, -v) unlimited core file size (blocks, -c) unlimited

==> weird.log <== {"ts":1628778320.235141,"uid":"CNqx5R9duhtAnedM9","id.orig_h":"209.141.49.70","id.orig_p":52972,"id.resp_h":"172.31.21.152","id.resp_p":22,"name":"inappropriate_FIN","notice":false,"peer":"zeek","source":"TCP","bro_engine":"WEIRD"} {"ts":1628778369.358075,"uid":"ChVzv405FHX36l8i9","id.orig_h":"58.121.115.81","id.orig_p":32776,"id.resp_h":"172.31.21.152","id.resp_p":80,"name":"bad_TCP_checksum","notice":false,"peer":"zeek","source":"TCP","bro_engine":"WEIRD"} {"ts":1628778369.852492,"uid":"CYoitv4teckFCQ2x52","id.orig_h":"58.121.115.81","id.orig_p":60279,"id.resp_h":"172.31.21.152","id.resp_p":80,"name":"data_before_established","notice":false,"peer":"zeek","source":"TCP","bro_engine":"WEIRD"} {"ts":1628778561.11068,"uid":"CnuT2k1bc3gDVyceIe","id.orig_h":"172.31.21.152","id.orig_p":57088,"id.resp_h":"172.31.39.241","id.resp_p":1514,"name":"active_connection_reuse","notice":false,"peer":"zeek","source":"TCP","bro_engine":"WEIRD"} {"ts":1628778561.133641,"uid":"CTeSqb4Xw03MQHdDzf","id.orig_h":"172.31.39.241","id.orig_p":1514,"id.resp_h":"172.31.21.152","id.resp_p":57088,"name":"data_before_established","notice":false,"peer":"zeek","source":"TCP","bro_engine":"WEIRD"} {"ts":1628778664.553603,"uid":"Ck1vnWsRrsd1gFSj5","id.orig_h":"221.131.165.23","id.orig_p":23573,"id.resp_h":"172.31.21.152","id.resp_p":22,"name":"data_before_established","notice":false,"peer":"zeek","source":"TCP","bro_engine":"WEIRD"} {"ts":1628778937.184541,"uid":"CVSxUj1g1bbrPRsW3h","id.orig_h":"219.91.190.148","id.orig_p":6065,"id.resp_h":"172.31.21.152","id.resp_p":22,"name":"data_before_established","notice":false,"peer":"zeek","source":"TCP","bro_engine":"WEIRD"} {"ts":1628778980.773122,"uid":"C499s24eUSmDxnwfrb","id.orig_h":"198.74.56.99","id.orig_p":61000,"id.resp_h":"172.31.21.152","id.resp_p":443,"name":"bad_TCP_checksum","notice":false,"peer":"zeek","source":"TCP","bro_engine":"WEIRD"} {"ts":1628778982.786137,"uid":"C499s24eUSmDxnwfrb","id.orig_h":"198.74.56.99","id.orig_p":61000,"id.resp_h":"172.31.21.152","id.resp_p":443,"name":"data_before_established","notice":false,"peer":"zeek","source":"TCP","bro_engine":"WEIRD"} {"ts":1628778982.890886,"uid":"C499s24eUSmDxnwfrb","id.orig_h":"198.74.56.99","id.orig_p":61000,"id.resp_h":"172.31.21.152","id.resp_p":443,"name":"inappropriate_FIN","notice":false,"peer":"zeek","source":"TCP","bro_engine":"WEIRD"}

juliancnn commented 3 years ago

Hi @dipenpatel235

The log collection configuration is correct. Could you check the connection between the agent and the manager? For this, from the manager side, you could execute the following command:

/var/ossec/bin/agent_control -l

You will get an output like the following that will indicate which agents are connected at that moment:

╰─# /var/ossec/bin/agent_control -l     

Wazuh agent_control. List of available agents:
   ID: 000, Name: 35-u20-manager4 (server), IP: 127.0.0.1, Active/Local
   ID: 010, Name: 37-c8-agent4, IP: any, Disconnected
   ID: 014, Name: win2012_4, IP: any, Disconnected
   ID: 008, Name: 32-c8-agent4, IP: any, Active
   ID: 022, Name: 44-c6-agent4, IP: any, Active

You have to keep in mind that not all logs will generate alerts, i.e. you will not see all logs in Kibana but only those that generate alerts. You could activate <logall_json> (changing no with yes) in the ossec.conf in the manager (must restart the service to apply changes) to see in archives every log reaching the manager and also what it is doing with each log.

Once you have activated logall_json you can see new logs arriving (in real time) at the manager with tail command: tail -f /var/ossec/logs/archives/archives.json

If you force the generation of logs, this will help you to know if they are arriving. Note that the logall_json configuration should be temporary and enabled for testing purposes only, as storing all logs received from all agents can be a waste of memory.

Please let me know if you found this helpful. Regards, Julian

dipenpatel235 commented 3 years ago

Hi @juliancnn

Yes, I tried things that you mentioned but it still does not display any data in kibana.

Look at this screenshot kibana

see also archive.json log

archive-json

And hear is agent list of manager,

agent-control

If you are available for some time then let me know. So, I will show you my all the configuration file via screen sharing. So, you can get an exact idea about all.

juliancnn commented 3 years ago

Hi dipenpatel235,

The screenshot you share with us of the archive.json, gives us good information:

image

Green line divides the processing of 2 logs:

Up to this point there are no problems. Are these alerts displayed in kibana? For this, you can filter by rule id and search up to one month back: image

Let me know if the alerts are displayed in kibana. Best regards, Julian

dipenpatel235 commented 3 years ago

Hello @juliancnn

Yes, I tried this thing you mentioned and I also try for the last year, but It still not display any records.

Please look at the screenshot for the last year. index_range

juliancnn commented 3 years ago

@dipenpatel235

I see.... Usually, when an alert is in alert.json but it is not possible to see it from kibana it is a problem on elastic stack.

Could you confirm me if you can see other alerts in Kibana, i.e. when restarting the manager, can you see the alert that the manager has started? image

If this is the case, then the problem is in filebeat, which reads the alerts from alerts.json and sends them to elasticsearch. Could you share with me the zeek alerts that you can't see in kibana, in plain text. This will help me to analyze them and see if there is any problem with the alert.

Regards, Julian

dipenpatel235 commented 3 years ago

Hello @juliancnn

Yes, I see the wazuh-manager is started and displays logs in kibana. 25-8-kibana

And hear is alert.json file in text formate.

{"timestamp":"2021-08-25T04:49:04.283+0000","rule":{"level":5,"description":"Zeek: SSH Connection","id":"66001","firedtimes":89,"mail":false,"groups":["zeek","ids"]},"agent":{"id":"012","name":"liveness.server","ip":"172.31.21.152"},"manager":{"name":"wazuh-manager"},"id":"1629866944.15751292","full_log":"{\"ts\":1629866937.6732931,\"uid\":\"CbZm0X3Ntova265R3g\",\"id.orig_h\":\"176.111.173.85\",\"id.orig_p\":17962,\"id.resp_h\":\"172.31.21.152\",\"id.resp_p\":22,\"auth_attempts\":0,\"direction\":\"INBOUND\",\"client\":\"SSH-2.0-libssh_0.9.5\",\"bro_engine\":\"SSH\"}","decoder":{"name":"json"},"data":{"ts":"1629866937.673293","uid":"CbZm0X3Ntova265R3g","id":{"orig_h":"176.111.173.85","orig_p":"17962","resp_h":"172.31.21.152","resp_p":"22"},"auth_attempts":"0","direction":"INBOUND","client":"SSH-2.0-libssh_0.9.5","bro_engine":"SSH"},"location":"/usr/local/zeek/logs/current/ssh.log"}
{"timestamp":"2021-08-25T04:49:08.288+0000","rule":{"level":5,"description":"Zeek: Connection detail","id":"66004","firedtimes":545,"mail":false,"groups":["zeek","ids"]},"agent":{"id":"012","name":"liveness.server","ip":"172.31.21.152"},"manager":{"name":"wazuh-manager"},"id":"1629866948.15751907","full_log":"{\"ts\":1629866941.511791,\"uid\":\"CYv7Rk33iFIqKurOfc\",\"id.orig_h\":\"107.189.31.98\",\"id.orig_p\":59480,\"id.resp_h\":\"172.31.21.152\",\"id.resp_p\":22,\"proto\":\"tcp\",\"duration\":0.31515789031982422,\"orig_bytes\":0,\"resp_bytes\":0,\"conn_state\":\"SH\",\"local_orig\":false,\"local_resp\":true,\"missed_bytes\":0,\"history\":\"ScAF\",\"orig_pkts\":5,\"orig_ip_bytes\":268,\"resp_pkts\":0,\"resp_ip_bytes\":0,\"bro_engine\":\"CONN\"}","decoder":{"name":"json"},"data":{"ts":"1629866941.511791","uid":"CYv7Rk33iFIqKurOfc","id":{"orig_h":"107.189.31.98","orig_p":"59480","resp_h":"172.31.21.152","resp_p":"22"},"proto":"tcp","duration":"0.315158","orig_bytes":"0","resp_bytes":"0","conn_state":"SH","local_orig":"false","local_resp":"true","missed_bytes":"0","history":"ScAF","orig_pkts":"5","orig_ip_bytes":"268","resp_pkts":"0","resp_ip_bytes":"0","bro_engine":"CONN"},"location":"/usr/local/zeek/logs/current/conn.log"}
{"timestamp":"2021-08-25T04:49:14.093+0000","rule":{"level":5,"description":"Zeek: Connection detail","id":"66004","firedtimes":546,"mail":false,"groups":["zeek","ids"]},"agent":{"id":"003","name":"esign.kycaml.systems","ip":"172.31.9.38"},"manager":{"name":"wazuh-manager"},"id":"1629866954.15752818","full_log":"{\"ts\":1629866952.767386,\"uid\":\"CpTBQ1aO5afQRWU26\",\"id.orig_h\":\"172.31.9.38\",\"id.orig_p\":35688,\"id.resp_h\":\"52.95.16.182\",\"id.resp_p\":443,\"proto\":\"tcp\",\"conn_state\":\"OTH\",\"local_orig\":true,\"local_resp\":false,\"missed_bytes\":0,\"history\":\"C\",\"orig_pkts\":0,\"orig_ip_bytes\":0,\"resp_pkts\":0,\"resp_ip_bytes\":0,\"bro_engine\":\"CONN\"}","decoder":{"name":"json"},"data":{"ts":"1629866952.767386","uid":"CpTBQ1aO5afQRWU26","id":{"orig_h":"172.31.9.38","orig_p":"35688","resp_h":"52.95.16.182","resp_p":"443"},"proto":"tcp","conn_state":"OTH","local_orig":"true","local_resp":"false","missed_bytes":"0","history":"C","orig_pkts":"0","orig_ip_bytes":"0","resp_pkts":"0","resp_ip_bytes":"0","bro_engine":"CONN"},"location":"/usr/local/zeek/logs/current/conn.log"}

logs collected but it's not showing in the kibana dashboard.

juliancnn commented 3 years ago

Hi @dipenpatel235 ,

First of all, I edited your previous comment to be able to see the logs as code blocks, this allows all the characters to be displayed, I hope this doesn't bother you. I performed the test regarding to inserting these logs into the alerts.json file and then analyzing the Filebeat behavior while in debug mode. These logs are not present in Kibana because Elasticsearch cannot index them, due to a problem with the Filbeat's template. The ID field is expected to be a string and not a json object. On Filebeat, you get the following logs:

2021-08-27T13:36:23.221Z    WARN    [elasticsearch] elasticsearch/client.go:408 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xc04258758bc036ad, ext:165343414356, loc:(*time.Location)(0x42417a0)}, Meta:{"pipeline":"filebeat-7.10.2-wazuh-alerts-pipeline"}, Fields:{"agent":{"ephemeral_id":"4dcb71b0-1628-4491-aefd-10910ed5a1c6","hostname":"manager","id":"21a55aae-0516-4eb7-aee7-86f0e8761c26","name":"manager","type":"filebeat","version":"7.10.2"},"ecs":{"version":"1.6.0"},"event":{"dataset":"wazuh.alerts","module":"wazuh"},"fields":{"index_prefix":"wazuh-alerts-4.x-"},"fileset":{"name":"alerts"},"host":{"name":"manager"},"input":{"type":"log"},"log":{"file":{"path":"/var/ossec/logs/alerts/alerts.json"},"offset":7347},"message":"{\"timestamp\":\"2021-08-25T04:49:04.283+0000\",\"rule\":{\"level\":5,\"description\":\"Zeek: SSH Connection\",\"id\":\"66001\",\"firedtimes\":89,\"mail\":false,\"groups\":[\"zeek\",\"ids\"]},\"agent\":{\"id\":\"012\",\"name\":\"liveness.server\",\"ip\":\"172.31.21.152\"},\"manager\":{\"name\":\"wazuh-manager\"},\"id\":\"1629866944.15751292\",\"full_log\":\"{\\\"ts\\\":1629866937.6732931,\\\"uid\\\":\\\"CbZm0X3Ntova265R3g\\\",\\\"id.orig_h\\\":\\\"176.111.173.85\\\",\\\"id.orig_p\\\":17962,\\\"id.resp_h\\\":\\\"172.31.21.152\\\",\\\"id.resp_p\\\":22,\\\"auth_attempts\\\":0,\\\"direction\\\":\\\"INBOUND\\\",\\\"client\\\":\\\"SSH-2.0-libssh_0.9.5\\\",\\\"bro_engine\\\":\\\"SSH\\\"}\",\"decoder\":{\"name\":\"json\"},\"data\":{\"ts\":\"1629866937.673293\",\"uid\":\"CbZm0X3Ntova265R3g\",\"id\":{\"orig_h\":\"176.111.173.85\",\"orig_p\":\"17962\",\"resp_h\":\"172.31.21.152\",\"resp_p\":\"22\"},\"auth_attempts\":\"0\",\"direction\":\"INBOUND\",\"client\":\"SSH-2.0-libssh_0.9.5\",\"bro_engine\":\"SSH\"},\"location\":\"/usr/local/zeek/logs/current/ssh.log\"}","service":{"type":"wazuh"}}, Private:file.State{Id:"native::67161127-2049", PrevId:"", Finished:false, Fileinfo:(*os.fileStat)(0xc000462a90), Source:"/var/ossec/logs/alerts/alerts.json", Offset:8241, Timestamp:time.Time{wall:0xc042584ec3e3c18f, ext:10211525922, loc:(*time.Location)(0x42417a0)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x400cc27, Device:0x801}, IdentifierName:"native"}, TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [data.id] of type [keyword] in document with id 'NM_Th3sBkjI4v_RsZDGv'. Preview of field's value: '{orig_p=17962, resp_h=172.31.21.152, orig_h=176.111.173.85, resp_p=22}'","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:215"}}
2021-08-27T13:36:23.221Z    WARN    [elasticsearch] elasticsearch/client.go:408 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xc04258758bc4b3a5, ext:165343708492, loc:(*time.Location)(0x42417a0)}, Meta:{"pipeline":"filebeat-7.10.2-wazuh-alerts-pipeline"}, Fields:{"agent":{"ephemeral_id":"4dcb71b0-1628-4491-aefd-10910ed5a1c6","hostname":"manager","id":"21a55aae-0516-4eb7-aee7-86f0e8761c26","name":"manager","type":"filebeat","version":"7.10.2"},"ecs":{"version":"1.6.0"},"event":{"dataset":"wazuh.alerts","module":"wazuh"},"fields":{"index_prefix":"wazuh-alerts-4.x-"},"fileset":{"name":"alerts"},"host":{"name":"manager"},"input":{"type":"log"},"log":{"file":{"path":"/var/ossec/logs/alerts/alerts.json"},"offset":8241},"message":"{\"timestamp\":\"2021-08-25T04:49:08.288+0000\",\"rule\":{\"level\":5,\"description\":\"Zeek: Connection detail\",\"id\":\"66004\",\"firedtimes\":545,\"mail\":false,\"groups\":[\"zeek\",\"ids\"]},\"agent\":{\"id\":\"012\",\"name\":\"liveness.server\",\"ip\":\"172.31.21.152\"},\"manager\":{\"name\":\"wazuh-manager\"},\"id\":\"1629866948.15751907\",\"full_log\":\"{\\\"ts\\\":1629866941.511791,\\\"uid\\\":\\\"CYv7Rk33iFIqKurOfc\\\",\\\"id.orig_h\\\":\\\"107.189.31.98\\\",\\\"id.orig_p\\\":59480,\\\"id.resp_h\\\":\\\"172.31.21.152\\\",\\\"id.resp_p\\\":22,\\\"proto\\\":\\\"tcp\\\",\\\"duration\\\":0.31515789031982422,\\\"orig_bytes\\\":0,\\\"resp_bytes\\\":0,\\\"conn_state\\\":\\\"SH\\\",\\\"local_orig\\\":false,\\\"local_resp\\\":true,\\\"missed_bytes\\\":0,\\\"history\\\":\\\"ScAF\\\",\\\"orig_pkts\\\":5,\\\"orig_ip_bytes\\\":268,\\\"resp_pkts\\\":0,\\\"resp_ip_bytes\\\":0,\\\"bro_engine\\\":\\\"CONN\\\"}\",\"decoder\":{\"name\":\"json\"},\"data\":{\"ts\":\"1629866941.511791\",\"uid\":\"CYv7Rk33iFIqKurOfc\",\"id\":{\"orig_h\":\"107.189.31.98\",\"orig_p\":\"59480\",\"resp_h\":\"172.31.21.152\",\"resp_p\":\"22\"},\"proto\":\"tcp\",\"duration\":\"0.315158\",\"orig_bytes\":\"0\",\"resp_bytes\":\"0\",\"conn_state\":\"SH\",\"local_orig\":\"false\",\"local_resp\":\"true\",\"missed_bytes\":\"0\",\"history\":\"ScAF\",\"orig_pkts\":\"5\",\"orig_ip_bytes\":\"268\",\"resp_pkts\":\"0\",\"resp_ip_bytes\":\"0\",\"bro_engine\":\"CONN\"},\"location\":\"/usr/local/zeek/logs/current/conn.log\"}","service":{"type":"wazuh"}}, Private:file.State{Id:"native::67161127-2049", PrevId:"", Finished:false, Fileinfo:(*os.fileStat)(0xc000462a90), Source:"/var/ossec/logs/alerts/alerts.json", Offset:9484, Timestamp:time.Time{wall:0xc042584ec3e3c18f, ext:10211525922, loc:(*time.Location)(0x42417a0)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x400cc27, Device:0x801}, IdentifierName:"native"}, TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [data.id] of type [keyword] in document with id 'Nc_Th3sBkjI4v_RsZDGv'. Preview of field's value: '{orig_p=59480, resp_h=172.31.21.152, orig_h=107.189.31.98, resp_p=22}'","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:403"}}
2021-08-27T13:36:23.221Z    WARN    [elasticsearch] elasticsearch/client.go:408 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xc04258758bc795b8, ext:165343897440, loc:(*time.Location)(0x42417a0)}, Meta:{"pipeline":"filebeat-7.10.2-wazuh-alerts-pipeline"}, Fields:{"agent":{"ephemeral_id":"4dcb71b0-1628-4491-aefd-10910ed5a1c6","hostname":"manager","id":"21a55aae-0516-4eb7-aee7-86f0e8761c26","name":"manager","type":"filebeat","version":"7.10.2"},"ecs":{"version":"1.6.0"},"event":{"dataset":"wazuh.alerts","module":"wazuh"},"fields":{"index_prefix":"wazuh-alerts-4.x-"},"fileset":{"name":"alerts"},"host":{"name":"manager"},"input":{"type":"log"},"log":{"file":{"path":"/var/ossec/logs/alerts/alerts.json"},"offset":9484},"message":"{\"timestamp\":\"2021-08-25T04:49:14.093+0000\",\"rule\":{\"level\":5,\"description\":\"Zeek: Connection detail\",\"id\":\"66004\",\"firedtimes\":546,\"mail\":false,\"groups\":[\"zeek\",\"ids\"]},\"agent\":{\"id\":\"003\",\"name\":\"esign.kycaml.systems\",\"ip\":\"172.31.9.38\"},\"manager\":{\"name\":\"wazuh-manager\"},\"id\":\"1629866954.15752818\",\"full_log\":\"{\\\"ts\\\":1629866952.767386,\\\"uid\\\":\\\"CpTBQ1aO5afQRWU26\\\",\\\"id.orig_h\\\":\\\"172.31.9.38\\\",\\\"id.orig_p\\\":35688,\\\"id.resp_h\\\":\\\"52.95.16.182\\\",\\\"id.resp_p\\\":443,\\\"proto\\\":\\\"tcp\\\",\\\"conn_state\\\":\\\"OTH\\\",\\\"local_orig\\\":true,\\\"local_resp\\\":false,\\\"missed_bytes\\\":0,\\\"history\\\":\\\"C\\\",\\\"orig_pkts\\\":0,\\\"orig_ip_bytes\\\":0,\\\"resp_pkts\\\":0,\\\"resp_ip_bytes\\\":0,\\\"bro_engine\\\":\\\"CONN\\\"}\",\"decoder\":{\"name\":\"json\"},\"data\":{\"ts\":\"1629866952.767386\",\"uid\":\"CpTBQ1aO5afQRWU26\",\"id\":{\"orig_h\":\"172.31.9.38\",\"orig_p\":\"35688\",\"resp_h\":\"52.95.16.182\",\"resp_p\":\"443\"},\"proto\":\"tcp\",\"conn_state\":\"OTH\",\"local_orig\":\"true\",\"local_resp\":\"false\",\"missed_bytes\":\"0\",\"history\":\"C\",\"orig_pkts\":\"0\",\"orig_ip_bytes\":\"0\",\"resp_pkts\":\"0\",\"resp_ip_bytes\":\"0\",\"bro_engine\":\"CONN\"},\"location\":\"/usr/local/zeek/logs/current/conn.log\"}","service":{"type":"wazuh"}}, Private:file.State{Id:"native::67161127-2049", PrevId:"", Finished:false, Fileinfo:(*os.fileStat)(0xc000462a90), Source:"/var/ossec/logs/alerts/alerts.json", Offset:10593, Timestamp:time.Time{wall:0xc042584ec3e3c18f, ext:10211525922, loc:(*time.Location)(0x42417a0)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x400cc27, Device:0x801}, IdentifierName:"native"}, TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [data.id] of type [keyword] in document with id 'Ns_Th3sBkjI4v_RsZDGv'. Preview of field's value: '{orig_p=35688, resp_h=52.95.16.182, orig_h=172.31.9.38, resp_p=443}'","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:345"}}

I will analyze this with the team and get back to you as soon as possible.

dipenpatel235 commented 3 years ago

Thanks you @julian . We are waiting for your reply.

Thanks Dipen

On Fri, Aug 27, 2021, 7:16 PM Julian Morales @.***> wrote:

Hi @dipenpatel235 https://github.com/dipenpatel235 ,

First of all, I edited your previous comment to be able to see the logs as code blocks, this allows all the characters to be showed, I hope this doesn't bother you. I did the test of inserting these logs in the alerts.json file and analyzing the behavior of filebeat in debug mode. These logs are not displayed in kibana because elasticsearch cannot index them, due to a problem with the filbeat template. In Filebeat, you get the following logs:

2021-08-27T13:36:23.221Z WARN [elasticsearch] elasticsearch/client.go:408 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xc04258758bc036ad, ext:165343414356, loc:(time.Location)(0x42417a0)}, Meta:{"pipeline":"filebeat-7.10.2-wazuh-alerts-pipeline"}, Fields:{"agent":{"ephemeral_id":"4dcb71b0-1628-4491-aefd-10910ed5a1c6","hostname":"manager","id":"21a55aae-0516-4eb7-aee7-86f0e8761c26","name":"manager","type":"filebeat","version":"7.10.2"},"ecs":{"version":"1.6.0"},"event":{"dataset":"wazuh.alerts","module":"wazuh"},"fields":{"index_prefix":"wazuh-alerts-4.x-"},"fileset":{"name":"alerts"},"host":{"name":"manager"},"input":{"type":"log"},"log":{"file":{"path":"/var/ossec/logs/alerts/alerts.json"},"offset":7347},"message":"{\"timestamp\":\"2021-08-25T04:49:04.283+0000\",\"rule\":{\"level\":5,\"description\":\"Zeek: SSH Connection\",\"id\":\"66001\",\"firedtimes\":89,\"mail\":false,\"groups\":[\"zeek\",\"ids\"]},\"agent\":{\"id\":\"012\",\"name\":\"liveness.server\",\"ip\":\"172.31.21.152\"},\"manager\":{\"name\":\"wazuh-manager\"},\"id\":\"1629866944.15751292\",\"full_log\":\"{\\"ts\\":1629866937.6732931,\\"uid\\":\\"CbZm0X3Ntova265R3g\\",\\"id.orig_h\\":\\"176.111.173.85\\",\\"id.orig_p\\":17962,\\"id.resp_h\\":\\"172.31.21.152\\",\\"id.resp_p\\":22,\\"auth_attempts\\":0,\\"direction\\":\\"INBOUND\\",\\"client\\":\\"SSH-2.0-libssh_0.9.5\\",\\"bro_engine\\":\\"SSH\\"}\",\"decoder\":{\"name\":\"json\"},\"data\":{\"ts\":\"1629866937.673293\",\"uid\":\"CbZm0X3Ntova265R3g\",\"id\":{\"orig_h\":\"176.111.173.85\",\"orig_p\":\"17962\",\"resp_h\":\"172.31.21.152\",\"resp_p\":\"22\"},\"auth_attempts\":\"0\",\"direction\":\"INBOUND\",\"client\":\"SSH-2.0-libssh_0.9.5\",\"bro_engine\":\"SSH\"},\"location\":\"/usr/local/zeek/logs/current/ssh.log\"}","service":{"type":"wazuh"}}, Private:file.State{Id:"native::67161127-2049", PrevId:"", Finished:false, Fileinfo:(os.fileStat)(0xc000462a90), Source:"/var/ossec/logs/alerts/alerts.json", Offset:8241, Timestamp:time.Time{wall:0xc042584ec3e3c18f, ext:10211525922, loc:(time.Location)(0x42417a0)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x400cc27, Device:0x801}, IdentifierName:"native"}, TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [data.id] of type [keyword] in document with id 'NM_Th3sBkjI4v_RsZDGv'. Preview of field's value: '{orig_p=17962, resp_h=172.31.21.152, orig_h=176.111.173.85, resp_p=22}'","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:215"}} 2021-08-27T13:36:23.221Z WARN [elasticsearch] elasticsearch/client.go:408 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xc04258758bc4b3a5, ext:165343708492, loc:(time.Location)(0x42417a0)}, Meta:{"pipeline":"filebeat-7.10.2-wazuh-alerts-pipeline"}, Fields:{"agent":{"ephemeral_id":"4dcb71b0-1628-4491-aefd-10910ed5a1c6","hostname":"manager","id":"21a55aae-0516-4eb7-aee7-86f0e8761c26","name":"manager","type":"filebeat","version":"7.10.2"},"ecs":{"version":"1.6.0"},"event":{"dataset":"wazuh.alerts","module":"wazuh"},"fields":{"index_prefix":"wazuh-alerts-4.x-"},"fileset":{"name":"alerts"},"host":{"name":"manager"},"input":{"type":"log"},"log":{"file":{"path":"/var/ossec/logs/alerts/alerts.json"},"offset":8241},"message":"{\"timestamp\":\"2021-08-25T04:49:08.288+0000\",\"rule\":{\"level\":5,\"description\":\"Zeek: Connection detail\",\"id\":\"66004\",\"firedtimes\":545,\"mail\":false,\"groups\":[\"zeek\",\"ids\"]},\"agent\":{\"id\":\"012\",\"name\":\"liveness.server\",\"ip\":\"172.31.21.152\"},\"manager\":{\"name\":\"wazuh-manager\"},\"id\":\"1629866948.15751907\",\"full_log\":\"{\\"ts\\":1629866941.511791,\\"uid\\":\\"CYv7Rk33iFIqKurOfc\\",\\"id.orig_h\\":\\"107.189.31.98\\",\\"id.orig_p\\":59480,\\"id.resp_h\\":\\"172.31.21.152\\",\\"id.resp_p\\":22,\\"proto\\":\\"tcp\\",\\"duration\\":0.31515789031982422,\\"orig_bytes\\":0,\\"resp_bytes\\":0,\\"conn_state\\":\\"SH\\",\\"local_orig\\":false,\\"local_resp\\":true,\\"missed_bytes\\":0,\\"history\\":\\"ScAF\\",\\"orig_pkts\\":5,\\"orig_ip_bytes\\":268,\\"resp_pkts\\":0,\\"resp_ip_bytes\\":0,\\"bro_engine\\":\\"CONN\\"}\",\"decoder\":{\"name\":\"json\"},\"data\":{\"ts\":\"1629866941.511791\",\"uid\":\"CYv7Rk33iFIqKurOfc\",\"id\":{\"orig_h\":\"107.189.31.98\",\"orig_p\":\"59480\",\"resp_h\":\"172.31.21.152\",\"resp_p\":\"22\"},\"proto\":\"tcp\",\"duration\":\"0.315158\",\"orig_bytes\":\"0\",\"resp_bytes\":\"0\",\"conn_state\":\"SH\",\"local_orig\":\"false\",\"local_resp\":\"true\",\"missed_bytes\":\"0\",\"history\":\"ScAF\",\"orig_pkts\":\"5\",\"orig_ip_bytes\":\"268\",\"resp_pkts\":\"0\",\"resp_ip_bytes\":\"0\",\"bro_engine\":\"CONN\"},\"location\":\"/usr/local/zeek/logs/current/conn.log\"}","service":{"type":"wazuh"}}, Private:file.State{Id:"native::67161127-2049", PrevId:"", Finished:false, Fileinfo:(os.fileStat)(0xc000462a90), Source:"/var/ossec/logs/alerts/alerts.json", Offset:9484, Timestamp:time.Time{wall:0xc042584ec3e3c18f, ext:10211525922, loc:(time.Location)(0x42417a0)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x400cc27, Device:0x801}, IdentifierName:"native"}, TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [data.id] of type [keyword] in document with id 'Nc_Th3sBkjI4v_RsZDGv'. Preview of field's value: '{orig_p=59480, resp_h=172.31.21.152, orig_h=107.189.31.98, resp_p=22}'","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:403"}} 2021-08-27T13:36:23.221Z WARN [elasticsearch] elasticsearch/client.go:408 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xc04258758bc795b8, ext:165343897440, loc:(time.Location)(0x42417a0)}, Meta:{"pipeline":"filebeat-7.10.2-wazuh-alerts-pipeline"}, Fields:{"agent":{"ephemeral_id":"4dcb71b0-1628-4491-aefd-10910ed5a1c6","hostname":"manager","id":"21a55aae-0516-4eb7-aee7-86f0e8761c26","name":"manager","type":"filebeat","version":"7.10.2"},"ecs":{"version":"1.6.0"},"event":{"dataset":"wazuh.alerts","module":"wazuh"},"fields":{"index_prefix":"wazuh-alerts-4.x-"},"fileset":{"name":"alerts"},"host":{"name":"manager"},"input":{"type":"log"},"log":{"file":{"path":"/var/ossec/logs/alerts/alerts.json"},"offset":9484},"message":"{\"timestamp\":\"2021-08-25T04:49:14.093+0000\",\"rule\":{\"level\":5,\"description\":\"Zeek: Connection detail\",\"id\":\"66004\",\"firedtimes\":546,\"mail\":false,\"groups\":[\"zeek\",\"ids\"]},\"agent\":{\"id\":\"003\",\"name\":\"esign.kycaml.systems\",\"ip\":\"172.31.9.38\"},\"manager\":{\"name\":\"wazuh-manager\"},\"id\":\"1629866954.15752818\",\"full_log\":\"{\\"ts\\":1629866952.767386,\\"uid\\":\\"CpTBQ1aO5afQRWU26\\",\\"id.orig_h\\":\\"172.31.9.38\\",\\"id.orig_p\\":35688,\\"id.resp_h\\":\\"52.95.16.182\\",\\"id.resp_p\\":443,\\"proto\\":\\"tcp\\",\\"conn_state\\":\\"OTH\\",\\"local_orig\\":true,\\"local_resp\\":false,\\"missed_bytes\\":0,\\"history\\":\\"C\\",\\"orig_pkts\\":0,\\"orig_ip_bytes\\":0,\\"resp_pkts\\":0,\\"resp_ip_bytes\\":0,\\"bro_engine\\":\\"CONN\\"}\",\"decoder\":{\"name\":\"json\"},\"data\":{\"ts\":\"1629866952.767386\",\"uid\":\"CpTBQ1aO5afQRWU26\",\"id\":{\"orig_h\":\"172.31.9.38\",\"orig_p\":\"35688\",\"resp_h\":\"52.95.16.182\",\"resp_p\":\"443\"},\"proto\":\"tcp\",\"conn_state\":\"OTH\",\"local_orig\":\"true\",\"local_resp\":\"false\",\"missed_bytes\":\"0\",\"history\":\"C\",\"orig_pkts\":\"0\",\"orig_ip_bytes\":\"0\",\"resp_pkts\":\"0\",\"resp_ip_bytes\":\"0\",\"bro_engine\":\"CONN\"},\"location\":\"/usr/local/zeek/logs/current/conn.log\"}","service":{"type":"wazuh"}}, Private:file.State{Id:"native::67161127-2049", PrevId:"", Finished:false, Fileinfo:(os.fileStat)(0xc000462a90), Source:"/var/ossec/logs/alerts/alerts.json", Offset:10593, Timestamp:time.Time{wall:0xc042584ec3e3c18f, ext:10211525922, loc:(*time.Location)(0x42417a0)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x400cc27, Device:0x801}, IdentifierName:"native"}, TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [data.id] of type [keyword] in document with id 'Ns_Th3sBkjI4v_RsZDGv'. Preview of field's value: '{orig_p=35688, resp_h=52.95.16.182, orig_h=172.31.9.38, resp_p=443}'","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:345"}}

I will analyze this with the team and get back to you as soon as possible.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/wazuh/wazuh/issues/9662#issuecomment-907215128, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB6CDOK57I5K37OAFGACOIDT66JJRANCNFSM5B6IICRQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

juliancnn commented 3 years ago

Hi @dipenpatel235 ,

Sorry for the late reply, we have discussed this with the team and decided to mark this as a bug. We have also created an issue to better detail the problem and to better track it: #9949. Any information related to this problem will be integrated into issue #9949.

In addition to this, I have created issue https://github.com/wazuh/wazuh-documentation/issues/4213 in the documentation repository to make an analysis of the field types, this will help people to identify similar problems.

I will close this issue as the problem is identified and reported with a new issue. Feel free to create new issues for new concerns, we are here to help.

Regards, Julian

dipenpatel235 commented 3 years ago

Hello @julian, Thanks for the update.

Waiting for your update now

Thank you so much to you and your team

Thanks Dipen

On Tue, Aug 31, 2021, 8:03 PM Julian Morales @.***> wrote:

Hi @dipenpatel235 https://github.com/dipenpatel235 ,

Sorry for the late reply, we have discussed this with the team and decided to mark this as a bug. We have also created an issue to better detail the problem and to better track it: #9949 https://github.com/wazuh/wazuh/issues/9949. Any information related to this problem will be integrated into issue

9949 https://github.com/wazuh/wazuh/issues/9949.

In addition to this, I have created issue wazuh/wazuh-documentation#4213 https://github.com/wazuh/wazuh-documentation/issues/4213 in the documentation repository to make an analysis of the field types, this will help people to identify similar problems.

I will close this issue as the problem is identified and reported with a new issue. Feel free to create new issues for new concerns, we are here to help.

Regards, Julian

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/wazuh/wazuh/issues/9662#issuecomment-909294247, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB6CDOI455S4MPLC4ZI7DGTT7TRZXANCNFSM5B6IICRQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

dipenpatel235 commented 1 year ago

Hello @juliancnn,

I am waiting for your update very long time. Is there any update?

Thanks, Dipen

juliancnn commented 1 year ago

Hi @dipenpatel235,

We know that there is a limitation with custom fields and we also know that this is sometimes a problem when indexing. This is why we are developing a proof of concept https://github.com/wazuh/wazuh/issues/11334 of a new log analysis engine (currently under development, which, perhaps, in the future could replace analisysd).

The new engine tries to handle json, when it receives any type of log, it adds it to a json field. This event enters in the chain of operations that will be defined by the 'Assets' (they would be like filters, decoders and rules). These assets will be defined by the user in YMl documents and will perform the task of enriching and normalizing the event (Removing the indexing problem and standardizing the name of the fields for further analysis). The engine aims to be as transparent as possible, receiving the events as they are picked up by the agent, and inserting them into a chain of operations that are easy to define, read and manipulate by the users. We want there to be no "hidden" or unavoidable manipulations, such as the predecoder or embedded decoders of Wazuh-Analisis.

I encourage you to take a look at the epic and share your thoughts with us!