Dates and timestamps are not correct

erezhazan1 commented 3 years ago

Describe the problem

For some reason, since I'm the only one that's opening an issue on this, the logs that I'm getting from Winlogbeat are with a wrong timestamp, which is in some cases a day ahead, so I need to search for logs that are 1 day ahead of me. Also the event_original_time, and the event_recorded_time are with wrong dates (about almost 1 day ahead). The only right date stamp is the etl_processed_time.

@timestamp - 2020-09-11 @ 19:50:32.816 event_original_time - 2020-09-11T16:50:32.816Z event_recorded_time - 2020-09-11T06:52:32.322Z

etl_processed_time - 2020-09-11 @ 09:52:35.563

in the Even original message, I have the UtcTime: 2020-09-11 16:50:32.816

--

Provide the output of the following commands

Get operating system and version for linux (except Mac) use:
cat /etc/os-release
for Mac/OSX use:
sw_vers
Get disk space, memory, processor cores, and docker storage
echo -e "\nDocker Space:" && df -h /var/lib/docker; echo -e "\nMemory:" && free -g; echo -e "\nCores:" && getconf _NPROCESSORS_ONLN
Get output of the HELK docker containers:
docker ps --filter "name=helk"

NAME="Ubuntu"
VERSION="18.04.4 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.4 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

Docker Space:
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1       97G   20G   78G  20% /

Memory:
              total        used        free      shared  buff/cache   available
Mem:             15          12           0           0           3           2
Swap:             0           0           0

Cores:
4

CONTAINER ID        IMAGE                                                 COMMAND                  CREATED             STATUS              PORTS                                                                                                                                                                                                  NAMES
f6461a6220d8        confluentinc/cp-ksql-cli:5.1.3                        "/bin/sh"                2 days ago          Up 2 days                                                                                                                                                                                                                  helk-ksql-cli
3e04ea417574        confluentinc/cp-ksql-server:5.1.3                     "/etc/confluent/dock…"   2 days ago          Up 2 days           0.0.0.0:8088->8088/tcp                                                                                                                                                                                 helk-ksql-server
48d2a4c4cdb8        otrf/helk-spark-worker:2.4.5                          "./spark-worker-entr…"   2 days ago          Up 2 days                                                                                                                                                                                                                  helk-spark-worker
c805931a65e0        otrf/helk-kafka-broker:2.4.0                          "./kafka-entrypoint.…"   2 days ago          Up 2 days           0.0.0.0:9092->9092/tcp                                                                                                                                                                                 helk-kafka-broker
6038ed8421d8        otrf/helk-spark-master:2.4.5                          "./spark-master-entr…"   2 days ago          Up 2 days           7077/tcp, 0.0.0.0:8080->8080/tcp                                                                                                                                                                       helk-spark-master
d0d54e9552af        docker_helk-jupyter                                   "/opt/jupyter/script…"   2 days ago          Up 2 days           8000/tcp, 8888/tcp                                                                                                                                                                                     helk-jupyter
180eed09cb6f        otrf/helk-elastalert:0.4.0                            "./elastalert-entryp…"   2 days ago          Up 2 days                                                                                                                                                                                                                  helk-elastalert
5809266d129d        otrf/helk-zookeeper:2.4.0                             "./zookeeper-entrypo…"   2 days ago          Up 2 days           2181/tcp, 2888/tcp, 3888/tcp                                                                                                                                                                           helk-zookeeper
a9fbc5eb2d89        otrf/helk-nginx:0.3.0                                 "/opt/helk/scripts/n…"   2 days ago          Up 2 days           0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp                                                                                                                                                               helk-nginx
f242e14784f4        otrf/helk-logstash:7.6.2.1                            "/usr/share/logstash…"   2 days ago          Up 2 days           0.0.0.0:3515->3515/tcp, 0.0.0.0:5044->5044/tcp, 0.0.0.0:5514->5514/tcp, 0.0.0.0:5514->5514/udp, 0.0.0.0:8515-8516->8515-8516/tcp, 0.0.0.0:8531->8531/tcp, 0.0.0.0:8515-8516->8515-8516/udp, 9600/tcp   helk-logstash
32d80b62f651        docker.elastic.co/kibana/kibana:7.6.2                 "/usr/share/kibana/s…"   2 days ago          Up 2 days           5601/tcp                                                                                                                                                                                               helk-kibana
fd3b66dfba60        docker.elastic.co/elasticsearch/elasticsearch:7.6.2   "/usr/share/elastics…"   2 days ago          Up 2 days           9200/tcp, 9300/tcp                                                                                                                                                                                     helk-elasticsearch

Provide the HELK installation logs located at /var/log/helk-install.log if you are having install errors

Place the output here

What version of HELK are you using

run the command from within the HELK root directory cat .git/refs/heads/master
and include what date you cloned the HELK repo

couple of days ago

What version of Winlogbeat are you using if you are using Windows/WEF logs

Version 7.6.2

What steps did you take trying to fix the issue

Recreate the Index with default timestamp of etl_processed_time

How could we replicate the issue

Fresh eval Windows 10, version 1809 with sysmon64 installed and Winlogbeat Version 7.6.2, and nothing more.

Any additionally code or log context you would like to provide

Get-Content C:\ProgramData\winlogbeat\logs\winlogbeat -Tail 10 -Wait

following is a snipped timestamp from the command: 
2020-09-11T10:12:26.314+0300    INFO    beater/eventlogger.go:86        EventLog[Microsoft-windows-sysmon/operational] successfully published 1 events

Any additional context or input you have

I also set-up the HELK with neu5ron repo and Winlogbeat Version 7.7.1, and I got the same output

Just as FYI if it helps I did it initially with this guide:

https://www.blackhillsinfosec.com/how-to-deploy-windows-optics-commands-downloads-instructions-and-screenshots/

but without the WEF / WEC part.

erezhazan1 commented 3 years ago

Hey! I just read this medium article https://medium.com/@7a616368/replaying-windows-event-logs-against-elastalert-and-sigma-rules-using-helk-6db6edde6760

That exactly describes the issue that I have.

"Elasticsearch will naturally use the original event time as the data for the @timestamp field"

"To get around this, we can actually use the timestamp_field in the rule definition itself to tell Elastalert to look at the time stamp of ingestion into HELK, rather than the true event time as by adding the line below: timestamp_field: etl_processed_time"

Which means that the time field etl_processed_time gives the correct time of the event, but when searching for events, it gives me the time by the time field @timestamp, which is not the correct one.

Does anyone have the same issue?

neu5ron commented 3 years ago

@erezhazan1 what timezone is HELK using? what timezones are the logs coming in as? are you using NTP - and verify everything in sync? time stamps are really hard if not impossible, some of the above may help the issue - but if it's not just some simple timezone things - it's best to use UTC as reported from the box.. and why we add a lot of timestamps in the pipeline re-open if I can help anymore

Cyb3rWard0g / HELK