Issue with Parsing Windows Event Logs || Very Slow Parsing with Errors

[Aboalfadl]

Hello there, At first i want to thank you about this great tool, i saw the videos and the descriptions and it's really will be very helpful and great. But there is a very big issue i don't know what is the reason of it but let's see.

If i tried to upload any Security.evtx log for sure if the log was not cleared for more than 5 or 7 months you will have lot of records. Now i can't import any security event log or the most correct word here the tool can't parse my security event logs because these logs contains more than 250K records and maybe with millions. so such logs like these are you in consideration to make your tool handle this or not ? as example this is an error came from parsing one security event:

[*] Script start. 2019/11/13 05:40:09
[*] Time zone is 4.
[*] Last record number is 8743702.
[*] Start parsing the EVTX file.
[*] Parse the EVTX file Security.evtx.
[*] Now loading 10900 records.Traceback (most recent call last):
  File "../LogonTracer/logontracer.py", line 1096, in <module>
    main()
  File "../LogonTracer/logontracer.py", line 1084, in main
    parse_evtx(args.evtx)
  File "../LogonTracer/logontracer.py", line 642, in parse_evtx
    for node, err in xml_records(evtx_file):
  File "../LogonTracer/logontracer.py", line 532, in xml_records
    for xml, record in evtx_file_xml_view(evtx.get_file_header()):
  File "/usr/local/lib/python3.7/dist-packages/Evtx/Views.py", line 240, in evtx_file_xml_view
    record_str = evtx_record_xml_view(record)
  File "/usr/local/lib/python3.7/dist-packages/Evtx/Views.py", line 204, in evtx_record_xml_view
    return render_root_node(record.root())
  File "/usr/local/lib/python3.7/dist-packages/Evtx/Views.py", line 191, in render_root_node
    return render_root_node_with_subs(root_node, subs)
  File "/usr/local/lib/python3.7/dist-packages/Evtx/Views.py", line 176, in render_root_node_with_subs
    rec(c, acc)
  File "/usr/local/lib/python3.7/dist-packages/Evtx/Views.py", line 126, in rec
    rec(child, acc)
  File "/usr/local/lib/python3.7/dist-packages/Evtx/Views.py", line 166, in rec
    sub = render_root_node(sub.root())
  File "/usr/local/lib/python3.7/dist-packages/Evtx/Views.py", line 191, in render_root_node
    return render_root_node_with_subs(root_node, subs)
  File "/usr/local/lib/python3.7/dist-packages/Evtx/Views.py", line 176, in render_root_node_with_subs
    rec(c, acc)
  File "/usr/local/lib/python3.7/dist-packages/Evtx/Views.py", line 126, in rec
    rec(child, acc)
  File "/usr/local/lib/python3.7/dist-packages/Evtx/Views.py", line 126, in rec
    rec(child, acc)
  File "/usr/local/lib/python3.7/dist-packages/Evtx/Views.py", line 159, in rec
    sub = escape_value(sub.string())
  File "/usr/local/lib/python3.7/dist-packages/Evtx/Nodes.py", line 1118, in string
    return self._string().rstrip("\x00")
  File "/usr/local/lib/python3.7/dist-packages/Evtx/BinaryParser.py", line 211, in explicit_length_handler
    return f(offset, length)
  File "/usr/local/lib/python3.7/dist-packages/Evtx/BinaryParser.py", line 490, in unpack_wstring
    return bytes(self._buf[start:end]).decode("utf16")
UnicodeDecodeError: 'utf-16-le' codec can't decode bytes in position 2-3: illegal encoding

if i tried to parse another log smaller than this event log because this last one contains million of records, it will parse the records very slow as you can see here this log contains 352133 record and it contains all event ids that you may need but it took lot of time without anything working it parse like 100 record/second so if you want to parse 100000 record you will need 1000 second so approximate time to parse all of this log will be like 60 Minutes to parse this log. it's very huge time to just parse one log!!

also in the coming error you will find that the log that contains 352133 has not been parsed as well your tool just told that

[*] Now loading 31800 records.
[*] Load finished.
[*] Total Event log is 31840.

However the log contains more than 350K record.

kindly check the next ouput after trying to parse this security event log that contains also most of important event ids.

[*] Script start. 2019/11/13 05:46:28
[*] Time zone is 4.
[*] Last record number is 352133.
[*] Start parsing the EVTX file.
[*] Parse the EVTX file /Security.evtx.
[*] Now loading 31800 records.
[*] Load finished.
[*] Total Event log is 31840.
[*] Calculate ChangeFinder.
[*] Calculate Hidden Markov Model.
/usr/local/lib/python3.7/dist-packages/hmmlearn/hmm.py:412: RuntimeWarning: divide by zero encountered in log
  return np.log(self.emissionprob_)[:, np.concatenate(X)].T
[*] Calculate PageRank.
[*] Creating a graph data.
../LogonTracer/logontracer.py:956: DeprecationWarning: Transaction.append(...) is deprecated, use Transaction.run(...) instead
  tx.append(statement_ip, {"IP": ipaddress, "rank": ranks[ipaddress], "hostname": hostname})
../LogonTracer/logontracer.py:989: DeprecationWarning: Transaction.append(...) is deprecated, use Transaction.run(...) instead
  "detect": ",".join(map(str, detects[i]))})
../LogonTracer/logontracer.py:994: DeprecationWarning: Transaction.append(...) is deprecated, use Transaction.run(...) instead
  tx.append(statement_domain, {"domain": domain})
../LogonTracer/logontracer.py:999: DeprecationWarning: Transaction.append(...) is deprecated, use Transaction.run(...) instead
  "status": events["status"], "count": events["count"], "authname": events["authname"], "date": events["date"]})
../LogonTracer/logontracer.py:1003: DeprecationWarning: Transaction.append(...) is deprecated, use Transaction.run(...) instead
  tx.append(statement_dr, {"user": username[:-1], "domain": domain})
../LogonTracer/logontracer.py:1007: DeprecationWarning: Transaction.append(...) is deprecated, use Transaction.run(...) instead
  "end": datetime.datetime(*endtime.timetuple()[:4]).strftime("%Y-%m-%d %H:%M:%S")})
[*] Creation of a graph data finished.
[*] Script end. 2019/11/13 05:54:38

Also after this nothing happened on the GUI and nothing uploaded or parsed.

If you can solve this it will be very nice.

Thanks a lot.

[shu-tom]

Thank you for using our tool. The 1st error message is come from python module python-evtx. It's looks like EVTX file is corrupted. https://github.com/williballenthin/python-evtx/issues/43

The 2nd error message is neo4j database connection failure. What is your neo4j version?

Tips: Importing the EVTX file takes time to parse the file. To speed up, please convert the EVTX file to the XML file with the Event Viewer and import it.

[Aboalfadl]

Thank you for using our tool. The 1st error message is come from python module python-evtx. It's looks like EVTX file is corrupted. williballenthin/python-evtx#43

The 2nd error message is neo4j database connection failure. What is your neo4j version?

Tips: Importing the EVTX file takes time to parse the file. To speed up, please convert the EVTX file to the XML file with the Event Viewer and import it.

I just installed Neo4j yesterday and it was the latest version Neo4j Browser version: 3.2.20 Neo4j Server version: 3.5.12

Also my apt is update d and upgraded to the latest thing so i have no issue with my python or anything related to my python and why i can say that the file was no corrupted because i can open it with event viewer and with windows event viewer(another tool not from Microsoft) so the logs are not corrupted.

Also i have created a user with password on the neo4j and there is no connection error also i do not know why !

[Aboalfadl]

i extracted a new log from windows log viewer with xml format and uploaded it using the GUI and using the command lines and i stucked here:

/usr/local/lib/python3.7/dist-packages/hmmlearn/hmm.py:412: RuntimeWarning: divide by zero encountered in log return np.log(self.emissionprob_)[:, np.concatenate(X)].T 
[*] Load finished. 
[*] Total Event log is 31470. 
[*] Calculate ChangeFinder.
[*] Calculate Hidden Markov Model. 
[*] Calculate PageRank. [*] Creating a graph data. 
[*] Creation of a graph data finished. 
[*] Script end. 2019/11/14 17:51:29

i searched how to fix but also i found nothing , do you have a solution for this ?

[shu-tom]

hmmlearn message is warning not error. There is no error in LogonTracer, so there may be an issue with the event log.

[Aboalfadl]

i tried log of different event logs ! nothing showed in the GUI , it failed every time

[Aboalfadl]

After downloading the tool after updating i tried again and also with the same error/warning and nothing loaded on the tool and nothing to view.

[+] Script start. 2019/11/15 05:52:02
[+] Neo4j Kernel version: 3.5.12
[+] Delete all nodes and relationships from this Neo4j database.
[+] Time zone is 2.
[+] Last record number is 31470.
[+] Start parsing the EVTX file.
[+] Parse the EVTX file Security.xml.
[+] Now loading 31400 records.
[+] Load finished.
[+] Total Event log is 31470.
[+] Fildered Event log is 76.
[+] Calculate ChangeFinder.
[+] Calculate Hidden Markov Model.
/usr/local/lib/python3.7/dist-packages/hmmlearn/hmm.py:412: RuntimeWarning: divide by zero encountered in log
  return np.log(self.emissionprob_)[:, np.concatenate(X)].T
[+] Calculate PageRank.
[+] Creating a graph data.
[+] Creation of a graph data finished.
[+] Script end. 2019/11/15 05:52:08

[shu-tom]

Please check web browser console logs.

[Aboalfadl]

It is the same my friend:

[+] Script start. 2019/11/15 05:50:46 [+] Neo4j Kernel version: 3.5.12 [+] Delete all nodes and relationships from this Neo4j database. [+] Time zone is 2. [+] Last record number is 31526. [+] Start parsing the EVTX file. [+] Parse the EVTX file /root/Hacker/Tools/Forensics/LogonTracer/upload/0.evtx. [+] Now loading 100 records. [+] Now loading 200 records. [+] Now loading 300 records. [+] Now loading 400 records. [+] Now loading 500 records. [+] Now loading 600 records. [+] Now loading 700 records. [+] Now loading 800 records. [+] Now loading 900 records. [+] Now loading 1000 records. [+] Now loading 1100 records. [+] Now loading 1200 records. [+] Now loading 1300 records. [+] Now loading 1400 records. [+] Now loading 1500 records. [+] Now loading 1600 records. [+] Now loading 1700 records. [+] Now loading 1800 records. [+] Now loading 1900 records. [+] Now loading 2000 records. [+] Now loading 2100 records. [+] Now loading 2200 records. [+] Now loading 2300 records. [+] Now loading 2400 records. [+] Now loading 2500 records. [+] Now loading 2600 records. [+] Now loading 2700 records. [+] Now loading 2800 records. [+] Now loading 2900 records. [+] Now loading 3000 records. [+] Now loading 3100 records. [+] Now loading 3200 records. [+] Now loading 3300 records. [+] Now loading 3400 records. [+] Now loading 3500 records. [+] Now loading 3600 records. [+] Now loading 3700 records. [+] Now loading 3800 records. [+] Now loading 3900 records. [+] Now loading 4000 records. [+] Now loading 4100 records. [+] Now loading 4200 records. [+] Now loading 4300 records. [+] Now loading 4400 records. [+] Now loading 4500 records. [+] Now loading 4600 records. [+] Now loading 4700 records. [+] Now loading 4800 records. [+] Now loading 4900 records. [+] Now loading 5000 records. [+] Now loading 5100 records. [+] Now loading 5200 records. [+] Now loading 5300 records. [+] Now loading 5400 records. [+] Now loading 5500 records. [+] Now loading 5600 records. [+] Now loading 5700 records. [+] Now loading 5800 records. [+] Now loading 5900 records. [+] Now loading 6000 records. [+] Now loading 6100 records. [+] Now loading 6200 records. [+] Now loading 6300 records. [+] Now loading 6400 records. [+] Now loading 6500 records. [+] Now loading 6600 records. [+] Now loading 6700 records. [+] Now loading 6800 records. [+] Now loading 6900 records. [+] Now loading 7000 records. [+] Now loading 7100 records. [+] Now loading 7200 records. [+] Now loading 7300 records. [+] Now loading 7400 records. [+] Now loading 7500 records. [+] Now loading 7600 records. [+] Now loading 7700 records. [+] Now loading 7800 records. [+] Now loading 7900 records. [+] Now loading 8000 records. [+] Now loading 8100 records. [+] Now loading 8200 records. [+] Now loading 8300 records. [+] Now loading 8400 records. [+] Now loading 8500 records. [+] Now loading 8600 records. [+] Now loading 8700 records. [+] Now loading 8800 records. [+] Now loading 8900 records. [+] Now loading 9000 records. [+] Now loading 9100 records. [+] Now loading 9200 records. [+] Now loading 9300 records. [+] Now loading 9400 records. [+] Now loading 9500 records. [+] Now loading 9600 records. [+] Now loading 9700 records. [+] Now loading 9800 records. [+] Now loading 9900 records. [+] Now loading 10000 records. [+] Now loading 10100 records. [+] Now loading 10200 records. [+] Now loading 10300 records. [+] Now loading 10400 records. [+] Now loading 10500 records. [+] Now loading 10600 records. [+] Now loading 10700 records. [+] Now loading 10800 records. [+] Now loading 10900 records. [+] Now loading 11000 records. [+] Now loading 11100 records. [+] Now loading 11200 records. [+] Now loading 11300 records. [+] Now loading 11400 records. [+] Now loading 11500 records. [+] Now loading 11600 records. [+] Now loading 11700 records. [+] Now loading 11800 records. [+] Now loading 11900 records. [+] Now loading 12000 records. [+] Now loading 12100 records. [+] Now loading 12200 records. [+] Now loading 12300 records. [+] Now loading 12400 records. [+] Now loading 12500 records. [+] Now loading 12600 records. [+] Now loading 12700 records. [+] Now loading 12800 records. [+] Now loading 12900 records. [+] Now loading 13000 records. [+] Now loading 13100 records. [+] Now loading 13200 records. [+] Now loading 13300 records. [+] Now loading 13400 records. [+] Now loading 13500 records. [+] Now loading 13600 records. [+] Now loading 13700 records. [+] Now loading 13800 records. [+] Now loading 13900 records. [+] Now loading 14000 records. [+] Now loading 14100 records. [+] Now loading 14200 records. [+] Now loading 14300 records. [+] Now loading 14400 records. [+] Now loading 14500 records. [+] Now loading 14600 records. [+] Now loading 14700 records. [+] Now loading 14800 records. [+] Now loading 14900 records. [+] Now loading 15000 records. [+] Now loading 15100 records. [+] Now loading 15200 records. [+] Now loading 15300 records. [+] Now loading 15400 records. [+] Now loading 15500 records. [+] Now loading 15600 records. [+] Now loading 15700 records. [+] Now loading 15800 records. [+] Now loading 15900 records. [+] Now loading 16000 records. [+] Now loading 16100 records. [+] Now loading 16200 records. [+] Now loading 16300 records. [+] Now loading 16400 records. [+] Now loading 16500 records. [+] Now loading 16600 records. [+] Now loading 16700 records. [+] Now loading 16800 records. [+] Now loading 16900 records. [+] Now loading 17000 records. [+] Now loading 17100 records. [+] Now loading 17200 records. [+] Now loading 17300 records. [+] Now loading 17400 records. [+] Now loading 17500 records. [+] Now loading 17600 records. [+] Now loading 17700 records. [+] Now loading 17800 records. [+] Now loading 17900 records. [+] Now loading 18000 records. [+] Now loading 18100 records. [+] Now loading 18200 records. [+] Now loading 18300 records. [+] Now loading 18400 records. [+] Now loading 18500 records. [+] Now loading 18600 records. [+] Now loading 18700 records. [+] Now loading 18800 records. [+] Now loading 18900 records. [+] Now loading 19000 records. [+] Now loading 19100 records. [+] Now loading 19200 records. [+] Now loading 19300 records. [+] Now loading 19400 records. [+] Now loading 19500 records. [+] Now loading 19600 records. [+] Now loading 19700 records. [+] Now loading 19800 records. [+] Now loading 19900 records. [+] Now loading 20000 records. [+] Now loading 20100 records. [+] Now loading 20200 records. [+] Now loading 20300 records. [+] Now loading 20400 records. [+] Now loading 20500 records. [+] Now loading 20600 records. [+] Now loading 20700 records. [+] Now loading 20800 records. [+] Now loading 20900 records. [+] Now loading 21000 records. [+] Now loading 21100 records. [+] Now loading 21200 records. [+] Now loading 21300 records. [+] Now loading 21400 records. [+] Now loading 21500 records. [+] Now loading 21600 records. [+] Now loading 21700 records. [+] Now loading 21800 records. [+] Now loading 21900 records. [+] Now loading 22000 records. [+] Now loading 22100 records. [+] Now loading 22200 records. [+] Now loading 22300 records. [+] Now loading 22400 records. [+] Now loading 22500 records. [+] Now loading 22600 records. [+] Now loading 22700 records. [+] Now loading 22800 records. [+] Now loading 22900 records. [+] Now loading 23000 records. [+] Now loading 23100 records. [+] Now loading 23200 records. [+] Now loading 23300 records. [+] Now loading 23400 records. [+] Now loading 23500 records. [+] Now loading 23600 records. [+] Now loading 23700 records. [+] Now loading 23800 records. [+] Now loading 23900 records. [+] Now loading 24000 records. [+] Now loading 24100 records. [+] Now loading 24200 records. [+] Now loading 24300 records. [+] Now loading 24400 records. [+] Now loading 24500 records. [+] Now loading 24600 records. [+] Now loading 24700 records. [+] Now loading 24800 records. [+] Now loading 24900 records. [+] Now loading 25000 records. [+] Now loading 25100 records. [+] Now loading 25200 records. [+] Now loading 25300 records. [+] Now loading 25400 records. [+] Now loading 25500 records. [+] Now loading 25600 records. [+] Now loading 25700 records. [+] Now loading 25800 records. [+] Now loading 25900 records. [+] Now loading 26000 records. [+] Now loading 26100 records. [+] Now loading 26200 records. [+] Now loading 26300 records. [+] Now loading 26400 records. [+] Now loading 26500 records. [+] Now loading 26600 records. [+] Now loading 26700 records. [+] Now loading 26800 records. [+] Now loading 26900 records. [+] Now loading 27000 records. [+] Now loading 27100 records. [+] Now loading 27200 records. [+] Now loading 27300 records. [+] Now loading 27400 records. [+] Now loading 27500 records. [+] Now loading 27600 records. [+] Now loading 27700 records. [+] Now loading 27800 records. [+] Now loading 27900 records. [+] Now loading 28000 records. [+] Now loading 28100 records. [+] Now loading 28200 records. [+] Now loading 28300 records. [+] Now loading 28400 records. [+] Now loading 28500 records. [+] Now loading 28600 records. [+] Now loading 28700 records. [+] Now loading 28800 records. [+] Now loading 28900 records. [+] Now loading 29000 records. [+] Now loading 29100 records. [+] Now loading 29200 records. [+] Now loading 29300 records. [+] Now loading 29400 records. [+] Now loading 29500 records. [+] Now loading 29600 records. [+] Now loading 29700 records. [+] Now loading 29800 records. [+] Now loading 29900 records. [+] Now loading 30000 records. [+] Now loading 30100 records. [+] Now loading 30200 records. [+] Now loading 30300 records. [+] Now loading 30400 records. [+] Now loading 30500 records. [+] Now loading 30600 records. [+] Now loading 30700 records. [+] Now loading 30800 records. [+] Now loading 30900 records. [+] Now loading 31000 records. [+] Now loading 31100 records. [+] Now loading 31200 records. [+] Now loading 31300 records. [+] Now loading 31400 records. [+] Now loading 31500 records./usr/local/lib/python3.7/dist-packages/hmmlearn/hmm.py:412: RuntimeWarning: divide by zero encountered in log return np.log(self.emissionprob_)[:, np.concatenate(X)].T [+] Load finished. [+] Total Event log is 31526. [+] Fildered Event log is 76. [+] Calculate ChangeFinder. [+] Calculate Hidden Markov Model. [+] Calculate PageRank. [+] Creating a graph data. [+] Creation of a graph data finished. [+] Script end. 2019/11/15 05:54:07

[shu-tom]

Can you share the EVTX file to me in order to resolve this issue? If you can share it please send to logontracer.help (at) gmail.com

[Aboalfadl]

Done i sent the mail.

[Aboalfadl]

The issue was solved by uploading and doing everything on the same machine. Neo4j has connection error using the WebSocket. i figured out that i can't upload logs or use the application from another IP on the network i should do everything from the same machine and i didn't know how to fix it but finally i made the tool work with me. Thanks a lot.