Output/Upload CSV

[DYarizadeh]

Is there a way to upload the results to Velociraptor as a CSV, instead of .json?

The output is hard to parse/unusable, especially when attempting to scan multiple systems. Would like to pivot the output to a tool like Timeline Explorer and sort results by severity.

[pH-T]

hi @DYarizadeh ,

have you tried https://thor-util-manual.nextron-systems.com/en/latest/usage/log-conversion.html ?

we might add this in the future to the artifact as default...

thanks for reporting!

[DYarizadeh]

Thank you for your response.

When running with the flags --from-json --to-csv, I get the error "line [line number]: level must be a string"

Where [line number] is a specific line(s)

[pH-T]

can you show me the full cmdline output?

[DYarizadeh]

.\thor-util.exe logconvert --from-json --file "C:\Users\DYARI\Downloads\ThorOutput\results\ThorOutput.json" --to-csv --output ThorOutputConvert.csv

Nov 9 15:13:41 THOR THOR_UTIL: Warning: could not parse 'C:\Users\DYARI\Downloads\ThorOutput\results\ThorOutput.json', line 1: level must be a string

It then continues to line 2, line 3, line 4, ect...

Thank you again for your help!

[pH-T]

hi @DYarizadeh , i just tried to reproduce:

1. downloaded .json files (linux and windows) which were uploaded to velociraptor
2. used ./thor-util logconvert --from-json --file ~/Downloads/DESKTOP-PVRH3U5_thor_2023-10-30_0929.json --to-csv --output test.csv to convert -
3. --> no issues

head of the linux json file for comparison:

> head ~/Downloads/bullseye_thor_2023-10-30_0829.json 
{"time":"2023-10-30T08:29:40Z","hostname":"bullseye","level":"Notice","module":"Init","message":"Some modules and features are not available in Lite version and will be disabled","scanid":"S-hNuFlx14n74","log_version":1}
{"time":"2023-10-30T08:29:40Z","hostname":"bullseye","level":"Notice","module":"Startup","message":"This THOR Lite license permits non-commercial use only. It is strictly prohibited to sell THOR Lite or sell services that include the use of THOR Lite. For details, see the EULA in the ./docs folder. For a special license that covers these cases, allows Sigma scanning and suppresses this message, please contact our sales via https://www.nextron-systems.com/get-started/","scanid":"S-hNuFlx14n74","log_version":1}
{"time":"2023-10-30T08:29:45Z","hostname":"bullseye","level":"Info","module":"Startup","message":"Thor Version: 10.7.9","scanid":"S-hNuFlx14n74","log_version":1}
{"time":"2023-10-30T08:29:45Z","hostname":"bullseye","level":"Info","module":"Startup","message":"Thor Build: 9f8d57058e7e (2023-07-28 06:19:19)","scanid":"S-hNuFlx14n74","log_version":1}
{"time":"2023-10-30T08:29:45Z","hostname":"bullseye","level":"Info","module":"Startup","message":"Run on system: bullseye","scanid":"S-hNuFlx14n74","log_version":1}
{"time":"2023-10-30T08:29:45Z","hostname":"bullseye","level":"Info","module":"Startup","message":"Running as user: root","scanid":"S-hNuFlx14n74","log_version":1}
{"time":"2023-10-30T08:29:45Z","hostname":"bullseye","level":"Info","module":"Startup","message":"User has admin rights: yes","scanid":"S-hNuFlx14n74","log_version":1}
{"time":"2023-10-30T08:29:45Z","hostname":"bullseye","level":"Info","module":"Startup","message":"Working Directory: /tmp/tmp67751496","scanid":"S-hNuFlx14n74","log_version":1}
{"time":"2023-10-30T08:29:45Z","hostname":"bullseye","level":"Info","module":"Startup","message":"Thor Scan started","scanid":"S-hNuFlx14n74","start_time":"2023-10-30T08:29:45.057997498Z","log_version":1}
{"time":"2023-10-30T08:29:45Z","hostname":"bullseye","level":"Info","module":"Startup","message":"Effective argument list: [--module Users --json --rebase-dir /tmp/tmp67751496]","scanid":"S-hNuFlx14n74","log_version":1}
......

[DYarizadeh]

My testing is utilizing Thor-Lite/Thor-Lite artifact

This is an example of my output:

{"Line":"{\"time\":\"2023-11-09T02:01:42Z\",\"hostname\":\"REDACTED\",\"level\":\"Info\",\"module\":\"Startup\",\"message\":\"Running as user: NT AUTHORITY\\SYSTEM\",\"scanid\":\"S-SxUMm0ZXdLE\",\"log_version\":\"v1.0.0\"}","FlowId":"F.CL63QA6JFDFOC.H","ClientId":"C.b2171e590b9ebc36","Fqdn":"REDACTED"}

Looks like our outputs are a bit different.

[pH-T]

ah, could it be that you use the files from the "Results" tab instead of the "Uploaded Files" tab? (tab ~= Flow Details)

[DYarizadeh]

Yes, that seems to be the difference. I am testing auditing all my systems at once instead of one device at a time.

[DYarizadeh]

Do we know if there is a work around?

Thanks!

[pH-T]

why cant you use the uploaded files?

[DYarizadeh]

Because, and correct me if I'm wrong, if I use the data from the uploaded files tab, then I have to download and then parse the data 1 machine at a time.

When you download the data from the results tab, there's a folder called "results" that has the data compiled from all devices into one CSV/json file. It would be much more convenient to parse a CSV that contains all the data from all the devices, as opposed to downloading the data one device at a time.

[pH-T]

you can download all the results in one zip: hunt overview --> results --> "full download" --> each host has a folder with the uploaded files.

but i added the idea of a single output file to the roadmap.

EDIT: you can also use jq to parse the file you tried working with, e.g. something like: cat All\ Generic.Scanner.Thor%2FThorResultsJson.json | jq '.Line | fromjson' > nice.json