Artifacts collection stalling for long hours time

[moksharthab]

Hi Team, We are doing a great deal of testing with Velociraptor on RHEL 8 Linux server.

Whenever our Threat hunting team does some hunts for Yara files, those were getting hanged (or) without any progress.

Our Forensics team, when they run artifacts like - Windows.KapeFiles.Targets etc, it runs for a while and it stalls completely.

When we go to Log, below is the last message we get.

vql: Creating 30 workers for foreach plugin

We also see the Timeout is 600 seconds (10 minutes), but the job is still showing as running 4 hours later. Can you help us get around this problem on what is happening ?

I have attached below artifact files:

Server.Monitor.Profile (and also Collection Report) Generic.Client.Profile

Regards,

Mokshartha

[moksharthab]

Collection report

[moksharthab]

H.CE322QG6C6994.zip

[moksharthab]

server-F.CE32NTSQ93UL6.zip

[scudette]

Can you please also include the output from velociraptor version?

Also can you please include the exact collection you are running including parameters for the yara scan?

[moksharthab]

[moksharthab]

[moksharthab]

[moksharthab]

As requested, screenshots are added.

[scudette]

Thanks for including the collection - I am not sure what it is trying to do though?

It seems to be searching for yara signatures in the C:\ directory - normally the only files there are the pagefile which can be very large so it can take a very long time to scan. But why would you want to scan the page file?

I would recommend using the most recent Velociraptor version as well (0.6.7-1). If you still want to use 0.6.5 the latest release in 0.6.5 branch was 0.6.5-2

[moksharthab]

But we ran the same hunt on Windows server last time (in June) and it was working fine.

Its the exact same setting.

[scudette]

looking at the profile you sent I can see the following line:

{"Type":"query","File":null,"Line":{"Query":"SELECT * FROM if(then=Windows_Search_FileFinder_0_4, condition=precondition_Windows_Search_FileFinder_0, else= { SELECT * FROM scope() WHERE log(message='Query skipped due to precondition') AND FALSE})","Start":"2022-11-28T12:16:55.9658432Z","Duration":99584092700}}

Which indicates the collection completed in 99 seconds - did you have a timeout specified in the resources section?

[moksharthab]

I left it as defaults. We always do.

[moksharthab]

Were you able to find the reason/root cause ?

Please let us know if you need more detail/logs.

[moksharthab]

Any progress on this issue ?

[scudette]

I would recommend you try with the latest build 0.6.7.

[moksharthab]

Server is upgraded to 6.7 now.

We ran several forensics artifacts:

System.shares Forensics.sam Forensics.timeline Prefetch and several others

All of which worked perfectly fine.

[moksharthab]

But whenever we run Kapefiles.targets

It runs for a while and then hungs up.

Ran client profile artifact. Please have a look.

[moksharthab]

C.8146f8b5e16e1578-F.CE7FV1N19NFU8.zip

[moksharthab]

logs (1).csv

[moksharthab]

Also attached the log file of the hunt

[moksharthab]

And please let us know if you need any other debug files for finding the root cause

For Kape hunt, we increased the time out to 1800. FYI

[scudette]

Thanks for trying the latest version.

The profile you attached is from a Velociraptor process which has only been up for a few minutes:

{"Type":"metrics","File":null,"Line":{"name":"uptime","help":"Time since process start.","value":215}}

The collection log you show is from about a week earlier. Is it possible to re-collect the artifact and schedule a profile at the same time? It might be that the client is restarting for some reason half way through

Do you get this problem on every client you try or only certain ones? Can you run the client in interactive mode (using velociraptor --config client.config.yaml client -v) and see what happens to it if it restarting?

[scudette]

In 0.6.8 there are some more improvements that might have fixed this issue or make it easier to debug.

You can now set a trace in a collection to collect the profile automatically as it goes along.

Additionally the file collector should work better now since it was materializing the file list into memory and that was cause memory issues in some cases. In 0.6.8 this is no longer a problem.

I'm closing this one but feel free to reopen if this still does not work in 0.6.8