WindowsPersistenceMechanismFiles - Hunt Error

[Se1ecto]

Greetings, I believe I found an issue or a bug concerning a Google Rapid Response hunt and the way some clients are working. I wish to run the “Collectors>ArtifactCollectorFlow>WindowsPersistenceMechanismFiles” hunt. As of now, I am only trying to run it locally and limiting the hunt to around two hours, 10 clients, and only my computer running the agent. However this task, while it appears as completed, offers a load of errors concerning some registry lookup. I believe it is either related to the hunt not finding the files it is looking for, or to a lack of permissions on the client. The hunt provides with error logs on the Admin UI in this particular hunt. I am sending you a sample of the output. The entire log consists of that same error pattern:

2018-02-15 20:49:01 UTC     Artifact WindowsAppInitDLLs data collection failed. Status: message GrrStatus { cpu_time_used : message CpuSeconds { system_cpu_time : 0.0 user_cpu_time : 0.0 } error_message : u”File not found: message PathSpec {\n path : u’HKEY_USERS\\\\S-1-5-21-812718187-1142311462-14044502-9073\\\\Software\\\\Wow6432Node\\\\Microsoft\\\\Windows NT\\\\CurrentVersion\\\\Windows\\\\AppInit_DLLs’\n pathtype : REGISTRY\n}” network_bytes_sent : 358 status : IOERROR }.
2018-02-15 20:49:01 UTC     Artifact WindowsAppInitDLLs data collection failed. Status: message GrrStatus { cpu_time_used : message CpuSeconds { system_cpu_time : 0.0 user_cpu_time : 0.0 } error_message : u”File not found: message PathSpec {\n path : u’HKEY_USERS\\\\S-1-5-21-812718187-1142311462-14044502-10284\\\\Software\\\\Wow6432Node\\\\Microsoft\\\\Windows NT\\\\CurrentVersion\\\\Windows\\\\AppInit_DLLs’\n pathtype : REGISTRY\n}” network_bytes_sent : 359 status : IOERROR }.
2018-02-15 20:49:01 UTC     Artifact WindowsAppInitDLLs data collection failed. Status: message GrrStatus { cpu_time_used : message CpuSeconds { system_cpu_time : 0.0 user_cpu_time : 0.0 } error_message : u”File not found: message PathSpec {\n path : u’HKEY_USERS\\\\S-1-5-21-1094346107-1713775765-2398829250-500\\\\Software\\\\Wow6432Node\\\\Microsoft\\\\Windows NT\\\\CurrentVersion\\\\Windows\\\\AppInit_DLLs’\n pathtype : REGISTRY\n}” network_bytes_sent : 360 status : IOERROR }.
2018-02-15 20:49:01 UTC     Artifact WindowsAppInitDLLs data collection failed. Status: message GrrStatus { cpu_time_used : message CpuSeconds { system_cpu_time : 0.0 user_cpu_time : 0.0 } error_message : u”File not found: message PathSpec {\n path : u’HKEY_USERS\\\\S-1-5-21-812718187-1142311462-14044502-9073\\\\Software\\\\Microsoft\\\\Windows NT\\\\CurrentVersion\\\\Windows\\\\AppInit_DLLs’\n pathtype : REGISTRY\n}” network_bytes_sent : 345 status : IOERROR }.
2018-02-15 20:49:01 UTC     Artifact WindowsAppInitDLLs data collection failed. Status: message GrrStatus { cpu_time_used : message CpuSeconds { system_cpu_time : 0.0 user_cpu_time : 0.0 } error_message : u”File not found: message PathSpec {\n path : u’HKEY_USERS\\\\S-1-5-21-812718187-1142311462-14044502-500\\\\Software\\\\Wow6432Node\\\\Microsoft\\\\Windows NT\\\\CurrentVersion\\\\Windows\\\\AppInit_DLLs’\n pathtype : REGISTRY\n}” network_bytes_sent : 357 status : IOERROR }.

In order to resolve this issue, I was looking into Windows’ Process Monitor. It encountered many anomalies when it comes to the client accessing various registries. I am also providing you with a screenshot of an example of errors coming up (Excluding success log entries):

https://imgur.com/a/2g7Aj

It would be a pleasure to provide any information if necessary. Thank you very much for your time.

[grrrrrrrrr]

Hey,

These 'errors' are kind of expected, they just indicate that the registry key could not be opened. Do any of the values that throw the error actually exist on the machine? If not, this is working as expected...

Cheers, -Andy

[Se1ecto]

Hello Andy, Thank you for answering! As I am not capable right now of verifying if those values exist, I would like to know what might be the causes of the errors if they were to actually exist?

Me and my colleagues find odd to have a persistence hunt which provides no results whatsoever.

Thank you very much.

[Karneades]

I would like to know what might be the causes of the errors if they were to actually exist?

The error would not shown up IF the registry keys exists :)

I tested the flow and got the files from the persistence mechanisms for a test machine.

Set some own values on a test client and verify, if the file is found, e.g. set cmd.exe in the run key.

[Se1ecto]

I will certainly do that! Thank you!

I just dug up a Persistence hunt I ran locally yesterday as well. As part of the hunt log, it is looking for the following value:

2018-02-22 15:18:25 UTC
ArtifactCollectorFlow
Artifact WindowsAppInitDLLs data collection failed. Status: message GrrStatus { cpu_time_used : message CpuSeconds { system_cpu_time : 0.0 user_cpu_time : 0.0 } error_message : u"File not found: message PathSpec {\n path : u'HKEY_USERS\\\\S-1-5-80-1044544286-2763731348-267423293-2293503259-2593316332\\\\Software\\\\Wow6432Node\\\\Microsoft\\\\Windows NT\\\\CurrentVersion\\\\Windows\\\\AppInit_DLLs'\n pathtype : REGISTRY\n}" network_bytes_sent : 376 status : IOERROR }.

However the "S-1-5-80-1044544286-2763731348-267423293-2293503259-2593316332" directory doesn't even exist in the registries (only showing a portion of the log). However, it is also looking for valid directories, but the sub directories cannot be found either. It stops at "AUTORUN", where the reg value cannot be found:

Artifact WindowsCommandProcessorAutoRun data collection failed. Status: message GrrStatus { cpu_time_used : message CpuSeconds { system_cpu_time : 0.0 user_cpu_time : 0.015625 } error_message : u"File not found: message PathSpec {\n path : u'HKEY_USERS\\\\S-1-5-21-823518204-362288127-1606980848-518115\\\\Software\\\\Microsoft\\\\Command Processor\\\\AutoRun'\n pathtype : REGISTRY\n}" network_bytes_sent : 324 status : IOERROR }.

Is it normal for the hunt to look for directories which do not even exist? Is this the reason why no results are provided in the first place? Will attempt to run a hunt for a particular persistence in the meantime.

Thank you everyone! :)

[Karneades]

Is it normal for the hunt to look for directories which do not even exist?

The GRR server does not know in advance which directories/registry keys are available on the client, so therefore every defined registry key in the artifact (in the current case WindowsCommandProcessorAutoRun is searched.

You do not find the key "HKEY_USERS\\S-1-5-80-1044544286-2763731348-267423293-2293503259-2593316332" in the registry? THAT would be strange :) GRR enumerates all the SIDs (user ids) and reads the registry keys/values then.

Why the hunt stops at "AUTORUN" is not clear to me - according to the error log, only the artifact "AUTORUN" was stopped but that doesn't mean the whole hunt is stopped. Maybe someone else can jump in :)

[Se1ecto]

The hunt doesn't stop at the "autorun" search, just to be clear! It generally has trouble finding files and reg values.

Another example would be:

Artifact WindowsAppInitDLLs data collection failed. Status: message GrrStatus { cpu_time_used : message CpuSeconds { system_cpu_time : 0.0 user_cpu_time : 0.0 } error_message : u"File not found: message PathSpec {\n path : u'HKEY_USERS\\\\S-1-5-21-823518204-362288127-1606980848-518115\\\\Software\\\\Wow6432Node\\\\Microsoft\\\\Windows NT\\\\CurrentVersion\\\\Windows\\\\AppInit_DLLs'\n pathtype : REGISTRY\n}" network_bytes_sent : 360 status : IOERROR }.

In this case, the "Windows NT" directory cannot be found.

To confirm, "HKEY_USERS\S-1-5-80-1044544286-2763731348-267423293-2293503259-2593316332" is nowhere to be found in regedit. It is strange to find no persistence mechanism files whatsoever.

[Se1ecto]

I have some updates on this issue:

Actually, we believe clients remain on outstanding: The hunt launches, produces some logs, appears to crash mid-hunt which leaves it on "Outstanding". It only ends due to the expiry time.

you can see the entire log of our latest hunt here. While some steps of the hunt are entirely working, others seem to have trouble hashing or finding files. In the end, the hunt remains on "Outstanding" due to, we believe, an agent crashing.

The artifact list is up to date and no errors are in the errors section. It would be a pelasure to provide additional information in order to fix this issue.

Thanks!

[Se1ecto]

Hello everyone! Me and my teammates believed to have found a simple fix to our issue: Enabling "IGNORE_INTERPOLATION".

We would like to know, in general, what this option does. I've had trouble finding the relevant documentation to this.

Thank you for your time!