Does sysmon64 lock/remove files?

[lesscodedotnet]

We've been having a very strange issue with a particular Azure Devops pipeline over the last few months. We have three build agents, and the pipeline would consistently fail on two of them, and consistently work fine on the third. The failures are always the same - one specific project would fail to compile (always the same project) because it could not write the assembly object file (CSC2012: Access denied).

Analysis using Procmon showed that on the problem agents, immediately after the file is written by the compiler, subsequent access to the path would show PENDING DELETE.

We immediately suspected some heuristic rule in AV software (we run Symantec and Cylance) was quarantining the file, but after completely uninstalling both we were still seeing the error.

Then we noticed that sysmon64 was only running on the two problem agents and not on the third. After removing sysmon from one of the problem agents, the build started working fine on that agent.

Is there a way to mitigate this or do we simply have to turn off sysmon on these agents?

[darkoperator]

Having so many agents is a bad mix since all will install minifilter drivers attach to the drives. What eventtypes are set in your sysmon config?

Sent from my iPhone

On Dec 1, 2021, at 9:53 PM, lesscodedotnet @.***> wrote:

 We've been having a very strange issue with a particular Azure Devops pipeline over the last few months. We have three build agents, and the pipeline would consistently fail on two of them, and consistently work fine on the third. The failures are always the same - one project would fail to compile because it could not write the assembly object file (CSC2012: Access denied).

Analysis using Procmon showed that on the problem agents, immediately after the file is written by the compiler, subsequent access to the path would show PENDING DELETE.

We immediately suspected some heuristic rule in AV software (we run Symantec and Cylance) was quarantining the file, but after completely uninstalling both we were still seeing the error.

Then we noticed that sysmon64 was only running on the two problem agents and not on the third. After removing sysmon from one of the problem agents, the build started working fine.

Is there a way to mitigate this or do we simply have to turn off sysmon on these agents?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[lesscodedotnet]

@darkoperator By agents, you mean Symantec, Cylance and sysmon64? We had turned off Symantec and Cylance and the problem was still occurring until we also turned off sysmon, so presumably sysmon was the sole agent at that point.

I don't know how sysmon was configured - that's owned by a separate security team, but I can find out.

[darkoperator]

I mention it because when a minifilter is set for drives only the product with the lowest altitude number will trigger, report and/or block, you can not correlate between them. In the case of Sysmon I suspect there is a file delete rule that is causing the block, that is why I asked about the config.

[lesscodedotnet]

@darkoperator If I'm understanding the docs correctly, a file delete rule would watch for files being deleted and just log an event and/or copy the file, right? I'm looking to understand why a file is apparently being prevented from being created (or accessed after creation) by sysmon. It was my understanding that sysmon is only about monitoring and not active prevention.

I have meeting with the security team this afternoon - I'll try to get the full config file they're using.

[lesscodedotnet]

I got their config file and it has a ton of rules, but none of them are FileDelete or FileDeleteDetected rules. We ran some more experiments and we can consistently see that the build works fine when the sysmon64 service is stopped, and always fails to write one specific object dll file when the sysmon64 service is running.

[darkoperator]

Strange, only thing that comes to mind is the file is being hashed because it was logged by one of the rules. Have you used tools like resmon or procmon to confirm? Do any rules would match the build?

Sent from my iPhone

On Jan 11, 2022, at 9:17 PM, lesscodedotnet @.***> wrote:

 I got their config file and it has a ton of rules, but none of them are FileDelete or FileDeleteDetected rules. We ran some more experiments and we can consistently see that the build works fine when the sysmon64 service is stopped, and always fails to write one specific object dll file when the sysmon64 service is running.

— Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you were mentioned.

[lesscodedotnet]

By trial and error we traced the issue to the Event ID 7 (ImageLoad) rules. If we commented out all includes and excludes in this section the issue disappeared.

We then added an ImageLoad exclude rule for devenv.exe and the problem also went away (with all of the other rules restored).

[darkoperator]

Interesting! Wonder why. Thanks for letting me know. Did you post the issue to the sysinternals forum for the devs to know?

Sent from my iPhone

On Feb 10, 2022, at 7:23 PM, lesscodedotnet @.***> wrote:

 By trial and error we traced the issue to the Event ID 7 (ImageLoad) rules. If we commented out all includes and excludes in this section the issue disappeared.

We then added an ImageLoad exclude rule for devenv.exe and the problem also went away (with all of the other rules restored).

— Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you were mentioned.

[lesscodedotnet]

@darkoperator

I did post on Microsoft Q&A under the [windows-sysinternals-sysmon] tag: (https://docs.microsoft.com/answers/topics/164854/windows-sysinternals-sysmon.html)

Is there a more suitable spot?