Sysmon hangs when deleting large files on USB-devices

[SysmonMike]

When deleting large files located on USB-devices Sysmon "hangs" and the file is never deleted until Sysmon is killed.

I'm using the latest version of Sysmon (v14.12) and the problem seems to start with files larger than about 1.1 GB. Deleting files of sizes 1.03 GB, 969 MB, and 619 MB all works but is a bit slower than usual.

Sizes like 1.1 GB and 1.4 GB doesn't work and the error messages shown below are displayed:

If I select Try Again the same error message is shown again, and again, ...

I guess this is a bug!?

[foxmsft]

What filesystem is there on that drive?

[SysmonMike]

The USB is a FAT32 drive.

[SysmonMike]

I did another test on a exFAT-device and the result is the same. The file isn't be deleted due to Sysmon.

[marcuswestlund]

I can add that this issue has been there since at least since Sysmon 12.03 (x64).

[foxmsft]

What are the steps to reproduce this? Deletes from slower drives do take longer (due to the data being read for checksums before), but eventually they do complete.

Does this "The action can't be completed" dialog pop up as soon as shift-del is issued? Are there other Sysmon events besides FileDelete involved?

[SysmonMike]

@foxmsft I don't think it is a question of slow drives or me being impatient. When the "action can't be completed" appears I am unable to delete the file(s). I can retry about 10 times and the file is still not deleted. I've tried both small USB-memory sticks and larger USB 3 hard drives and neither of them are "slow".

Deleting files with SHIFT+DEL is the most troublesome. For larger files the error message always seem to appear. Deleting with DEL only (i.e. "via" the Recycle bin) works much better but usually results in an error message for the last file deleted from the Recycle bin. Note that the file size here is reported as 0 bytes!

I tried running Sysmon with an empty config file (logging no events what so ever) I was able to get the zero size error message above on one single occasion when emptying the Recycle bin!?

The event in question isn't FileDelete (EID 23) but rather FileDeleteDetected (EID 26). The rule that matches my large test files in the one:

<TargetFilename name="FileDeleteDetected - Container file deleted" condition="end with">.iso</TargetFilename>

I hope this gives you enough information to investigate further.

[SysmonMike]

Sorry, I forgot. The delay between SHIFT+DEL and the error message is about 8-10 seconds.

[foxmsft]

This is exactly what I needed to know, thank you! I'll get back here with an update.

[foxmsft]

@SysmonMike do you have other relevant bits of the config to share? I can't get this to repro with either SSD or spindle exFAT USB storage.

I'm interested in the HashAlgorithms directive or anything else that might be relevant here. Thanks!

[SysmonMike]

@foxmsft, We have set the hashes to all hash types in the config file like this: <HashAlgorithms>*</HashAlgorithms>

I don't think it matters but we have also changed the archive directory: <ArchiveDirectory>Sysmon_Deleted_Files</ArchiveDirectory>

And finally, I doubt that it matters but we log all file creations (EID 11) on USB-devices with rules like these: `

D:\ E:\`

[SysmonMike]

@foxmsft, I seems that the problem is indeed related to the HashAlgo configuration.

I changed the configuration to; <HashAlgorithms>SHA1,IMPHASH</HashAlgorithms> and now it works better (but unfortunately not perfect). I still get the error message, but if I press retry I'm able to delete the file(s) in 1 or 2 retries.

[foxmsft]

Does this still happen for you with just FileDeleteDetected? Did you find the minimal config where this behavior can be observed?

<Sysmon schemaversion="4.83">
  <HashAlgorithms>*</HashAlgorithms>
  <ArchiveDirectory>Sysmon_Deleted_Files</ArchiveDirectory>

  <EventFiltering>
    <ProcessCreate onmatch="include" />
    <ProcessTerminate onmatch="include" />

    <FileDeleteDetected onmatch="include">
      <TargetFilename name="FileDeleteDetected - Container file deleted" condition="end with">.iso</TargetFilename>
    </FileDeleteDetected>
  </EventFiltering>
</Sysmon>

[SysmonMike]

@foxmsft I tried the config above at it works fine. No errors are displayed when deleting large files.

[foxmsft]

Now it's a matter of figuring out which 2 or more rules interact to create this situation. Can you share the full config?

[SysmonMike]

Unfortunately I can't do that. Company policy... I can share pieces but not the entire file.

And figuring out the combination of rules might be tricky since the config file is rather large (almost 7 000 lines).

Any suggestions what to try next or where to start looking for "culprit" rules?

[foxmsft]

No worries - you can send it via email if allowed. Otherwise keep slicing the config - for example "halving" it every time. FileDelete* rules are to be kept, otherwise everything network or process related is probably safe for removal.

[SysmonMike]

@foxmsft I've managed to identify a minimal configuration file that causes the problematic situation:

<Sysmon schemaversion="4.83">
    <HashAlgorithms>*</HashAlgorithms> 
    <EventFiltering> 
        <!-- Event ID 1: Process creation -->
        <ProcessCreate onmatch="include" />

        <!-- Event ID 5: Process terminated -->
        <ProcessTerminate onmatch="include" />

        <!-- Event ID 23: FileDelete (A file delete was detected) -->
        <RuleGroup name="FileDelete - Exclude" groupRelation="or">
            <FileDelete onmatch="exclude">
                <!-- MD5 of __PSScriptPolicyTest_RANDOM.ps1 file -->
                <Hashes condition="contains">D17FE0A3F47BE24A6453E9EF58C94641</Hashes>
            </FileDelete>
        </RuleGroup>

        <!-- Event ID 26: FileDeleteDetected (File Delete logged)  -->
        <RuleGroup name="FileDeleteDetected - Include" groupRelation="or">
            <FileDeleteDetected onmatch="include">
                <TargetFilename name="FileDeleteDetected - Container file" condition="end with">.iso</TargetFilename>
            </FileDeleteDetected>
        </RuleGroup>

    </EventFiltering>
</Sysmon>

[foxmsft]

Thanks for all the support, fix is incoming :)

[marcuswestlund]

Thanks for all the support, fix is incoming :)

Great work @foxmsft and @SysmonMike !

[foxmsft]

Latest version, v14.14 should fix this behavior. There are some performance improvements around the file events, upcoming in the near future.