[Known Issue] The perf of emptyDir on Windows nodes is bad

[AbelHu]

Describe the bug emptyDir on Windows nodes have bad perf. Please see more details and the status in https://github.com/microsoft/Windows-Containers/issues/345

Additional context Use this issue to track the bug fix.

[ghost]

@immuzz, @justindavies would you be able to assist?

Issue Details

---

**Describe the bug** emptyDir on Windows nodes have bad perf. Please see more details and the status in https://github.com/microsoft/Windows-Containers/issues/345 **Additional context** Use this issue to track the bug fix.

Author:	AbelHu
Assignees:	AbelHu
Labels:	`known-issue`, `windows`
Milestone:	-

[AbelHu]

Copied the update from Howard-Haiyang-Hao from https://github.com/microsoft/Windows-Containers/issues/345#issuecomment-1611838074

The fix has successfully passed most release rings and will be available to the public within 2-3 weeks through the Windows Update channel. Thank you for your patience!

[cailyoung]

@AbelHu could you leave this issue open until the Windows vhd is updated with the fix, please?

[AbelHu]

@AbelHu could you leave this issue open until the Windows vhd is updated with the fix, please?

@cailyoung The update is rolled out by Windows defender team and the fix will be fetched from the Microsoft Windows update service automatically.

[matt-richardson]

@AbelHu - does this have a kb number or something similar? How can we determine whether the fix is present on our nodes?

[AbelHu]

@AbelHu - does this have a kb number or something similar? How can we determine whether the fix is present on our nodes?

We asked the same question but get no answer. The latest update on 7/13/2023 is this should be 25% of the broad ring by today. As you can see from the schedule above, it will reach 100% of the population in about 6 days from now. We will spend 3 days at 25% then 3 more days at 50% then, we'll go to 100%

[AbelHu]

Update from Windows defender team:

The fix is expected to be released with our July payload. ETA for the release is first half of August.

To verify that you’re running a version that includes this fix you can run the following from a Powershell prompt: 

Get-MpComputerStatus | select -Property AMProductVersion

If the returned version is something of the form 4.18.23070.xxxx, then you’re running a version of Defender that includes the fix.

[AbelHu]

Update from Windows defender team

The build containing this fix is expected to be released broadly by 8/8.

[cailyoung]

@AbelHu there hasn't been a Windows 2019 OS image update since 16 July; should we expect an update soon?

[AbelHu]

@cailyoung AKS has published new AKS Windows images 17763.4737.230808 with 2023.08B. You do not need to do anything to get the fix in defender. You can reference https://github.com/microsoft/Windows-Containers/issues/345#issuecomment-1677971139 to check the defender version.

[cailyoung]

@AbelHu that's great news, however https://github.com/Azure/AKS/releases shows that the image hasn't yet shipped out in AKS.

[AbelHu]

@AbelHu that's great news, however https://github.com/Azure/AKS/releases shows that the image hasn't yet shipped out in AKS.

Thanks for reporting it, @cailyoung. I think that the new AKS Windows image versions are missed in Release Release 2023-08-13 · Azure/AKS (github.com). @CocoWang-wql will help to fix it.

[CocoWang-wql]

Thanks for the feedback. Addressed the release notes: https://github.com/Azure/AKS/releases/tag/2023-08-13

[cailyoung]

@AbelHu We have updated to 230808 and the Defender platform fix is not there:

Get-MpComputerStatus                                                                                                         

AMEngineVersion                 : 1.1.23060.1005                                                                                                    
AMProductVersion                : 4.18.1807.18075                                                                                                   
AMServiceEnabled                : True                                                                                                              
AMServiceVersion                : 4.18.1807.18075                                                                                                   
AntispywareEnabled              : True                                                                                                              
AntispywareSignatureAge         : 15                                                                                                                
AntispywareSignatureLastUpdated : 8/8/2023 2:33:05 PM                                                                                               
AntispywareSignatureVersion     : 1.393.2613.0                                                                                                      
AntivirusEnabled                : True                                                                                                              
AntivirusSignatureAge           : 15                                                                                                                
AntivirusSignatureLastUpdated   : 8/8/2023 2:33:05 PM                                                                                               
AntivirusSignatureVersion       : 1.393.2613.0                                                                                                      
BehaviorMonitorEnabled          : True                                                                                                              
ComputerID                      : 870A04E4-077F-4013-AD62-35C45F0BA850                                                                              
ComputerState                   : 0                                                                                                                 
FullScanAge                     : 4294967295                                                                                                        
FullScanEndTime                 :                                                                                                                   
FullScanStartTime               :                                                                                                                   
IoavProtectionEnabled           : True                                                                                                              
LastFullScanSource              : 0                                                                                                                 
LastQuickScanSource             : 0                                                                                                                 
NISEnabled                      : True                                                                                                              
NISEngineVersion                : 1.1.23060.1005                                                                                                    
NISSignatureAge                 : 15                                                                                                                
NISSignatureLastUpdated         : 8/8/2023 2:33:05 PM                                                                                               
NISSignatureVersion             : 1.393.2613.0                                                                                                      
OnAccessProtectionEnabled       : True                                                                                                              
QuickScanAge                    : 4294967295                                                                                                        
QuickScanEndTime                :                                                                                                                   
QuickScanStartTime              :                                                                                                                   
RealTimeProtectionEnabled       : True                                                                                                              
RealTimeScanDirection           : 0                                                                                                                 
PSComputerName                  :  

[AbelHu]

AMEngineVersion : 1.1.23060.1005
AMProductVersion : 4.18.1807.18075
AMServiceEnabled : True
AMServiceVersion : 4.18.1807.18075
AntispywareEnabled : True
AntispywareSignatureAge : 15
AntispywareSignatureLastUpdated : 8/8/2023 2:33:05 PM

The rollout is controlled by the defender team and we were told that the rollout is 100%. We have reported this to the defender team and will update you later.

[cailyoung]

@AbelHu any update? We opened support ticket 2308240030000090 in case you're able to use that.

[AbelHu]

@cailyoung The fix in defender should have a big perf improvement. But it seems like that there is still some perf issues and Windows team is still investigating it. Please use the support ticket to follow up it since you have one.