Office-docs causes VM run out of memory

[ClaudioWayne]

This is opensource and you getting free support so be friendly!

• Free support from doomedraven ended, no whiskey no support. For something he updated the documentation :)

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• [x] I am running the latest version
• [x] I checked the documentation and found no answer
• [x] I checked to make sure that this issue has not already been filed
• [x] I'm reporting the issue to the correct repository (for multi-repository projects)

Expected Behavior

Processing is successful

Current Behavior

Hello,

Hello, During the analysis of office-documents, the memory usage increases more and more until the vm runs out of memory. This causes an error during the processing task. I can see that a svchost.exe causes the memory usage that leads to winmgmt-Service. I tried to run the agent both via cmd with administrative privileges and in startup folder for all users C:\ProgramData\Microsoft\Windows\Start Menu\Programs\StartUp (shell:common startup). Firewall and Windows-Defender are off. If i stop the winmgmt-service manually during analysis, analysis and processing are successful. I tried also different vm-setups (different .net frameworks, WMF 5.1, different Python-Versions, with/without pillow, choco.bat etc.) but i can't find the error. Did also a complete fresh install of ubuntu, kvm/qemu and cape. No success. PE-files etc works flawless.

Host: Ubuntu 20.04.01 / Guest: Windows7x64 (6.1.7601 SP1 Build 7601) with Office 2010SP2

My workaround for now is to set the Timout < 100 secs

Maybe someone can point in a direction, where the problem could be.

Failure Information (for bugs)

I added analysis.log, cuckoo.log and process.log in failure logs section

Context

Please provide any relevant information about your setup. This is important in case the issue is not reproducible except for under certain conditions.

Question	Answer
Git commit	6472f3d28f090aafce6caed2c40a50bcfb7a7e91
OS version	Host: Ubuntu 20.04.01 / Guest: Windows7x64 (6.1.7601 SP1 Build 7601) with Office 2010SP2

Failure Logs

analysis.log Cuckoo.log ProcessingError.log

[doomedraven]

@kevoreilly can you check this? we tried few things in private with @ClaudioWayne but no sucess

[doomedraven]

can't be reproduced on our side :(, Kevin any idea how to debug this?

https://capesandbox.com/submit/status/114355/ https://capesandbox.com/submit/status/114354/ https://capesandbox.com/submit/status/114353/

[ClaudioWayne]

More investigation: Today i tried with admin-account (net user administrator /active:yes) -> still runs out of memory

I noticed that svchost.exe 4636 -k netsvcs in Behavioral Analysis is missing on my side (https://capesandbox.com/analysis/114354/)

what i also found inside VM:

[kevoreilly]

Hi folks, sorry not to reply sooner. I've tried to recreate this without success. But as you've worked out the WMI service (winmgmt) is hosted by the svchost (netsvcs) process and therefore this process is injected into as part of the Office detonation - this is atttempted in your case in the analysis.log:

2021-01-29 10:52:07,484 [lib.api.process] INFO: Monitor config for process 2316: C:\tmp8gi15g80\dll\2316.ini 2021-01-29 10:52:07,484 [lib.api.process] INFO: Option 'office' with value '1' sent to monitor 2021-01-29 10:52:07,484 [lib.api.process] INFO: 64-bit DLL to inject is C:\tmp8gi15g80\dll\jtFApDM.dll, loader C:\tmp8gi15g80\bin\ZeXPLqvy.exe 2021-01-29 10:52:07,500 [root] DEBUG: ReadConfig: Successfully loaded pipe name \.\PIPE\SZPTmlGsU. 2021-01-29 10:52:07,500 [root] DEBUG: Loader: Injecting process 2316 (thread 0) with C:\tmp8gi15g80\dll\jtFApDM.dll. 2021-01-29 10:52:07,500 [root] DEBUG: InjectDll: No thread ID supplied, GetProcessInitialThreadId failed, falling back to thread injection. 2021-01-29 10:52:07,500 [root] DEBUG: Python path set to 'C:\Users\Sandra\AppData\Local\Programs\Python\Python38-32'. 2021-01-29 10:52:07,500 [root] DEBUG: Dropped file limit defaulting to 100. 2021-01-29 10:52:07,500 [root] INFO: Disabling sleep skipping. 2021-01-29 10:52:07,500 [root] DEBUG: CAPE initialised: 64-bit monitor loaded in process 2316 at 0x000007FEF29D0000, thread 2480, image base 0x00000000FF290000, stack from 0x00000000012A6000-0x00000000012B0000 2021-01-29 10:52:07,500 [root] DEBUG: Commandline: C:\Windows\sysnative\svchost.exe -k netsvcs

This seems to be the last mention of the process in the log until it fails to communicate with it for the 'terminate event' at the end.

At the moment I'm not sure what might be causing this, but I would first suggest trying the old loader option on submission. Another loader option that is worth trying is no-iat=1. Please give each of these a go and let me know if they make any difference.

[ClaudioWayne]

Hi, here are the results of the suggested measures:

1) Oldloader-option -> no success (still linear increase of memory) analysis_oldloader.log cuckoo_oldloader.log processing_oldloader.log

2) no-iat=1 option (first try) First time I tried this, I saw different behavior here. There was sharply increase of memory-usage which then went down again. analysis_option_no-iat_success.log cuckoo_option_no-iat_success.log processing_option_no-iat_success.log

3) no-iat=1 option (second try) To check if i can reproduce this behavior with this option i tried a second analysis with this option and same setting. I did not restart any cape-service (rooter, cuckoo etc) or so, just resubmit the sample with same option and settings. But unfortunately the linear increase of memory appeared again. analysis_option_no-iat_fail..log cuckoo_option_no-iat_fail.log processing_option_no-iat_fail.log

I also noticed a different behavior today on capesandbox.com

The detection of the powershell execution is missing on x64 machines in behavioral analysis section (therefore, no entries in DNS section): https://capesandbox.com/analysis/116167/ https://capesandbox.com/analysis/116171/

In earlier x64 analysis powershell execution has been detetced: https://capesandbox.com/analysis/113907/

On x86 everything seems fine: https://capesandbox.com/analysis/116169/

So i am not sure if this issue is worth investigating further because i can not identify if this general CAPE problem, or is it just a problem on my side. So its up to you if want close this issue since it can not be reproduced on CAPE side :) But if you want more information or to try something else, i will do my best.

[ClaudioWayne]

Update:

Today i tested all samples with a x86 machines -> everything works perfectly It seems that the problem exists only for 64 bit machines (x64 tag is set). I have also tested Pafish.exe in both machines x64 -> linear increase of memory x86 -> no issues

[doomedraven]

x64 tag is only for PE32+ as we autoinject tag

did you install x64 or x86 office in x64 vm?

[ClaudioWayne]

x86 office in both machines, could that be the problem?

[doomedraven]

no, just needed more details about setup

[ClaudioWayne]

I have tried to identify the most significant differences between 32 and 64 bit analysis (differences_32vs64.txt).

Diffrences_32vs64.txt 32Bit_success_analysis.log 64Bit_fail_analysis.log

To sum it up(main differences):

1 WARNING: b'Unable to place hook on LockResource' WARNING: b'Unable to hook LockResource' -> can be reproduced https://capesandbox.com/analysis/116685/

2 DEBUG: InjectDllViaIAT: IAT patching with dll name C:\tmpktjo5va6\dll\kwQQfvU.dll. DEBUG: InjectDllViaIAT: Failed to allocate region in target process for new import table. -> can be reproduced https://capesandbox.com/analysis/116685/

3 Yara error: Internal error: 4 -> can be reproduced https://capesandbox.com/analysis/116685/

Maybe one if these error is critical. I have no clue :)

btw1: also tested in Virtualbox with ubuntu qemu-kvm package with other win7 64 bit iso -> no success

btw2: also noticed in https://capesandbox.com/analysis/116685/: (same https://github.com/kevoreilly/CAPEv2/issues/437) GoogleUpdate.exe 4048 /svc taskhost.exe 696 taskhost.exe $(Arg0) svchost.exe 2712 C:\Windows\System32\svchost.exe -k WerSvcGroup WerFault.exe 4156 C:\Windows\system32\WerFault.exe -u -p 696 -s 320

WerFault.exe seems to reports errors in Windows to microsoft. C:\Program Files (x86)\Google\Update\GoogleUpdate.exe don't know why its there

it seems to appear at a long analysis duration, in my case 500

I think this was my last attempt to get rid of this problem, as I have no more ideas how to fix or debug the problem. So feel free to close this issue since it can not be reproduced on CAPE side or no others came up with that problem.

[kevoreilly]

It's a tricky one for sure. But we can maybe rule out some more 'stuff' by disabling with the appropriate options. Here is a list of a load of them to try all together or individually to see if they make any difference:

minhook=1,yarascan=0,caller-dump=0,injection=0,api-rate-cap=0

[ClaudioWayne]

okay thanks, i will give it a try

[ClaudioWayne]

With option minhook=1 the memory issue does not apper. However the powershell execution is not recognized.

In anaysis.log with minhook=1 option

WARNING: b'Unable to place hook on LockResource' WARNING: b'Unable to hook LockResource'

and

Yara error: Internal error: 4

both errors no longer exist.

The rest of the option has no effect.

minhook1_analysis.log

[ClaudioWayne]

Update: I updated my local CAPE this week (commit d972e678934b1ce2d016306bc39493105c975f3a) Miraculously, there is no more increase in memory :+1:

But it is still the case, that the powershell execution is not detected by x64 machines in Behavioral Analysis. Only on x86. Same on Capesandbox.com x64 https://capesandbox.com/analysis/124527/ x86 https://capesandbox.com/analysis/125422/ (Before the update i had the memory issue but the powershell execution was detected by x64 machines)

So actually the issue is solved :) But there is still the problem with the powershell detection. Maybe its this particular sample but i have no other sample like this. It could be that the memory problem does not appear because the powershell command is not executed in x64.

[ClaudioWayne]

Hi,

It works properly now on my local CAPE on a x64 machine.

Tested on capesandbox.com today -> powershell execution is still not detected. https://capesandbox.com/analysis/138265/

So for me personally, the problem has been solved :)

[doomedraven]

Glad that works, i guess is related to exec timing as on public it has short limit of 100 seconds

El lun., 19 abr. 2021 9:05, ClaudioWayne @.***> escribió:

Hi,

It works properly now on my local CAPE on a x64 machine. [image: image] https://user-images.githubusercontent.com/35531629/115194086-7fa98f00-a0dc-11eb-8179-34f181612275.png [image: image] https://user-images.githubusercontent.com/35531629/115194158-97811300-a0dc-11eb-8826-9e3f3c5827d2.png [image: image] https://user-images.githubusercontent.com/35531629/115194263-c0090d00-a0dc-11eb-84a7-ff940e981370.png

Tested on capesandbox.com today -> powershell execution is still not detected. https://capesandbox.com/analysis/138265/

So for me personally, the problem has been solved :)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kevoreilly/CAPEv2/issues/422#issuecomment-822226140, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOFH325GRUDS4JL4TLUNRDTJPI4NANCNFSM4WYWSUFA .

[ClaudioWayne]

Hi doomedraven,

Do you mean analyses time? My default is 100 seconds timeout in cuckoo.conf

[timeouts]
default = 100

[doomedraven]

yes, but as you can see on screenshot it was executed for 139 seconds, maybe that is related

[ClaudioWayne]

got it, yes maybe. On https://capesandbox.com/analysis/138265/ it shows 769 seconds. But i don't now :) So should we close this issue?

[doomedraven]

lets leave it for decide to @kevoreilly

[kevoreilly]

Guys I much appreciate you keeping me in mind as ultimately we all know for the good of cape my logic needs to be:

Is cape working perfectly? No -> keep on workin, keep on fixin. Somehow x64 detonation is failing again despite my hook audit last time. I learned then that if you see dllhost.exe C:\Windows\system32\DllHost.exe /Processid:{F9717507-6651-4EDB-BFF7-AE615179BCCF} it's some sort of WMI failure/crash and it's basically bad news.

So perhaps this is really a separate issue but ultimately this needs swatting and I needed reminding as it's hard to keep track. I will reaffirm this issue's position on top of my to-do list...

[kevoreilly]

Weird first test for me and it worked:

[kevoreilly]

Hmm well it may actually be something that is inadvertently fixed in the monitor build I'm using which I will publish soon so hopefully this will fix it.

[ClaudioWayne]

Did a quick test series today (x64 machine)

1) no powershell execution 2) success 3) success 4) powershell has stopped working -> 3/5 DNS entries 5) No powershell execution 6) success 7) no powershell execution 8) no powershell execution 9) powershell has stopped working -> 0/5 DNS entries 10) powershell has stopped working -> 3/5 DNS entries

so t seems to be random in some way :) I am curious about the new monitor build. I appreciate your work.

[kevoreilly]

By all means test this build - I'm hoping to publish soon but in the meantime... capemon.zip

[kevoreilly]

Thanks for testing by the way

[doomedraven]

@ClaudioWayne can you test this again please with latest capemon? there was one fix with run out of memory

[ClaudioWayne]

Run out of memory problem is gone -> solved There are still some powershell crashes followed by a memory could not be read error and sometimes no entries in the DNS section

Tests on capesandbox.com https://capesandbox.com/analysis/169674/ ->(x64) | powershell crash = yes | memory could not be read = no | DNS entries = no https://capesandbox.com/analysis/169686/ ->(x64) | powershell crash = yes | memory could not be read = yes | DNS entries = yes https://capesandbox.com/analysis/169691/ ->(x86) | powershell crash = no | memory could not be read = no | DNS entries = no https://capesandbox.com/analysis/169697/ ->(x86) | powershell crash = yes | memory could not be read = no | DNS entries = no

Tests on local CAPE

1. 208 (x86) | powershell crash = no | memory could not be read = no | DNS entries = yes -> perfect
2. 209 (x64) | powershell crash = no | memory could not be read = no | DNS entries = no
3. 210 (x64) | powershell crash = no | memory could not be read = no | DNS entries = yes -> perfect
4. 211 (x86) | powershell crash = no | memory could not be read = no | DNS entries = yes -> perfect

Screenshot of local analysis 4 (211)

git commit: 6b46c727

[doomedraven]

about that reference memory im suffering that a lot too and without powershell, thanks for testing

[kevoreilly]

I can't recreate these at the moment.

[doomedraven]

i think that is should be fixed now, closing it, thanks for reporting and testing this