cisagov / ESXiArgs-Recover

A tool to recover from ESXiArgs ransomware
Creative Commons Zero v1.0 Universal
294 stars 41 forks source link

Successful decrypt but #4

Open okamah opened 1 year ago

okamah commented 1 year ago

I have Success decrypt vm file, and re-register vm, but when trying power on VM, errot operating system not found. How to fix that?

okamah commented 1 year ago

vmware tool not installed after decrypted

cablej commented 1 year ago

Hi @okamah, are you running a windows VM? If so, you may consider trying the steps at https://enes.dev/#update-1.

yortx85 commented 1 year ago

Same for me, 1 linux vm, 1 windows. No operating system in both cases.

cablej commented 1 year ago

Hi @yortx85, were you able to have any luck with the guidance at https://enes.dev/#update-1 for the Windows VM?

For the linux VM, could you share more info regarding your configuration and the error you're getting?

cablej commented 1 year ago

Hi @okamah and @yortx85, I have made some changes to the script that may fix some bugs if you would like to try running again. Additionally, please ensure that the vmx file in the encrypted_files folder is not included on the "Select VMs for registration" when registering a VM. I have made an update to change the extension of these files so that should no longer be a problem.

Let me know if you're still experiencing errors.

yortx85 commented 1 year ago

Thanks @cablej, I've run it again, but guess some prep has to be done previously. I've run it in a vm's folder which I tried earlier, but it fails to find some file, because they've already been moved to encrypted_files folder. Should I move all original files back to its first path? Anyway, I've run it in another machine's folder, but, again, getting error 'no OS found'. Thank you for your help!!!

wzxjohn commented 1 year ago

I think they maybe improve there script to encrypt the head of the -flat.vmdk and -delta.vmdk. Here's what I found:

  1. Newly created 10G Disk is 10737418240 which is exactly 10 * 1024*1024*1024. But the 'encrypted' 30G Disk size become 32212255232 which is 512 bytes larger then 30G
  2. The xxx-flat.vmdk.args record 32212254720 whis is exactly 30 * 1024*1024*1024
  3. The snapshot file xxx-00000x-delta.vmdk is vmfs sparse format which must start with COWD (which means copy-on-write disk), but the 'encrypted' snapshot does not start with this magic.
  4. The 'encrypted' snapshot file is smaller than the size record in xxx-00000x-delta.vmdk.args. In my case, one 24G disk small 982528, one 27G disk small 941568.
  5. According to this doc, the COWD file has a 1024 byte file header which is clearly missing in my 'encrypted' snapshot file.

According to this doc from OVH, the args file stores arguments passed to the encrypt binary (number of MB to skip, number of MB in encryption block, file size). So the third column of args file should be the original file size.

Attack script is here: https://pastebin.com/y6wS2BXh

Update:

After reading script and du's doc, I think the size delta in snapshot is caused by the attacker's misuse of the du command. Because du is counting block, not the real file size, so when counting sparse file which size is not exactly an integer multiple of 1024, the size delta will appear.

merimo commented 1 year ago

Not working for snapshots. Is there any way to decrypt snapshots?

cablej commented 1 year ago

@merimo is the flat file encrypted as well? If it is, this script cannot be used. If not, let me know what you’re seeing.

digim-s15 commented 1 year ago

@merimo is the flat file encrypted as well? If it is, this script cannot be used. If not, let me know what you’re seeing. same for me, flat-vmdk seems has encrypted header now, but remains is not encrypted, i see plain text there, etc. seems they updated encoder and vmdk not bootable now. but can it be restored somehow? how can we know?

merimo commented 1 year ago

@merimo is the flat file encrypted as well? If it is, this script cannot be used. If not, let me know what you’re seeing. Flat file is not encrypted. Only the snapshots... Picture1

VRocker commented 1 year ago

I've found that for some -flat files, the encrypter scrubs the partition table of the disk. Easiest way to fix this is to attach the vmdk to another linux VM and run it through 'testdisk'. This should be able to rebuild the partitions for you. The tricky part comes when you have an EFI bootable system, these might not boot once the partitions have been restored. For linux systems, boot an installation and drop into recovery. A Debian recovery can chroot you directly into the existing environment so all you need to do is re-run grub-install. For Windows systems, you'd need to reinstate the bootloaded using BCD or bootrec, which is well documented online.

Even with this really useful script to restore access to the VMs, they're won't be perfect and will need a bit of fettling to get back to a good shape.

okamah commented 1 year ago

Hi friend... my case is windows vm.. All we did still same issue, when files was decrypted successfull, but operating sistem is not found.. @VRocker could you explain more about "you'd need to reinstate the bootloaded using BCD or bootrec"?

thx

Teliz commented 1 year ago

Hi, I used script and worked right I found my VM but just first disk and before snapshots

All my data was ok last snapshot and second disk too

Someone was able to solve this problem? I must modify recover.sh script and change name of disk? or there are other way?

Tnx

thaeus-vienna commented 1 year ago

Hi, i have an infected system with serveral linux and windows hosts. It seems all files where encryted.

16ntexpf1-2ea5c413.hlog 16ntexpf1.vmsd.args 16ntexpf1_1-flat.vmdk.args 16ntexpf1-2ea5c413.vswp.args 16ntexpf1.vmx 16ntexpf1_1.vmdk 16ntexpf1-flat.vmdk 16ntexpf1.vmx.args 16ntexpf1_1.vmdk.args 16ntexpf1-flat.vmdk.args 16ntexpf1.vmx.lck encrypted_files 16ntexpf1.nvram.args 16ntexpf1.vmxf vmx-16ntexpf1-782615571-1.vswp.args 16ntexpf1.vmdk 16ntexpf1.vmxf.args vmx-16ntexpf1-782615571-2.vswp 16ntexpf1.vmdk.args 16ntexpf1.vmx~ vmx-16ntexpf1-782615571-2.vswp.args 16ntexpf1.vmsd 16ntexpf1_1-flat.vmdk

i have tried your tool, have reregister the mashines, but they wont start, because no bootable filesystem is found. any hint for me ?

Saddamus commented 1 year ago

Same problem as above. It takes seconds to restore VM with the script, but after re-registration, system will not boot. If i check VM with testdisk, there are no partitions to restore.

RelayForge commented 1 year ago

Exactly same situation - script is working successfully, but after re-registration the system will not boot. Its observed, that there is some issue with disk sizes: if we are trying to edit the vm setting, its mentioning with red that the disk size is incorrect (and adding a huge number of digits after the dot, for example - 200.00000095367432). Did someone have managed this?

aleksarkun commented 1 year ago

Hello. Got an infection of a server with several virtual machines. Using your script managed to restore 3 out of 4 machines, windows server 2016 booted without any problems. There was a problem with the machine on which was snapshot. This is the list of files of the virtual machine: Screenshot_38

When specify AstRun_srv.vmx script, it passes successfully however after registering the machine does not start: Failed to power on virtual machine AstRun_srv. File AstRun_srv_1-000001.vmdk was not found

Here is the content of the .vmx file:

.encoding = "UTF-8"
config.version = "8"
virtualHW.version = "14"
vmci0.present = "TRUE"
floppy0.present = "FALSE"
numvcpus = "12"
memSize = "32768"
bios.bootRetry.delay = "10"
firmware = "efi"
powerType.suspend = "soft"
tools.upgrade.policy = "manual"
sched.cpu.units = "mhz"
sched.cpu.affinity = "all"
vm.createDate = "1593152686125618"
scsi0.virtualDev = "lsisas1068"
scsi0.present = "TRUE"
sata0.present = "TRUE"
usb_xhci.present = "TRUE"
scsi0:0.deviceType = "scsi-hardDisk"
scsi0:0.fileName = "AstRun_srv-000001.vmdk"
sched.scsi0:0.shares = "normal"
sched.scsi0:0.throughputCap = "off"
scsi0:0.present = "TRUE"
scsi0:1.deviceType = "scsi-hardDisk"
scsi0:1.fileName = "AstRun_srv_1-000001.vmdk"
sched.scsi0:1.shares = "normal"
sched.scsi0:1.throughputCap = "off"
scsi0:1.present = "TRUE"
ethernet0.virtualDev = "e1000e"
ethernet0.networkName = "LAN3_Grp_AstRus"
ethernet0.addressType = "generated"
ethernet0.present = "TRUE"
displayName = "AstRun_srv"
guestOS = "windows9-64"
uefi.secureBoot.enabled = "TRUE"
toolScripts.afterPowerOn = "TRUE"
toolScripts.afterResume = "TRUE"
toolScripts.beforeSuspend = "TRUE"
toolScripts.beforePowerOff = "TRUE"
tools.syncTime = "FALSE"
uuid.bios = "56 4d 8e 2c 82 0a 68 1c-da 6d ba d0 fb 06 c3 97"
uuid.location = "56 4d 8e 2c 82 0a 68 1c-da 6d ba d0 fb 06 c3 97"
vc.uuid = "52 b7 2d 1d 64 9b 2d 63-2d 89 6a a7 58 68 b5 ca"
sched.cpu.min = "0"
sched.cpu.shares = "normal"
sched.mem.min = "0"
sched.mem.minSize = "0"
sched.mem.shares = "normal"
ethernet0.generatedAddress = "00:0c:29:06:c3:97"
vmci0.id = "-83442793"
cleanShutdown = "FALSE"
extendedConfigFile = "AstRun_srv.vmxf"
mks.enable3d = "TRUE"
tools.guest.desktop.autolock = "FALSE"
nvram = "AstRun_srv.nvram"
pciBridge0.present = "TRUE"
svga.present = "TRUE"
pciBridge4.present = "TRUE"
pciBridge4.virtualDev = "pcieRootPort"
pciBridge4.functions = "8"
pciBridge5.present = "TRUE"
pciBridge5.virtualDev = "pcieRootPort"
pciBridge5.functions = "8"
pciBridge6.present = "TRUE"
pciBridge6.virtualDev = "pcieRootPort"
pciBridge6.functions = "8"
pciBridge7.present = "TRUE"
pciBridge7.virtualDev = "pcieRootPort"
pciBridge7.functions = "8"
hpet0.present = "TRUE"
RemoteDisplay.maxConnections = "-1"
sched.cpu.latencySensitivity = "normal"
svga.autodetect = "FALSE"
disk.EnableUUID = "TRUE"
numa.autosize.cookie = "120001"
numa.autosize.vcpu.maxPerVirtualNode = "12"
sched.swap.derivedName = "/vmfs/volumes/5ebac8f4-ba43223e-d524-309c239ced34/AstRun_srv/AstRun_srv-625b630d.vswp"
pciBridge0.pciSlotNumber = "17"
pciBridge4.pciSlotNumber = "21"
pciBridge5.pciSlotNumber = "22"
pciBridge6.pciSlotNumber = "23"
pciBridge7.pciSlotNumber = "24"
scsi0.pciSlotNumber = "160"
ethernet0.pciSlotNumber = "192"
usb_xhci.pciSlotNumber = "224"
vmci0.pciSlotNumber = "32"
sata0.pciSlotNumber = "33"
scsi0.sasWWID = "50 05 05 6c 82 0a 68 10"
ethernet0.generatedAddressOffset = "0"
vm.genid = "1025359355794109798"
vm.genidX = "5812918138057738809"
monitor.phys_bits_used = "43"
vmotion.checkpointFBSize = "4194304"
vmotion.checkpointSVGAPrimarySize = "67108864"
softPowerOff = "FALSE"
toolsInstallManager.lastInstallError = "0"
svga.guestBackedPrimaryAware = "TRUE"
tools.remindInstall = "FALSE"
toolsInstallManager.updateCounter = "2"
migrate.hostLog = "./AstRun_srv-625b630d.hlog"
svga.vramSize = "67108864"
sata0:0.startConnected = "FALSE"
scsi0:0.redo = ""
scsi0:1.redo = ""

When I try to feed the AstRun_srv-000001 file to the script, it gives the following errors:

[root@static:/vmfs/volumes/5ebac8f4-ba43223e-d524-309c239ced34/AstRun_srv] /tmp/recover.sh AstRun_srv-000001
mkdir: can't create directory 'encrypted_files': File exists
Moving encrypted AstRun_srv-000001.vmdk to encrypted_files
mv: can't rename 'AstRun_srv-000001.vmdk': No such file or directory
ls: AstRun_srv-000001-flat.vmdk: No such file or directory

Creating copy of AstRun_srv-000001-flat.vmdk
Invalid file length specifier: -d
rm: can't remove 'temp-flat.vmdk': No such file or directory

Adding AstRun_srv-000001.vmdk
sed: temp.vmdk: No such file or directory
sed: temp.vmdk: No such file or directory
mv: can't rename 'temp.vmdk': No such file or directory

Copying AstRun_srv-000001.vmx
mv: can't rename 'AstRun_srv-000001.vmx': No such file or directory
cp: can't stat 'AstRun_srv-000001.vmx~': No such file or directory
Error: unable to find vmx backup. You may be unable to re-register the virtual machine.

Moving encrypted AstRun_srv-000001.vmsd to encrypted_files
mv: can't rename 'AstRun_srv-000001.vmsd': No such file or directory

Moving encrypted AstRun_srv-000001.nvram to encrypted_files
mv: can't rename 'AstRun_srv-000001.nvram': No such file or directory

Validating...
Failed to open disk link /vmfs/volumes/5ebac8f4-ba43223e-d524-309c239ced34/AstRun_srv/AstRun_srv-000001.vmdk :The system cannot find the file specified (25)Disk chain is not consistent : The system cannot find the file specified (25)

Error. Trying to update the file size.
sed: AstRun_srv-000001.vmdk: No such file or directory
Failed to open disk link /vmfs/volumes/5ebac8f4-ba43223e-d524-309c239ced34/AstRun_srv/AstRun_srv-000001.vmdk :The system cannot find the file specified (25)Disk chain is not consistent : The system cannot find the file specified (25)

Error. Could not recover. Please consult CISA's guidance for further information: https://www.cisa.gov/uscert/ncas/alerts/aa23-039a

I guess because of the snapshot I have a disk divided into files AstRun_srv_1-000001-sesparse.vmdk and the script is looking for *flat.vmdk. Maybe someone has a solution how to run a virtual machine in this case? Maybe just pull the files from the drive? Maybe you could redo the script so that it would work on the files AstRun_srv_1-000001-sesparse.vmdk; AstRun_srv-000001-sesparse.vmdk; AstRun_srv-Snapshot1.vmem ?

If you connect the disks AstRun_srv.vmdk and AstRun_srv_1.vmdk in the new created virtual machine - it starts and works correctly but there is old data.

aleksarkun commented 1 year ago

mr @cablej can you see my files pls =)

PatrikO23 commented 1 year ago

Hi, can someone please help my with this syntax error? /tmp/recover.sh: line 9: syntax error: unexpected newline Thank you.

Saddamus commented 1 year ago

BE AWARE WITH YOUR STEPS ! This is update to my previous report. As first, I took a VM that didnt contain any important data. After restoration, VM didn't start and attached disk had additional zeros in size :) I have corrected the size on VM settings and this destroyed the data. For other VMs i started restored disk with Gparted live CD, without correcting its size. Testdisk app could restore messed partitions and data. Probably fixing decriptor will solve the problem, but NEVER RUN RESTORED DISK IMAGE WITHOUT HAVING COPY !!! As I'm migrating to Proxmox, not going to dig in deeper what is the problem with restored image size, but this restoration script for sure needs some corrections.

cablej commented 1 year ago

Hi all, I suspect some of this may be due to a later version of the ransomware which also encrypts the virtual disk (I will update the guidance shortly around this). Unfortunately, if the virtual disk is also encrypted then it is not possible to recover using these methods.

You can use the following command to check if the virtual disk was encrypted:

vmkfstools -x check my_disk.vmdk

If the disk was not encrypted, let me know and the errors you are experiencing and I can try to help. If the disk was encrypted, unfortunately unless you have backups it is likely not possible to recover from the encrypted disk.

cablej commented 1 year ago

Hi @wxthss82, can you try running the check command on a flat vmdk file?

cablej commented 1 year ago

Hi @wxthss82, it appears that the flat file may unfortunately be encrypted. Can you check the ransom note via “ cat /usr/lib/vmware/hostd/docroot/index.html”?

wxthss82 commented 1 year ago

Exactly same situation - script is working successfully, but after re-registration the system will not boot. Its observed, that there is some issue with disk sizes: if we are trying to edit the vm setting, its mentioning with red that the disk size is incorrect (and adding a huge number of digits after the dot, for example - 200.00000095367432). Did someone have managed this?

Also facing the same issue.

mariosomma commented 1 year ago

Even tough the /recover.sh returned success I'm unable to register the VM : No VMs were found in [datastore1] RH_79_template"

This is what I have in my VM directory:

[esx_server:/vmfs/volumes/5ceff160-7005ea2a-a3f2-ac1f6b0b18f2/RH_79_template] ls -altr total 262146240 -rw-r--r-- 1 root root 446 Jun 22 2021 RH_79_template-774f34e8.hlog -rwxr-xr-x 1 root root 2836 Jul 15 2021 RH_79_template.vmtx drwxr-xr-t 1 root root 81920 Nov 29 08:47 .. drwxr-xr-x 1 root root 73728 Feb 3 10:44 . -rw-r--r-- 1 root root 9 Feb 8 17:05 RH_79_template.vmdk.args -rw-r--r-- 1 root root 17 Feb 8 17:05 RH_79_template-flat.vmdk.args -rw-r--r-- 1 root root 9 Feb 8 17:05 RH_79_template.vmsd.args -rw-r--r-- 1 root root 1024 Feb 8 17:05 RH_79_template.vmsd -rw------- 1 root root 1557 Feb 8 17:05 RH_79_template.vmdk -rw-r--r-- 1 root root 10 Feb 8 17:05 RH_79_template.nvram.args -rw------- 1 root root 9708 Feb 8 17:05 RH_79_template.nvram -rw------- 1 root root 268435457024 Feb 8 18:20 RH_79_template-flat.vmdk

This is the result of /recover.sh RH_79_template ==> "" Copying RH_79_template.vmx mv: can't rename 'RH_79_template.vmx': No such file or directory cp: can't stat 'RH_79_template.vmx~': No such file or directory Error: unable to find vmx backup. You may be unable to re-register the virtual machine. .. .. Validating... Disk chain is consistent.

Success! Unregister the virtual machine and re-register it and you should be good to go. ""\

Nisham95 commented 1 year ago

hi @cablej , i was wondering, if the flat file was encrypted, is it still possible to recover?