search

clong / DetectionLab

Automate the creation of a lab environment complete with security tooling and logging best practices
MIT License
4.62k stars 985 forks source link

Bug: Win10 provisioning hangs on shared folders - VMWare Tools not installed #720

Open coffeegist opened 2 years ago

coffeegist commented 2 years ago

Please verify that you are building from an updated Master branch before filing an issue. √

Description of the issue:

While provisioning Win10, it always gets stuck on this stage:

==> win10: Fixed port collision for 5985 => 55985. Now on port 2200.
==> win10: Fixed port collision for 5986 => 55986. Now on port 2201.
==> win10: Fixed port collision for 22 => 2222. Now on port 2202.
==> win10: Starting the VMware VM...
==> win10: Waiting for the VM to receive an address...
==> win10: Forwarding ports...
    win10: -- 5985 => 2200
    win10: -- 5986 => 2201
    win10: -- 22 => 2202
==> win10: Waiting for machine to boot. This may take a few minutes...
    win10: WinRM address: 127.0.0.1:2200
    win10: WinRM username: vagrant
    win10: WinRM execution_time_limit: PT2H
    win10: WinRM transport: negotiate
==> win10: Machine booted and ready!
==> win10: Setting hostname...
==> win10: Configuring network adapters within the VM...
==> win10: Configuring secondary network adapters through VMware
==> win10: on Windows is not yet supported. You will need to manually
==> win10: configure the network adapter.
==> win10: Enabling and configuring shared folders...

Upon investigation, it seems the issue is VMWare Tools not being installed. I can log in as vagrant, load the tools installer iso, and start the installation as an administrator and it seems to work fine from there on out.

The DC does not seem to have this issue. Let me know if there's any more information I can provide.

Box versions: 

vagrant box update
==> logger: Box 'bento/ubuntu-18.04' not installed, can't check for updates.
==> wef: Checking for updates to 'detectionlab/win2016'
    wef: Latest installed version: 1.8
    wef: Version constraints:
    wef: Provider: vmware_desktop
==> wef: Updating 'detectionlab/win2016' with provider 'vmware_desktop' from version
==> wef: '1.8' to '1.9'...
==> wef: Loading metadata for box 'https://vagrantcloud.com/detectionlab/win2016'
==> wef: Adding box 'detectionlab/win2016' (v1.9) for provider: vmware_desktop
    wef: Downloading: https://vagrantcloud.com/detectionlab/boxes/win2016/versions/1.9/providers/vmware_desktop.box
    wef: Calculating and comparing box checksum...
==> wef: Successfully added box 'detectionlab/win2016' (v1.9) for 'vmware_desktop'!
==> win10: Checking for updates to 'detectionlab/win10'
    win10: Latest installed version: 1.8
    win10: Version constraints:
    win10: Provider: vmware_desktop
==> win10: Box 'detectionlab/win10' (v1.8) is running the latest version.
clong commented 2 years ago

Attempting to repro this. What version of VMware Fusion?

0xv1n commented 2 years ago

VMware 16.2 on Windows:

An error occurred while executing `vmrun`, a utility for controlling
VMware machines. The command and output are below:

Command: ["enableSharedFolders", "Z:\\DetectionLab\\Vagrant\\.vagrant\\machines\\dc\\vmware_desktop\\a48b65fc-a6f3-4d60-a6e4-771f1a85d6ce\\WindowsServer2016.vmx", {:notify=>[:stdout, :stderr]}]

Stdout: Error: The VMware Tools are not running in the virtual machine: Z:\DetectionLab\Vagrant\.vagrant\machines\dc\vmware_desktop\a48b65fc-a6f3-4d60-a6e4-771f1a85d6ce\WindowsServer2016.vmx
coffeegist commented 2 years ago

Hey @clong sorry for the delay, I’ve been away this weekend but will update as soon as I can. I know it was the most up to date version of VMWare Fusion (maybe 12.1.2 or something). Will confirm soon

clong commented 2 years ago

Hi folks, @n0leptr helped me get some insight into what's happening here in Slack. I'm not sure why it only seems to be happening in newer versions of Fusion/Workstation, but the workaround for now is to manually login to Windows so that VMware tools kicks off its configuration:

image

It appears VMware Tools doesn't fully install until that happens.

EDIT: This is only the case for Windows 10. It doesn't appear to be installed on 2016 at all.

clong commented 2 years ago

It's definitely intermittent on Win10, at least with VMware Workstation

==> win10: Machine booted and ready!
==> win10: Setting hostname...
==> win10: Waiting for machine to reboot...
==> win10: Configuring network adapters within the VM...
==> win10: Configuring secondary network adapters through VMware
==> win10: on Windows is not yet supported. You will need to manually
==> win10: configure the network adapter.
==> win10: Enabling and configuring shared folders...
    win10: -- C:/Users/build/DetectionLab/Vagrant: /vagrant
==> win10: Running provisioner: shell...
    win10: Running: scripts/fix-second-network.ps1 as C:\tmp\vagrant-shell.ps1
    win10: [14:55] Running fix-second-network.ps1...
    win10: [14:55] No VirtIO adapters, moving on...
    win10: [14:55] VMware Tools not found, no need to continue. Exiting.
clong commented 2 years ago

OK, I'll try to summarize where I think things are at:

  1. The Win2016 box does not appear to have VMware tools installed at all. However, it's clear the installer ran because I see the install logs in c:\Windows\Temp.
  2. The Win10 box has them installed, but due to the fact that the OS is booting post-sysprep, I think it has to modify/reinstall them. This introduces an unknown delay between the time the host is booted and when shared folders can be enabled.

What's odd to me is that I don't recall seeing this issue previously. Not sure if I somehow got lucky with my boxes or if the newest version of VMware tools is causing issues.

clong commented 2 years ago

OK, 1 isn't true either because I was just able to successfully provision DC without any issues and VMtools was present, which means this is problem is definitely intermittent (yay)

clong commented 2 years ago

Ah, new theory!

The machine boots up and does a login and reboot when it attempts to set the hostname. During that time, VMware tries to configure itself. If it's able to do it in time, it will work. If not, the host will reboot before VMware installs and vmware will fail to be present from that point forward.

clong commented 2 years ago

As an attempted workaround, I could try disabling synced folders, calling a powershell script inline to ensure vmware tools is installed correctly and then re-enabling it once that's complete. I'll give that a shot.

clong commented 2 years ago

Workaround doesn't work, pops up a dialog box about a previous failed installation of VMware Tools by SYSTEM. Only solution here is to somehow delay the reboot during the hostname change.

d0gg0d commented 2 years ago

Hey, I can confirm that I also have the same issue.

Command: ["enableSharedFolders", "A:\\DetectionLab\\Vagrant\\.vagrant\\machines\\dc\\vmware_desktop\\70ea408c-9530-414f-8b90-a0f6d43de460\\WindowsServer2016.vmx", {:notify=>[:stdout, :stderr]}]

Stdout: Error: The VMware Tools are not running in the virtual machine: A:\DetectionLab\Vagrant\.vagrant\machines\dc\vmware_desktop\70ea408c-9530-414f-8b90-a0f6d43de460\WindowsServer2016.vmx

EDIT: Was able to successfully move past this issue.

  1. Tried to install VMware Tools on the DC, but the VMware Tools Installation initially failed saying that there was a previous installation of it.
  2. Ran vagrant reload dc --provision and received the same issue that VMware Tools is not installed.
  3. Went into the system again and tried to install VMware Tools, but this time it was successful.
  4. Ran vagrant reload dc --provision and the provisioning seems to go smoothly at the current moment.
    • I'm not sure what exactly fixed it lol.
clong commented 2 years ago

@OlafHaalstra @coffeegist @tuedenn @n0leptr

Are any of you able to reproduce this issue reliably? I totally believe it exists, I just need someone to help me test the fix if anyone is willing to help

0xv1n commented 2 years ago

@OlafHaalstra @coffeegist @tuedenn @n0leptr

Are any of you able to reproduce this issue reliably? I totally believe it exists, I just need someone to help me test the fix if anyone is willing to help

Unfortunately, I think it's a coin flip where more often than not the provision routine is successful.

If the fix is pushed to the repo on some branch, I'll be happy to test tonight. I wonder if a recent VMWare or Vagrant update fixed the issue by happy accident. I have run the script a few times in the past month and never encountered this.

OlafHaalstra commented 2 years ago

@OlafHaalstra @coffeegist @tuedenn @n0leptr

Are any of you able to reproduce this issue reliably? I totally believe it exists, I just need someone to help me test the fix if anyone is willing to help

Unfortunately, I think it's a coin flip where more often than not the provision routine is successful.

If the fix is pushed to the repo on some branch, I'll be happy to test tonight. I wonder if a recent VMWare or Vagrant update fixed the issue by happy accident. I have run the script a few times in the past month and never encountered this.

I have the same experience. Let me reconnect as soon as it happens to me again. Is there any change you made already? Or are you never able to reproduce the issue locally?.

cfilz commented 2 years ago

Hi - I'm having the same issue when building the lab first time. On the 2016 boxen. hanging on enabling and configuring shared folders, then eventually fails with the 'vmware tools are not running' error. Will attempt to work around but wanted to share this continues to occur.

clong commented 2 years ago

I'm working on a fix for this at the moment

kiyori-lw commented 2 years ago

@OlafHaalstra @coffeegist @tuedenn @n0leptr

Are any of you able to reproduce this issue reliably? I totally believe it exists, I just need someone to help me test the fix if anyone is willing to help

this happened to me today when i was provisioning dc/2016. all i did was log in to the DC run vmware tools setup - it prompted me to clean up old processes and remnants from a failed install, which i did. then i ran the vagrant reload dc --provision and (minus velociraptor) it seemed to come up

bcheevers123 commented 2 years ago

Any update on a fix or workaround?

rufflabs commented 1 year ago

Just chiming in here, I am deploying the lab for the first time and experienced this with dc. Got stuck at enabling shared folders and eventually timed out with tools not running.

I logged in and saw that vmware tools was not installed, but vmware wouldn't let me install it as the option is grayed out.

Host is Win10 with VMware Workstation 17.0.0, vagrant 2.3.3.

Scr1ptK1dd1e commented 1 year ago

I am also having this issue on both the DC and Win10 machines. I did notice on login that I can't mount the vmware-shared-folders from DC or Win10 to install vmware tools for some reason. Didn't try on the other machine but isn't that how vmware tools is moved to and installed on the windows machines? Prior to this, on both machines I had the error below pertaining to "secondary network adapters". image just before the "Configuring share folders step" that it gets stuck on. I got this notification that vagrant will stop overwriting an ethernet setting in the VMX and it may need to be manually changed? See image below. It happened on every affected machine. Could a networking issue be occurring that's preventing the install of VMware tools? image image

EDIT: I located the vmware-tools msi installer file and attempted to run it on DC and this is what happened. I'm afraid that undoing the changes or simply installing it as this user won't fix the problem on it's own and that it needs to be executed as system for things to work properly. image image

EDIT AGAIN:

Domain Controller

I downloaded the sysinternals suite to get psexec.exe and gain a root shell to run as root. https://learn.microsoft.com/en-us/sysinternals/downloads/sysinternals-suite Add it to your path or add the directory it's in to path.. or just execute the psexec.exe psexec -i -s VMware tools64.msi to get a shell as system The file is located in C:\Windows\Temp{1FF5D624-5515-4343-837A-E54C101573E6}~setup\VMware Tools64.msi I would just CD there because the curly braces and everything tend to mess up inputting the path normally. This will finish the install as system as confirmed by my ability to copy & paste that path above from the VM. Rinse and repeat and then probably just run vagrant reload dc --provision like above to work around the issue?

I think the above comment referencing this command worked because, whatever the hold up is (like the installer needs) the reload re-ran the installer as NT-SYSTEM AFTER it occurred.

Seems Win10 sorted this out on it's own but I believe networking is presenting issues to connecting to domain controller now but I think that's another issue

NoraGithub commented 1 year ago

Any update on a fix or workaround?

0xv1n commented 1 year ago

Any update on a fix or workaround?

@NoraGithub this project is unfortunately no longer being maintained. I remember speaking about this issue in Slack, and from what I remember it's not easily reproducible. In the past, I would get around it by reprovisioning the offending box or destroying and starting over.

Scr1ptK1dd1e commented 1 year ago

I would really not recommend running it on VMWare. I was using the project as the starting point for a school project to get up a quick AD environment for pentesting demo's. I tried almost all of the install methods but on VMWare Workstation Pro with Vagrant had the most issues by far.I believe all the cloud installation methods with Terraform all worked perfectly fine on the first attempt. When I was doing it, it already wasn't being supported but looking back through the issues to troubleshoot each of the numerous issues, the VMWare version always had the most unresolved issues. If you can't afford or just don't like using Cloud, I'd recommend Hyper-V for your local hypervisor platform. Pretty much everyone can use it and IMHO it's better supported and more robust. I've never lost a VM to corruption or anything on Hyper-V but almost every other week, either me or someone I know will have a corrupted VMX or worse. Last week I inadvertently gave my computer a memory leak in trying to repair a corrupted .vmx file for one of my images. Also, the Vagrant/VMWare tool you need to get this project working right often didn't work as intended when installed latest version from Chocolate. It's just too many things to go wrong to use VMWare.

On Sat, Mar 11, 2023 at 3:20 AM Nora @.***> wrote:

Any update on a fix or workaround?

— Reply to this email directly, view it on GitHub https://github.com/clong/DetectionLab/issues/720#issuecomment-1464869060, or unsubscribe https://github.com/notifications/unsubscribe-auth/AP3TYOFP3U3QYBIBIM36IZTW3Q7VVANCNFSM5F3WNNJA . You are receiving this because you commented.Message ID: @.***>

lap1nou commented 1 year ago

Hey,

Personally I had this bug because I left a tab open on the VMWare window of an old machine that I had previously provisionned.

Regards.