Instructions to build on ESXi 6.5/6.7 with 2 NICs only

[scaery]

Here is my story about setting up Detection Lab up on ESXi 6.5. and my ramblings.

Building the Detection Lab on ESXi currently uses 3 network names. I did not used all 3 and removed the NAT network. Therefore I needed to change the IF naming conventions as well. Since I am working on VMware Workstation deployed ESXi, i just setup the additional "HostOnly Network" interface.

Changes to the code:

ESXi/main.tf (Line 52-56, 87-91, 122-126, 157-161)

<..>
    provisioner "remote-exec" {
    inline = [
      "sudo ifconfig eth0 up || echo 'eth0 up'",
      "sudo ifconfig eth1 up || echo 'eth1 up'",
      "sudo route add default gw 192.168.1.1 || echo 'route exists'"
    ]
<..>

# OPTIONAL: You can comment out this interface stanza ...
#network_interfaces {
#    virtual_network = var.nat_network
#    mac_address     = "00:50:56:xx:xx:xx"
#    nic_type        = "e1000"
#  }

ESXi/variables.tf (removed NAT)

#variable "nat_network" {
#  default = "NAT Network"
#}

ESXi/resources/01-netcfg.yaml (replaced eth2 with eth1)

network:
  version: 2
  ethernets:
    eth0:
      dhcp4: true
      gateway4: 192.168.1.1
      nameservers:
        addresses: [8.8.8.8,8.8.4.4]
    eth1:
      dhcp4: false
      addresses: [192.168.38.105/24]

For the inventory I had to lookup the IPs via ipconfig on the hosts manually :(

ESXi/ansible/inventory.yml

---

logger:
  hosts:
    192.168.1.140:
      ansible_user: vagrant
      ansible_password: vagrant
      ansible_port: 22
      ansible_connection: ssh
      ansible_ssh_common_args: '-o StrictHostKeyChecking=no'

dc:
  hosts:
    192.168.1.141:

wef:
  hosts:
    192.168.1.142:

win10:
  hosts:
    192.168.1.143:

Then I did the appropriate changes for the network interfaces on the Logger, DC, WEF and Win10:

ESXi/ansible/roles/logger/tasks/main.yml (Line 121)

      if [ "$(cat /sys/class/net/eth1/address)" == "00:50:56:a3:b1:c4" ]; then

ESXi/ansible/roles/dc/tasks/main.yml (Line 14)

win_shell: "New-NetIPAddress –InterfaceAlias Ethernet1 –AddressFamily IPv4 -IPAddress 192.168.38.102 –PrefixLength 24 -DefaultGateway 192.168.38.1"

ESXi/ansible/roles/wef/tasks/main.yml (Line 14)

win_shell: "New-NetIPAddress –InterfaceAlias Ethernet1 –AddressFamily IPv4 -IPAddress 192.168.38.103 –PrefixLength 24 -DefaultGateway 192.168.38.1"

ESXi/ansible/roles/win10/tasks/main.yml (Line 13)

win_shell: "New-NetIPAddress –InterfaceAlias Ethernet1 –AddressFamily IPv4 -IPAddress 192.168.38.104 –PrefixLength 24 -DefaultGateway 192.168.38.1"

But in my case the naming on the Windows boxes is not Ethernet2, it is Ethernet1 for the HostOnly Network. The internet accessible VM Network interface is named Ethernet0 2.

Maybe someone could enlight me to setup the network interface names in ESXi or terraform to rename them. Also, what is your setup so I can retry with the additional vm.nat interface? Then the datastore1 and datastore2 I had to replace manually in the main.tf, because they are hardcoded and won't fit my storage on the ESXi ;( By default or if using ESXi on USB the datastore1 nor 2 will exist, only if you setup ESXi having some additional space left.

After these changes it all should go well. Currently only the Splunk Logger did not seem to get any events from the WEF ( even after fixing the network names and the boxes can reach each other on the 'HostOnly Network'. More digging time needed.

I was hit with unexpected bugs and various configuration advices that were missing from the documentation (e.g. sshpass needs to installed, ulimit needs to be increased, and extra option -f1 will help ansible to proceed, you should add env no_proxy='*' in front of ansible for deployment on OSX). Just weird stuff I had to fix. Maybe good if you can add these to the documentation.

Could you please provide some more information on your ESXi network setup? Also I might think you have a mixed IP setup, very different the ones published in the repo. How can this be simplified or even automated?

TL;DR

Could you please provide a better documentation about setting this up with just 2 NICs or providing the right Networking Setup in your 3 NIC version setup? Still, I might think there is an issue with the interface naming.

[scaery]

Went through a second time with the instructions above and all got well ;)

PLAY RECAP ************************************************************************************************************************************************************
10.101.101.12              : ok=37   changed=23   unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
10.101.101.14              : ok=29   changed=21   unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
10.101.101.18              : ok=25   changed=23   unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
10.101.101.20              : ok=40   changed=25   unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Pre-instructions and documentation improvements for building Detection Lab on OSX:

brew install sshpass

vagrant plugin install vagrant-reload

sudo launchctl limit maxfiles 9655360 999900000

env no_proxy='*' ansible-playbook -vvv detectionlab.yml -f1

Post-Instructions for Copy&Paste via Remote Console (advanced parameters in VMX template):

isolation.tools.copy.disable          FALSE
isolation.tools.paste.disable         FALSE
isolation.tools.setGUIOptions.enable  TRUE

Maybe now its time for more improvements, e.g. a build script to decide whether 2 or 3 NICs for building.

[C0axx]

Curious how you got the Packer portion of the Lab built since VNC is no longer an option in 6.7/7.0.

[scaery]

If you want, you can either build on Vagrant, convert to ovf 1.0 and run on 7.0. But the setup above is vanilla ESXi 6.5u3 ;)

[scaery]

Tested quickly on 6.7 and packer deployment works too!

Just follow https://nickcharlton.net/posts/using-packer-esxi-6.html

PORT     STATE SERVICE
5900/tcp open  vnc

Maybe it wont survive a reboot but 6.7 is an option as well. Only 7.0 has dropped vnc complete. I did not managed to find a workaround through custom vib or using the SDK. Feel free to add your additions https://github.com/clong/DetectionLab/issues/499

[clong]

Hey @scaery - thanks for the write up and for the feedback! I'll definitely add some of the missing parts (e.g. sshpass, etc) and look into ways to standardize the network stuff. I think it probably makes sense to keep the network interfaces at 2 rather than 3 for simplicity, but the OS naming convention is where things get a little tricky. I'll look into this over the next few days and see what I can come up with.

[clong]

@scaery can you expand upon why you needed to bump the max file limit?

[scaery]

@clong Yes, this was related playing with ansible forks setting https://github.com/ansible/ansible/issues/12259

Unexpected Exception: [Errno 24] Too many open files

In fact at least -f1 solved that issue while building on OSX, but playing around with like 100 forks just broke the ansible execution. On Mac the maxfilesize limit is by default rather low, so I increased the limit.

You maybe don't need this setting if you just use the -f1 switch I guess.

[scaery]

Keeping this as closed reference. All steps from above. Will try to dockerize it for myself ;)

sed -i -e "25,28d" ESXi/variables.tf
sed -i "s/eth1/eth0/g" ESXi/main.tf
sed -i "s/eth2/eth1/g" ESXi/main.tf
sed -i -e "90,94d;125,129d;160,164d" ESXi/main.tf
sed -i "s/eth2/eth1/g" ESXi/ansible/roles/logger/tasks/main.yml 
sed -i "s/Ethernet2/Ethernet1/g" ESXi/ansible/roles/dc/tasks/main.yml
sed -i "s/Ethernet2/Ethernet1/g" ESXi/ansible/roles/wef/tasks/main.yml
sed -i "s/Ethernet2/Ethernet1/g" ESXi/ansible/roles/win10/tasks/main.yml

Just in case anyone needs a 2 NIC setup only without the extra NAT.