Closed cavepopo closed 5 years ago
Any hints about this ?
Look again carefully at your map file. you typed adaptator_type: instead of adapter_type. Another hint is to remove the network device from your template. Salt will instruct vmware to create it at build time. If you leave it, you will will run into mac address collisions when running jobs in parallel (salt-cloud -P). edited to add: I also assume that you have the following in your associated cloud profile configuration
customization: True power_on: True
I do not use connected: True in my configs. I haven't had a problem with interfaces not coming up. I also don't specify the adapter type because the default is the vmxnet3.
A quick outline of what is required to create vms by cloning in a VMWare environment:
/etc/salt/cloud.providers.d/ your provider information in a file / you obviously have this correct/ /etc/salt/cldoud.profiles.d/ you need to provide common parameters to build a vm in a file: mine is vmware-sizes.conf ---------------snip-----------------------------------------
atx-centos-small:
provider: Austin
clonefrom: centos7-salt-atx
deploy: True
power_on: True
password: sdb://mysqlite/root_pass
customization: True
num_cpus: 2
memory: 1GB
cluster: ATXSRVCluster1
datastore: "ATX SC2 All Tiers Linux"
folder: "Linux server tests"
grains:
join_domain: True
spacewalk: True
minion:
master:
- salt.<redacted>
saltenv: dev
dns_servers:
- 10.212.16.3
- 10.208.16.3
domain: redacted
atx-centos-standard:
provider: Austin
clonefrom: centos7-salt-atx
deploy: True
power_on: True
password: sdb://mysqlite/root_pass
customization: True
num_cpus: 2
memory: 4GB
cluster: ATXSRVCluster1
datastore: "ATX SC2 All Tiers Linux"
folder: "Linux server tests"
grains:
join_domain: True
spacewalk: True
minion:
master:
- salt.<redacted>
saltenv: dev
dns_servers:
- 10.212.16.3
- 10.208.16.3
domain: us.<redacted>
----------------snip-------------------------------------------------
lastly in your map files you need items specific to your machine and any overrides to previous options being inherited from above (like changing memory and cpu configs) example here ----------------------------------------snip-----------------------------------------------
atx-centos-standard:
- ussigatxmon004p:
cluster: ATXSRVCluster1
datastore: ATXPSEEVM16
folder: "Solarwinds"
power_on: True
customization: True
devices:
network:
Network adapter 1:
name: ATXPROD
switch_type: standard
ip: 10.212.17.186
gateway: [10.212.16.1]
subnet_mask: 255.255.254.0
disk:
Hard Disk 1:
thin_provision: True
size: 20
minion:
saltenv: prod
grains:
product: Monitoring
target_load: grafana
--------------------snip------------------------------------------------------------ notice I call out the profile in the map (very first line) which provides where to clone from.
I have crafted a template with my basic minimum config. I adapted a script off the internet to "sysprep" my linux machine by wiping all the configs that happen at first boot. It is the last thing I do before powering down the golden image to convert it to a template. I then remove the NIC info from the template. It will be recreated when I spin up a vm from a map file.
The machine name is specified by the second line " ussigatxmon004p" in the example.
I hope this helps. I spent quite a bit of time working out the details when I first started with salt.
Thanks for your comment, although I am new with the vmware driver (well kind of new) I am quite experienced with salt globally.
I understand well the profile, provider and map file, along with the inheritance logic, still I am trying now to copy your profile / map info, as well as removing the network card (which i tried already but hey, this won't kill me).
The interesting part is that you confirm this is a standard process with the vmware driver, so I should be able to get this work.
I still have a question : How is the salt bootstrap process able to connect to the vm as we remove the nic from the template ? Is the salt-cloud handling this ? I thought it was mandatory to have an ip address in order to bootstrap the salt minion... What kind a cleaning do you apply to your vm before moving them as a template ? Is there any magic like removing the nic from the template but keeping the nic config inside the template (I saw that here) Looking forward to more of your advices
[EDIT]: I tried with a template with a nic and "connected" + "connected at startup" checked as well as without any nic (removed from vcenter webui), bith ending with the nic created on the clone but "not connected" .... Any solution to this is welcome, I am lost here...
Here is my config files, please tell me if you find anything that may be the root cause for my issue: vmware provider:
l-vcsa:
protocol: https
url: redacted
flush_mine_on_destroy: True
driver: vmware
delete_sshkeys: True
verify_ssl: False
user: redacted
script_args: stable 2019.2.0
password: redacted
port: 443
vmware profile:
clonefrom: template_centos7
provider: l-vcsa
deploy: True
power_on: True
customization: True
password: redacted
ssh_username: redacted
annotation: Created by Salt-Cloud
cluster: redacted
dns_servers:
- redacted
- redacted
and the map file used for my tests:
bare-centos7_vmware:
- test-centos-vmware.redacted:
num_cpus: 4
memory: 8GB
power_on: True
customization: True
devices:
disk:
Hard disk 1:
size: 12
Hard disk 2:
size: 5
network:
Network adapter 1:
name: Lab
adaptater_type: vmxnet3
switch_type: standard
ip: redacted
gateway: [redacted]
subnet_mask: 255.255.255.0
minion:
master: redacted
I suspect the problem you are facing is what motivated me to "sysprep" the image before templating. By default centos configures the network script for the interface to include the mac address of the nic. If it doesn't match, it doesn't configure the interface. I will post the details of the script I adapted here shortly but you should be able to prove the situation by editing the image to not include the mac address before you shut it down for templating.
My adapted file. I don't remember where I collected the original from
EDIT: I found the original https://github.com/ifireball/linux-sysprep/blob/master/sysprep.sh
#!/usr/bin/env sh
# sysprep.sh - Prepare machine for use as template
#
# This script was designed to be compatible with all Linux systems with a
# GNU-based userspace, but was only tested on RHEL7 and CentOS7
#
usage() {
cat 1>&2 <<EOF
Usage $0 [OPTIONS]
Prepare system for use as template.
-f Actually do something, don't just say it
-h Print this help message
EOF
}
verbose() {
echo "$@"
}
do_cmd() {
verbose " [ $@ (noop) ]"
}
really_do_cmd() {
verbose " [ $@ ]"
cmd="$1"
shift
$cmd "$@"
}
main() {
parse_args "$@"
remove_rhn_id
remove_ssh_keys
remove_net_scripts
remove_net_persistent
remove_hostname
remove_machine_id
build_generic_initrd
clean_logs
}
parse_args() {
while getopts 'fh' opt; do
case "$opt" in
f)
do_cmd() {
really_do_cmd "$@"
}
;;
h)
usage
exit 0
;;
*)
usage
exit 1
;;
esac
done
}
remove_rhn_id() {
local rhn_id='/etc/sysconfig/rhn/systemid'
[[ -x "$rhn_id" ]] || return
verbose 'Removing RHN system ID'
do_cmd rm -f "$rhn_id"
}
remove_ssh_keys() {
verbose 'Removing ssh keys'
for key in /etc/ssh/ssh_host_*; do
[ -f "$key" ] || continue
verbose "- $key"
do_cmd rm -f "$key"
done
}
remove_net_scripts() {
verbose 'Removing network scripts'
for scr in /etc/sysconfig/network-scripts/ifcfg-*; do
[[ -f "$scr" ]] || continue
[[ "$scr" == */ifcfg-lo ]] && continue
verbose "- $scr"
do_cmd rm -f "$scr"
done
verbose "Creating generic network settings"
do_cmd write_file '/etc/sysconfig/network' < /dev/null
for device in /sys/class/net/*; do
[[ -d "$device" ]] || continue
[[ "$device" == /sys/class/net/lo ]] && continue
devicename=`basename $device`
echo "- $devicename "
do_cmd write_file /etc/sysconfig/network-scripts/ifcfg-$devicename <<EOF
DEVICE=$devicename
TYPE=Ethernet
ONBOOT=yes
BOOTPROTO=none
IPADDR=
PREFIX=32
GATEWAY=
DNS1=10.212.16.3
DNS2=10.208.16.3
DOMAIN=<redacted>
IPV4_FAILURE_FATAL=no
IPV6INIT=no
NAME=$devicename
DEFROUTE=yes
EOF
done
}
remove_net_persistent() {
rules='/etc/udev/rules.d/70-persistent-net.rules'
[ -f "$rules" ] || return
verbose 'Removing persistent net UDEV rules'
do_cmd rm -f "$rules"
}
remove_hostname() {
verbose 'Removing fixed hostname'
do_cmd rm -f '/etc/hostname'
}
remove_machine_id() {
local machine_id='/etc/machine-id'
[[ -r "$machine_id" ]] || return
# If the system is setup woth a machine-id bind-mounted from a tempfs, we
# can't and don't need to empty it
grep -qF "$machine_id" /proc/mounts && return
verbose 'Removing machine-id'
do_cmd write_file "$machine_id" < /dev/null
}
build_generic_initrd() {
[[ -x '/sbin/dracut' ]] || return
verbose 'Building a generic initrd image'
verbose '- This may take a while...'
do_cmd dracut --no-hostonly --force
verbose '- done!'
}
clean_logs() {
verbose 'Cleaning up logfiles'
find /var/log -type f | while read log; do
[ -f "$log" ] || continue
verbose "- $log"
do_cmd rm -f "$log"
done
}
write_file() {
cat > "$1"
}
main "$@"
exit 0
There is a section in the middle where I write some boilerplate into the network-script. Change to suit your needs.
Sorry, I didn't answer one of your main questions. If you remove the nic from the template, it will be created at creation time as long as you have customization: True declared in your map file. Salt will instruct the VMWare API to create one to match the specifications in the map. It will then start the VM. If the VM has a static IP or is set to DHCP, it will attain an IP address and this will be returned to salt via the API. Salt will keep trying to connect to ssh on this IP until it connects, or the max timeout occurs in 20 minutes (I think) as long as deploy: True is declared.
One more tidbit, I could not get the customization to work with open vmware tools. I had to install the vmware provided one instead. If you haven't installed vmware tools, it won't configure a network device as there is no mechanism for vmware to customize the OS at boot.
arf .... I am depressed ... I can't get it to work, I tried all the possible combination, using your script, the original, nothing make it works. I have a 6.7 vcenter with 6.0 esxi nodes, might be a problem. But when I manually clone the very same template the nic is brought up, I am still lost.
When I use the script, the vm can't be reached, I hit a timeout, well almost, I am not waiting as long as needed... (3 or 4 minutes still)
I am using official vmware tools.
Any advices, ideas, pray are welcome :(
Here is the vm specs at salt-cloud invocation:
[DEBUG ] clone_spec set to:
(vim.vm.CloneSpec) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
location = (vim.vm.RelocateSpec) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
service = <unset>,
folder = <unset>,
datastore = <unset>,
diskMoveType = <unset>,
pool = 'vim.ResourcePool:resgroup-10',
host = <unset>,
disk = (vim.vm.RelocateSpec.DiskLocator) [],
transform = <unset>,
deviceChange = (vim.vm.device.VirtualDeviceSpec) [],
profile = (vim.vm.ProfileSpec) []
},
template = false,
config = (vim.vm.ConfigSpec) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
changeVersion = <unset>,
name = <unset>,
version = <unset>,
createDate = <unset>,
uuid = <unset>,
instanceUuid = <unset>,
npivNodeWorldWideName = (long) [],
npivPortWorldWideName = (long) [],
npivWorldWideNameType = <unset>,
npivDesiredNodeWwns = <unset>,
npivDesiredPortWwns = <unset>,
npivTemporaryDisabled = <unset>,
npivOnNonRdmDisks = <unset>,
npivWorldWideNameOp = <unset>,
locationId = <unset>,
guestId = <unset>,
alternateGuestName = <unset>,
annotation = u'Created by Salt-Cloud',
files = <unset>,
tools = <unset>,
flags = <unset>,
consolePreferences = <unset>,
powerOpInfo = <unset>,
numCPUs = 4,
numCoresPerSocket = <unset>,
memoryMB = 8192,
memoryHotAddEnabled = <unset>,
cpuHotAddEnabled = <unset>,
cpuHotRemoveEnabled = <unset>,
virtualICH7MPresent = <unset>,
virtualSMCPresent = <unset>,
deviceChange = (vim.vm.device.VirtualDeviceSpec) [
(vim.vm.device.VirtualDeviceSpec) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
operation = 'edit',
fileOperation = <unset>,
device = (vim.vm.device.VirtualDisk) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
key = 2000,
deviceInfo = (vim.Description) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
label = 'Hard disk 1',
summary = '10,485,760 KB'
},
backing = (vim.vm.device.VirtualDisk.FlatVer2BackingInfo) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
fileName = '[datastore1] template_centos7/template_centos7.vmdk',
datastore = 'vim.Datastore:datastore-18',
backingObjectId = '',
diskMode = 'persistent',
split = false,
writeThrough = false,
thinProvisioned = false,
eagerlyScrub = <unset>,
uuid = '6000C299-40f5-fc76-e86a-cafb0b84f7c8',
contentId = 'b20919ca087efdb23a805b3fa0077e1f',
changeId = <unset>,
parent = <unset>,
deltaDiskFormat = <unset>,
digestEnabled = false,
deltaGrainSize = <unset>,
deltaDiskFormatVariant = <unset>,
sharing = 'sharingNone',
keyId = <unset>
},
connectable = <unset>,
slotInfo = <unset>,
controllerKey = 1000,
unitNumber = 0,
capacityInKB = 12582912,
capacityInBytes = 10737418240L,
shares = (vim.SharesInfo) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
shares = 1000,
level = 'normal'
},
storageIOAllocation = (vim.StorageResourceManager.IOAllocationInfo) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
limit = -1L,
shares = (vim.SharesInfo) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
shares = 1000,
level = 'normal'
},
reservation = 0
},
diskObjectId = '1-2000',
vFlashCacheConfigInfo = <unset>,
iofilter = (str) [],
vDiskId = <unset>,
nativeUnmanagedLinkedClone = <unset>
},
profile = (vim.vm.ProfileSpec) [],
backing = <unset>
},
(vim.vm.device.VirtualDeviceSpec) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
operation = 'edit',
fileOperation = <unset>,
device = (vim.vm.device.VirtualVmxnet3) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
key = 4000,
deviceInfo = (vim.Description) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
label = 'Network adapter 1',
summary = u'Lab'
},
backing = (vim.vm.device.VirtualEthernetCard.NetworkBackingInfo) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
deviceName = u'Lab',
useAutoDetect = <unset>,
network = 'vim.Network:network-20',
inPassthroughMode = <unset>
},
connectable = (vim.vm.device.VirtualDevice.ConnectInfo) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
migrateConnect = <unset>,
startConnected = true,
allowGuestControl = true,
connected = false,
status = 'untried'
},
slotInfo = (vim.vm.device.VirtualDevice.PciBusSlotInfo) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
pciSlotNumber = 160
},
controllerKey = 100,
unitNumber = 7,
addressType = 'assigned',
macAddress = '00:50:56:90:2a:f8',
wakeOnLanEnabled = true,
resourceAllocation = (vim.vm.device.VirtualEthernetCard.ResourceAllocation) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
reservation = 0L,
share = (vim.SharesInfo) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
shares = 50,
level = 'normal'
},
limit = -1L
},
externalId = <unset>,
uptCompatibilityEnabled = true
},
profile = (vim.vm.ProfileSpec) [],
backing = <unset>
},
(vim.vm.device.VirtualDeviceSpec) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
operation = 'add',
fileOperation = u'create',
device = (vim.vm.device.VirtualDisk) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
key = -2087,
deviceInfo = (vim.Description) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
label = u'Hard disk 2',
summary = u'5.0 GB'
},
backing = (vim.vm.device.VirtualDisk.FlatVer2BackingInfo) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
fileName = '',
datastore = <unset>,
backingObjectId = <unset>,
diskMode = u'persistent',
split = <unset>,
writeThrough = <unset>,
thinProvisioned = false,
eagerlyScrub = false,
uuid = <unset>,
contentId = <unset>,
changeId = <unset>,
parent = <unset>,
deltaDiskFormat = <unset>,
digestEnabled = <unset>,
deltaGrainSize = <unset>,
deltaDiskFormatVariant = <unset>,
sharing = <unset>,
keyId = <unset>
},
connectable = <unset>,
slotInfo = <unset>,
controllerKey = 1000,
unitNumber = 1,
capacityInKB = 5242880,
capacityInBytes = <unset>,
shares = <unset>,
storageIOAllocation = <unset>,
diskObjectId = <unset>,
vFlashCacheConfigInfo = <unset>,
iofilter = (str) [],
vDiskId = <unset>,
nativeUnmanagedLinkedClone = <unset>
},
profile = (vim.vm.ProfileSpec) [],
backing = <unset>
}
],
cpuAllocation = <unset>,
memoryAllocation = <unset>,
latencySensitivity = <unset>,
cpuAffinity = <unset>,
memoryAffinity = <unset>,
networkShaper = <unset>,
cpuFeatureMask = (vim.vm.ConfigSpec.CpuIdInfoSpec) [],
extraConfig = (vim.option.OptionValue) [
(vim.option.OptionValue) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
key = u'ethernet0.startConnected',
value = u'TRUE'
}
],
swapPlacement = <unset>,
bootOptions = <unset>,
vAppConfig = <unset>,
ftInfo = <unset>,
repConfig = <unset>,
vAppConfigRemoved = <unset>,
vAssertsEnabled = <unset>,
changeTrackingEnabled = <unset>,
firmware = <unset>,
maxMksConnections = <unset>,
guestAutoLockEnabled = <unset>,
managedBy = <unset>,
memoryReservationLockedToMax = <unset>,
nestedHVEnabled = <unset>,
vPMCEnabled = <unset>,
scheduledHardwareUpgradeInfo = <unset>,
vmProfile = (vim.vm.ProfileSpec) [],
messageBusTunnelEnabled = <unset>,
crypto = <unset>,
migrateEncryption = <unset>
},
customization = (vim.vm.customization.Specification) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
options = <unset>,
identity = (vim.vm.customization.LinuxPrep) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
hostName = (vim.vm.customization.FixedName) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
name = u'test-centos-vmware'
},
domain = u'lab.rms.loc',
timeZone = <unset>,
hwClockUTC = <unset>
},
globalIPSettings = (vim.vm.customization.GlobalIPSettings) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
dnsSuffixList = (str) [],
dnsServerList = (str) [
u'172.27.0.8',
u'172.27.0.22'
]
},
nicSettingMap = (vim.vm.customization.AdapterMapping) [
(vim.vm.customization.AdapterMapping) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
macAddress = <unset>,
adapter = (vim.vm.customization.IPSettings) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
ip = (vim.vm.customization.FixedIp) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
ipAddress = u'172.27.200.241'
},
subnetMask = u'255.255.255.0',
gateway = (str) [
u'172.27.200.1'
],
ipV6Spec = <unset>,
dnsServerList = (str) [],
dnsDomain = u'lab.rms.loc',
primaryWINS = <unset>,
secondaryWINS = <unset>,
netBIOS = <unset>
}
}
],
encryptionKey = (byte) []
},
powerOn = true,
snapshot = <unset>,
memory = <unset>
}
Have you inspected the appropriate network-script file for information? I know this is an obvious question but have you disabled the firewall in the template?
Thanks for the follow-up @denz7201 !
I did NOT disabled the firewalld (too used to debian containers .... my bad), regarding the network-script file i think i just tried every possible solution...
BUT it is working now (even with firewalld activated), i added the following extra option to my profile:
extra_config:
ethernet0.startConnected: "true"
So to summarize my working setup:
I am going to try more template combination just to know what is really required, i'll come back and add information here if anything useful comes up. It does the trick ! So I guess i am going to close this ticket now, my day will be a good one then :)
thanks again for following up with me.
Hit this but cannot fix with the extraconfig workaround, any other recommendation to try out?
The salt documentation is missing the OS requirements for VMware customization to work. I had this same issue and fixed it by installing perl on the os template. You also have to run a command on the template to allow the vmware tools to run scripts. These steps are in the VMware and Ansible documentation.
When you run salt cloud with the OS configured properly, the nic will come up disconnected but it will connect after vm tools runs the customizations.
Description of Issue
I am new to the vmware driver so this issue may be a result of my misunderstanding. When creating a vm from a template (centos 7) the nic is not being brought up (means there is no connection and the master wait forever to get teh required ssh connection). The template has a nic, with connected ticked + connected at power on ticked too.
the thing that makes a difference is if I try to define a new IP from the profile or map file: This does prevent the nic on the new clone to be brought up. But if i just remove the devices:networks:etc from the map (or profile) file the new clone get its nic brought up correctly.
Setup
map file (the commented lines makes it kind of works) :
Steps to Reproduce Issue
Try using the above map file.
Versions Report
(Provided by running
salt --versions-report
. Please also mention any differences in master/minion versions.)vcenter version: