Open nwesoccer opened 8 years ago
Hard to say from this cursory glimpse but can you try to edit your cluster to one windows node to see if that replicates the problem?
Yeah I'll give that a try. In the meantime, I did find this line of code. Is it possible that each machine resource is using the same instance of the driver and therefore the @config instance variable just keeps merging and merging and merging every new machine option? I've also found that it only happens when the server in question doesn't already exist, and since all the linux ones do exists, it being a windows machine might not have anything to do with it. I will test that as well and see if it just grows as I stand up more and more during an initial run.
you are correct that the configs are repeatedly merged into the same MergedConfig
however that should not be inflating the MergeConfi. Rather, new configs are overlayed on top of the older ones. I could see where that could cause issues with configs of different shapes but it looks like all of your machines share the same property keys.
Ah ok. I dug into MergedConfig code a little. Seems they actually maintain a list of all the configs and a calculated merge of them all. So I think just size and number of configs I have merged...the worst the calculations are. Also if I understand the code correctly, it only does the calculation and 'cache' the calculation per key when the key is first requested. In the driver 'full_description' method, there is a call to to_hash, which would cause the calculation to happen for every key....my guess...causing a ton calculations as it loops through every config for every key.
At this point, I think my only solution would be to call with_vsphere_driver inside my loop, so that it just replaces the driver each time, or instanciate my own ChefProvisioningVsphere::VsphereDriver.canonicalize_url
within the loop to pass to machine resource.
That sounds reasonable. Ideally the merging here should work differently to avoid this altogether.
Seems my solution didn't work either. The with_vsphere_driver seems to happen at compile time and therefore they end up all having the last one at runtime and I have the same issue. I believe my issue is at
def merge_options!(machine_options)
@config = Cheffish::MergedConfig.new(
{ machine_options: machine_options },
@config
)
end
Is anyone else having this issue? Seems unlikely that we are the only ones trying to stand up 10-15 servers in the same provisioning script and changing machine_options for each.
We replaced the above with
def merge_options!(machine_options)
@config = { machine_options: machine_options }
)
end
The below output (from converge of the 7th server) shows that somewhere else the machine_options are getting Merged. Change the above code, slowed the madness down to 19 merges on 7 server from the previous hundreds of merges that freezes up chef-client.
It seems not only the merge upon the merge per machine, but multiple times within a machine is causing exponential growth in merge count. Do we really need these merged options here since they are already being merged elsewhere? For now we will use our modified gem, if we find that the above merge is not needed, can we do a pull request to get that merged in?
customization_spec: #<Cheffish::MergedConfig:0x00000009f0b2f8 @configs=[{:org_name=>"???", :time_zone=>"America/New_York", :win_time_zone=>35, :product_id=>"???", :ipsettings=>{:ip=>"192.168.10.21", :subnetMask=>"255.255.255.224", :gateway=>["192.168.10.1"], :dnsServerList=>["192.168.1.2", "192.168.1.3"]}, :domain=>"???", :domainAdmin=>"???", :domainAdminPassword=>"???", :hostname=>"wnapi1"}, #<Cheffish::MergedConfig:0x00000009f0b370 @configs=[{:org_name=>"???", :time_zone=>"America/New_York", :win_time_zone=>35, :product_id=>"???", :ipsettings=>{:ip=>"192.168.10.55", :subnetMask=>"255.255.255.224", :gateway=>["192.168.10.33"], :dnsServerList=>["192.168.1.2", "192.168.1.3"]}, :domain=>"???", :domainAdmin=>"???", :domainAdminPassword=>"???", :hostname=>"wntasks1"}, #<Cheffish::MergedConfig:0x00000009f0b3e8 @configs=[{:org_name=>"???", :time_zone=>"America/New_York", :win_time_zone=>35, :product_id=>"???", :ipsettings=>{:ip=>"192.168.10.55", :subnetMask=>"255.255.255.224", :gateway=>["192.168.10.33"], :dnsServerList=>["192.168.1.2", "192.168.1.3"]}, :domain=>"???", :domainAdmin=>"???", :domainAdminPassword=>"???", :hostname=>"wntasks1"}, #<Cheffish::MergedConfig:0x00000009f0b460 @configs=[{:org_name=>"???", :time_zone=>"America/New_York", :win_time_zone=>35, :product_id=>"???", :ipsettings=>{:ip=>"192.168.10.42", :subnetMask=>"255.255.255.224", :gateway=>["192.168.10.33"], :dnsServerList=>["192.168.1.2", "192.168.1.3"]}, :domain=>"???", :domainAdmin=>"???", :domainAdminPassword=>"???", :hostname=>"wnhirus1"}, #<Cheffish::MergedConfig:0x00000009f0b4d8 @configs=[{:org_name=>"???", :time_zone=>"America/New_York", :win_time_zone=>35, :product_id=>"???", :ipsettings=>{:ip=>"192.168.10.42", :subnetMask=>"255.255.255.224", :gateway=>["192.168.10.33"], :dnsServerList=>["192.168.1.2", "192.168.1.3"]}, :domain=>"???", :domainAdmin=>"???", :domainAdminPassword=>"???", :hostname=>"wnhirus1"}, #<Cheffish::MergedConfig:0x00000009f0b550 @configs=[{:org_name=>"???", :time_zone=>"America/New_York", :win_time_zone=>35, :product_id=>"???", :ipsettings=>{:ip=>"192.168.10.74", :subnetMask=>"255.255.255.224", :gateway=>["192.168.10.65"], :dnsServerList=>["192.168.1.2", "192.168.1.3"]}, :domain=>"???", :domainAdmin=>"???", :domainAdminPassword=>"???", :hostname=>"wnsql1"}, #<Cheffish::MergedConfig:0x00000009f0b5c8 @configs=[{:org_name=>"???", :time_zone=>"America/New_York", :win_time_zone=>35, :product_id=>"???", :ipsettings=>{:ip=>"192.168.10.74", :subnetMask=>"255.255.255.224", :gateway=>["192.168.10.65"], :dnsServerList=>["192.168.1.2", "192.168.1.3"]}, :domain=>"???", :domainAdmin=>"???", :domainAdminPassword=>"???", :hostname=>"wnsql1"}, #<Cheffish::MergedConfig:0x00000009f0b640 @configs=[{:org_name=>"???", :time_zone=>"America/New_York", :win_time_zone=>35, :product_id=>"???", :ipsettings=>{:ip=>"192.168.10.49", :subnetMask=>"255.255.255.224", :gateway=>["192.168.10.33"], :dnsServerList=>["192.168.1.2", "192.168.1.3"]}, :domain=>"???", :domainAdmin=>"???", :domainAdminPassword=>"???", :hostname=>"lnes3"}, #<Cheffish::MergedConfig:0x00000009f0b6b8 @configs=[{:org_name=>"???", :time_zone=>"America/New_York", :win_time_zone=>35, :product_id=>"???", :ipsettings=>{:ip=>"192.168.10.49", :subnetMask=>"255.255.255.224", :gateway=>["192.168.10.33"], :dnsServerList=>["192.168.1.2", "192.168.1.3"]}, :domain=>"???", :domainAdmin=>"???", :domainAdminPassword=>"???", :hostname=>"lnes3"}, #<Cheffish::MergedConfig:0x00000009f0b730 @configs=[{:org_name=>"???", :time_zone=>"America/New_York", :win_time_zone=>35, :product_id=>"???", :ipsettings=>{:ip=>"192.168.10.48", :subnetMask=>"255.255.255.224", :gateway=>["192.168.10.33"], :dnsServerList=>["192.168.1.2", "192.168.1.3"]}, :domain=>"???", :domainAdmin=>"???", :domainAdminPassword=>"???", :hostname=>"lnes2"}, #<Cheffish::MergedConfig:0x00000009f0b7a8 @configs=[{:org_name=>"???", :time_zone=>"America/New_York", :win_time_zone=>35, :product_id=>"???", :ipsettings=>{:ip=>"192.168.10.48", :subnetMask=>"255.255.255.224", :gateway=>["192.168.10.33"], :dnsServerList=>["192.168.1.2", "192.168.1.3"]}, :domain=>"???", :domainAdmin=>"???", :domainAdminPassword=>"???", :hostname=>"lnes2"}, #<Cheffish::MergedConfig:0x00000009f0b820 @configs=[{:org_name=>"???", :time_zone=>"America/New_York", :win_time_zone=>35, :product_id=>"???", :ipsettings=>{:ip=>"192.168.10.47", :subnetMask=>"255.255.255.224", :gateway=>["192.168.10.33"], :dnsServerList=>["192.168.1.2", "192.168.1.3"]}, :domain=>"???", :domainAdmin=>"???", :domainAdminPassword=>"???", :hostname=>"lnes1"}, #<Cheffish::MergedConfig:0x00000009f0b898 @configs=[{:org_name=>"???", :time_zone=>"America/New_York", :win_time_zone=>35, :product_id=>"???", :ipsettings=>{:ip=>"192.168.10.47", :subnetMask=>"255.255.255.224", :gateway=>["192.168.10.33"], :dnsServerList=>["192.168.1.2", "192.168.1.3"]}, :domain=>"???", :domainAdmin=>"???", :domainAdminPassword=>"???", :hostname=>"lnes1"}, #<Cheffish::MergedConfig:0x00000009f0b910 @configs=[{:org_name=>"???", :time_zone=>"America/New_York", :win_time_zone=>35, :product_id=>"???", :ipsettings=>{:ip=>"192.168.10.27", :subnetMask=>"255.255.255.224", :gateway=>["192.168.10.1"], :dnsServerList=>["192.168.1.2", "192.168.1.3"]}, :domain=>"???", :domainAdmin=>"???", :domainAdminPassword=>"???", :hostname=>"lnredis2"}, #<Cheffish::MergedConfig:0x00000009f0b988 @configs=[{:org_name=>"???", :time_zone=>"America/New_York", :win_time_zone=>35, :product_id=>"???", :ipsettings=>{:ip=>"192.168.10.27", :subnetMask=>"255.255.255.224", :gateway=>["192.168.10.1"], :dnsServerList=>["192.168.1.2", "192.168.1.3"]}, :domain=>"???", :domainAdmin=>"???", :domainAdminPassword=>"???", :hostname=>"lnredis2"}, #<Cheffish::MergedConfig:0x00000009f0ba00 @configs=[{:org_name=>"???", :time_zone=>"America/New_York", :win_time_zone=>35, :product_id=>"???", :ipsettings=>{:ip=>"192.168.10.26", :subnetMask=>"255.255.255.224", :gateway=>["192.168.10.1"], :dnsServerList=>["192.168.1.2", "192.168.1.3"]}, :domain=>"???", :domainAdmin=>"???", :domainAdminPassword=>"???", :hostname=>"lnredis1"}, #<Cheffish::MergedConfig:0x00000009f0ba78 @configs=[{:org_name=>"???", :time_zone=>"America/New_York", :win_time_zone=>35, :product_id=>"???", :ipsettings=>{:ip=>"192.168.10.26", :subnetMask=>"255.255.255.224", :gateway=>["192.168.10.1"], :dnsServerList=>["192.168.1.2", "192.168.1.3"]}, :domain=>"???", :domainAdmin=>"???", :domainAdminPassword=>"???", :hostname=>"lnredis1"}, #<Cheffish::MergedConfig:0x00000009f0baf0 @configs=[{:org_name=>"???", :time_zone=>"America/New_York", :win_time_zone=>35, :product_id=>"???", :ipsettings=>{:ip=>"192.168.10.2", :subnetMask=>"255.255.255.224", :gateway=>["192.168.10.1"], :dnsServerList=>["192.168.1.2", "192.168.1.3"]}, :domain=>"???", :domainAdmin=>"???", :domainAdminPassword=>"???", :hostname=>"lnlb1"}, {:org_name=>"???", :time_zone=>"America/New_York", :win_time_zone=>35, :product_id=>"???", :ipsettings=>{:ip=>"192.168.10.2", :subnetMask=>"255.255.255.224", :gateway=>["192.168.10.1"], :dnsServerList=>["192.168.1.2", "192.168.1.3"]}, :domain=>"???", :domainAdmin=>"???", :domainAdminPassword=>"???", :hostname=>"lnlb1"}], @merge_arrays={}>], @merge_arrays={}>], @merge_arrays={}>], @merge_arrays={}>], @merge_arrays={}>], @merge_arrays={}>], @merge_arrays={}>], @merge_arrays={}>], @merge_arrays={}>], @merge_arrays={}>], @merge_arrays={}>], @merge_arrays={}>], @merge_arrays={}>], @merge_arrays={}>], @merge_arrays={}>], @merge_arrays={}>], @merge_arrays={}>], @merge_arrays={}>
I think this is a combination of the machine provider behavior with the driver behavior. The driver is recursively merging the configuration in multiple places, and each time the machine provider performs an action, it gets another recursive merge. Since it looks like the merging of configs, which is taking place in the driver, is already handled at the machine level. Perhaps this can be removed?
Situation: We have made our provisioning very dynamic in order to define our multiple environments in the json environment files. We have an array of servers that the loop in the code below goes through to call the machine resource. The code snippet below is just the pertinent part of our provisioning recipe, there is not much else other than the definitions of the variables and such. The issue we have is that it will machine :converge 6 linux boxes, then the first windows box the log spits out several thousand lines of:
As the code shows, we are just passing in a hash, where it is being converted into a Cheffish::MergedConfig I don't know, but it seems to be the merger of the same config hundreds of times. I'm not sure if it's just the fact that it's number 7, or if it's windows, or what....Are we just doing something wrong and not understanding ruby/chef-provisioning?
Provisioning Server Chef Versions
Provisioning Recipe