dsccommunity / FailoverClusterDsc

This module contains DSC resources for deployment and configuration of Windows Server Failover Cluster.
MIT License
60 stars 54 forks source link

Cluster: Add node to primary cluster that is located in another subnet #154

Open mohamednazar opened 7 years ago

mohamednazar commented 7 years ago

How to add secondary node to a existing running cluster which is created using xCluster DSC Resource.. I am using xWaitForCluster resource to add secondary node, but its not adding. Please see the below DSC resource code that I tried.

dsc_resource 'test-cluster' do resource :xWaitForCluster module_name 'xFailOverCluster' module_version '1.1' property :name, 'TestCluster1' property :RetryIntervalSec, 30 property :RetryCount, 100 end

So I tried using following code as well but nothing worked.

dsc_resource 'test-cluster' do resource :xCluster module_name 'xFailOverCluster' module_version '1.1' property :name, 'TestCluster1' property :staticipaddress, '10.0.0.3' property :domainadministratorcredential, ps_credential('mydomainname\\mydomainusername', mydomainpassword') end

Your help is much appreciated.

xCluster and xWaitForCluster DSC resource.

Windows 2012 R2 and PowerShell 5.x is running on target machine.

xFailOverCluster version is 1.1

johlju commented 7 years ago

@mohamednazar You need to use xCluster resource on second node after the xWaitForCluster resource, Please see the example 2-JoinAdditionalNodeToFailoverCluster.ps1.

mohamednazar commented 7 years ago

@johlju as you suggested I have tried xWaitForCluster then xCluster on second node but getting same error.

Code that I tried: `dsc_resource 'wait-cluster' do resource :xWaitForCluster module_name 'xFailOverCluster' module_version '1.8.0.0' property :name, 'testcluster1' property :PsDscRunAsCredential, ps_credential("#{node['sqlserver']['Username']}", "#{node['sqlserver']['Password']}") property :RetryIntervalSec, 30 property :RetryCount, 100 end

dsc_resource 'join-cluster' do resource :xCluster module_name 'xFailOverCluster' module_version '1.8.0.0' property :name, "#{node['sqlserver']['cluster_name']}" property :staticipaddress, "#{node['sqlserver']['static_ip']}" property :domainadministratorcredential, ps_credential("#{node['sqlserver']['Username']}", "#{node['sqlserver']['Password']}") end`

Error I am getting:

10.241.23.117 Compiling Cookbooks...←[0m 10.241.23.117 Converging 2 resources←[0m 10.241.23.117 Recipe: SqlServer::WaitForCluster←[0m 10.241.23.117 ←[0m Chef::Exceptions::PowershellCmdletException←[0m 10.241.23.117 -------------------------------------------←[0m 10.241.23.117 Powershell Cmdlet failed: Check the spelling of the cluster name. Otherwise, there might be a problem 10.241.23.117 ←[0m with your network. Make sure the cluster nodes are turned on and connected to 10.241.23.117 ←[0m the network or contact your network administrator. 10.241.23.117 ←[0m + CategoryInfo : ConnectionError: (:) [], CimException 10.241.23.117 ←[0m + FullyQualifiedErrorId : ClusterRpcConnection,Microsoft.FailoverClusters. 10.241.23.117 ←[0m PowerShell.AddClusterNodeCommand 10.241.23.117 ←[0m + PSComputerName : localhost 10.241.23.117 ←[0m 10.241.23.117 ←[0m The PowerShell DSC resource '[xCluster]DirectResourceAccess' with SourceInfo 10.241.23.117 ←[0m '' threw one or more non-terminating errors while running the 10.241.23.117 ←[0m Set-TargetResource functionality. These errors are logged to the ETW channel 10.241.23.117 ←[0m called Microsoft-Windows-DSC/Operational. Refer to this channel for more 10.241.23.117 ←[0m details. 10.241.23.117 ←[0m + CategoryInfo : InvalidOperation: (root/Microsoft/...gurationMan 10.241.23.117 ←[0m ager:String) [], CimException 10.241.23.117 ←[0m + FullyQualifiedErrorId : NonTerminatingErrorFromProvider 10.241.23.117 ←[0m + PSComputerName : localhost 10.241.23.117 ←[0m

Please help to resolve it..

Regards, Mohamed Nazar

johlju commented 7 years ago

This cluster your trying to configure, is it running in an on-premises Active Directory domarn, or in a cloud Active Directory domain? There are others having similar problems with adding a second node, so trying to figure out the scope. I have not been able to reproduce the problem the others have reported, yet.

mohamednazar commented 7 years ago

@johlju The cluster is running in AWS EC2 and Active Directory is in On-Premises.

mohamednazar commented 7 years ago

@johlju I have tried with xCluster after xWaitForCluster new set of servers and found secondary node successfully added to the cluster. But Primary node IP address only available due to that cluster went offline after I did move Core Cluster Services to "Best Possible Node" or "Select Node". Please find my code below.

` dsc_resource 'wait-cluster' do resource :xWaitForCluster module_name 'xFailOverCluster' module_version '1.8.0.0' property :name, "#{node['sqlserver']['cluster_name']}" property :PsDscRunAsCredential, ps_credential("#{node['sqlserver']['Username']}", "#{node['sqlserver']['Password']}") property :RetryIntervalSec, 30 property :RetryCount, 100 end

dsc_resource 'join-cluster' do resource :xCluster module_name 'xFailOverCluster' module_version '1.8.0.0' property :name, "#{node['sqlserver']['cluster_name']}" property :staticipaddress, '10.0.0.2' property :domainadministratorcredential, ps_credential("#{node['sqlserver']['Username']}", "#{node['sqlserver']['Password']}") end `

Please find the screenshot below. image

How to get added secondary node IP address as well to cluster? so that I can move cluster resources to secondary node. When we do manually the IPs are adding fine. See the screenshot below (Yellow marked IP added) after that I added manually.

image

Please help! regards, Mohamed Nazar

mohamednazar commented 6 years ago

@johlju or @kwirkykat or @mbreakey3

Waiting for your assistance. Anyone please help how to get added secondary node IP address to the cluster.

Regads, Mohamed Nazar

johlju commented 6 years ago

@mohamednazar A cluster has only one or more IP address for the CNO. They are not different for each cluster node. Each node has at least one IP-address on the NIC, and the CNO has at least one IP-address on the same subnet as the IP-address on the NIC.

For example: Subnet: 192.168.10.0/24

Node 1: 192.168.10.1 Node 2: 192.168.10.2 CNO: 192.168.10.3

johlju commented 6 years ago

@mohamednazar If you have another NIC in you system, with a different subnet, then you can add an IP-address for that subnet to the CNO as well.

Please let me know if I misunderstood your question,

mohamednazar commented 6 years ago

@johlju after cluster created and added secondary node successfully, I can see only one IP address (Primary server IP) under Cluster Core Resources which is in bottom of "Fail Over Cluster Manager" see the image below. Secondary server IP address not adding.

image

But it should be like below.

image

for cluster and SQL always on as per AWS documentation , the both (Primary and Secondary) servers should have 2 additional IPs on default NIC. See the example below.

Primary Server: Subnet A Default IP: 10.0.0.1 Listener IP: 10.0.0.2 Cluster IP: 10.0.0.3

Secondary Server: Subnet B Default IP: 10.0.1.1 Listener IP: 10.0.1.2 Cluster IP: 10.0.1.3

The secondary server cluster IP not adding to the cluster. Hope you clear my issue now.

Regards, Mohamed Nazar

johlju commented 6 years ago

Okay, you are gonna stretch the cluster over two regions? I personally have not stretched a cluster over different subnets. :/ I have to read up on it to know if it is supported. I won’t have time for that in the near future. Maybe someone else in the community familiar with that setup can tell if the resources support it today, or if we in the community need to make changes to the resources.

Do you have a link to the AWS documentation?

mohamednazar commented 6 years ago

I am going to setup cluster with two availability zone in single region. Please view the AWS documentation link

http://docs.aws.amazon.com/quickstart/latest/sql/implementation.html#ip-addressing

Regards, Mohamed Nazar

dead8171 commented 6 years ago

I am not able to join the second node to the cluster, I have 3 nodes 2 SQL nodes and a DC both the nodes are domain joined, and are in an availability set, I was able to create a cluster on Node1 But was not successful in joining the node, I am using Automation Account and both the nodes and the DC are in Azure.

mohamednazar commented 6 years ago

@dead8171 on secondary nodes use xWaitForCluster resource then use xCluster.. Check whether all cluster related ports are not blocked..

mohamednazar commented 6 years ago

@johlju any update on adding secondary node IP address on cluster core services?

johlju commented 6 years ago

@mohamednazar I did a quick google search yesterday, but haven't been able to read any, nor the AWS link you provided. Long days at the normal day job. But as far as I gather, if it means that one must provide two IP addresses when configuring the cluster, then xCluster does not support that yet. It only takes a single IP address in the StaticIPAddress parameter.

The New-Cluster cmdlet is called when creating the cluster, it's StaticAddress takes a string array. See documentation here New-Cluster. So it might be as simple as changing the StaticIPAddress parameter from a string to a string array. This change should be done regardless since New-Cluster expects an array.

mohamednazar commented 6 years ago

@johlju I think I need to use xClusterNetwork resource to add secondary node IP & Netmask to the cluster.. Correct me if I am wrong. If its correct should I want to apply the script on PrimaryReplica or SecondaryReplica?

dsc_resource 'dsc_clus_net_secondary' do resource :xClusterNetwork property :Address, "#{node['sqlserver']['static_ip']}" property :AddressMask, "#{node['sqlserver']['address_mask']}" property :Name, 'Clustername' property :PsDscRunAsCredential, ps_credential("Username", "Password") end

Regards, Mohamed Nazar

johlju commented 6 years ago

xClusterNetwork handles the properties of the cluster networks (subnets), please see issue #92. I think issue #151 is on the right track? What do you think?

mohamednazar commented 6 years ago

Yes seems to be same issue. Let me try xClusterNetwork properties...

mohamednazar commented 6 years ago

@johlju

I have executed following xClusterNetwork script on secondaryreplica server and getting error.

Script:

dsc_resource 'dsc_clus_net_secondary' do resource :xClusterNetwork property :Address, "IP" property :AddressMask, "mask" property :Name, "name_of_the_cluster" property :Metric, '10' property :Role, '3' property :PsDscRunAsCredential, ps_credential("Username", "password") end

Error:

[2017-11-27T03:59:12+00:00] INFO: Processing dsc_resource[dsc_clus_net_secondary] action run (SqlServer::dsc_clus_net_secondary line 7) [2017-11-27T03:59:14+00:00] INFO: Running queued delayed notifications before re-raising exception [2017-11-27T03:59:15+00:00] ERROR: Running exception handlers [2017-11-27T03:59:15+00:00] ERROR: Exception handlers complete [2017-11-27T03:59:15+00:00] FATAL: Stacktrace dumped to C:/chef/cache/chef-stacktrace.out [2017-11-27T03:59:15+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report [2017-11-27T03:59:15+00:00] FATAL: Chef::Exceptions::PowershellCmdletException: dsc_resource[dsc_clus_net_secondary] (SqlServer::dsc_clus_net_secondary line 7) had an error: Chef::Exceptions::PowershellCmdletException: Powershell Cmdlet failed: The property 'Name' cannot be found on this object. Verify that the property

exists and can be set.

+ CategoryInfo          : InvalidOperation: (:) [], CimException

+ FullyQualifiedErrorId : PropertyNotFound

+ PSComputerName        : localhost

You cannot call a method on a null-valued expression.

+ CategoryInfo          : InvalidOperation: (:) [], CimException

+ FullyQualifiedErrorId : InvokeMethodOnNull

+ PSComputerName        : localhost

The property 'Role' cannot be found on this object. Verify that the property

exists and can be set.

+ CategoryInfo          : InvalidOperation: (:) [], CimException

+ FullyQualifiedErrorId : PropertyNotFound

+ PSComputerName        : localhost

You cannot call a method on a null-valued expression.

+ CategoryInfo          : InvalidOperation: (:) [], CimException

+ FullyQualifiedErrorId : InvokeMethodOnNull

+ PSComputerName        : localhost

The property 'Metric' cannot be found on this object. Verify that the property

exists and can be set.

+ CategoryInfo          : InvalidOperation: (:) [], CimException

+ FullyQualifiedErrorId : PropertyNotFound

+ PSComputerName        : localhost

You cannot call a method on a null-valued expression.

+ CategoryInfo          : InvalidOperation: (:) [], CimException

+ FullyQualifiedErrorId : InvokeMethodOnNull

+ PSComputerName        : localhost

The PowerShell DSC resource '[xClusterNetwork]DirectResourceAccess' with

SourceInfo '' threw one or more non-terminating errors while running the

Set-TargetResource functionality. These errors are logged to the ETW channel

called Microsoft-Windows-DSC/Operational. Refer to this channel for more

details.

+ CategoryInfo          : InvalidOperation: (root/Microsoft/...gurationMan 

ager:String) [], CimException

+ FullyQualifiedErrorId : NonTerminatingErrorFromProvider

+ PSComputerName        : localhost

Please let me know should I want to change any property value or should I want to execute the above script on primaryreplica server?

Regards, Mohamed Nazar

mohamednazar commented 6 years ago

@johlju or @kwirkykat or @mbreakey3

Waiting for your assistance.

Regards, Mohamed Nazar

johlju commented 6 years ago

@mohamednazar xClusterNetwork only changes properties on existing networks. It does not have logic to add networks, nor can it have that kind of logic (as mentioned in issue #92). The xClusterNetwork can only change the properties Name, Role and Metric.

smastrorocco commented 6 years ago

Any update on this, have same issue with cluster across AWS subnets.

codykonior commented 6 years ago

I've been working on setting up a multi-subnet cluster where each node only has one subnet so I have some information to add to the discussion.

  New-Cluster: Static address 'xxx/24' was not found on any cluster network.

The way I envision you'd get this going is adding logic to xCluster so that when it's doing an Add-ClusterNode it will do more with the StaticIPAddress.

It looks like this:

Get-Cluster | Add-ClusterResource -Name "IP Address $ip" -Group "Cluster Group" -ResourceType "IP Address"
$network = Get-Cluster | Get-ClusterNetwork | Where-Object { (([Net.IPAddress] $_.Address).Address -band ([Net.IPAddress] $_.AddressMask).Address) -eq (([Net.IPAddress] $ip).Address -band ([Net.IPAddress] $_.AddressMask).Address)}
Get-Cluster | Get-ClusterResource -Name "IP Address $ip" | Set-ClusterParameter -Multiple @{ Address = $ip; Network = $network.Name; SubnetMask = $network.AddressMask; }
$expression = (Get-Cluster | Get-ClusterResourceDependency -Resource "Cluster Name").DependencyExpression
if ($expression -match "^\((.*)\)$") {
    $expression = $Matches[1] + " or [IP Address $ip]"
} else {
    $expression = $expression + " or [IP Address $ip]"
}
Get-Cluster | Set-ClusterResourceDependency -Resource "Cluster Name" -Dependency $expression
# Without this, it won't start automatically on first try
(Get-Cluster | Get-ClusterResource -Name "IP Address $clusterIPAddress").PersistentState = 1

Maybe this could be controlled through an additional flag so that it doesn't break any existing code that might have dummy StaticIPAddress passed in for nodes.

mgates86 commented 4 years ago

Has anyone solved this? @mohamednazar @johlju

mohamednazar commented 4 years ago

@mgates86

I used xWaitForCluster resource on secondary node after that it resolved my problem.

On Primary and Secondary I assigned one additional IP (Secondary IP)

1st I created cluster on Primary, after that secondary was able to add to cluster using xWaitForCluster resource.

Thanks

mgates86 commented 4 years ago

@mohamednazar thank you. curious were you doing this for a sql always on cluster? im having a similar issue with multiple subnets on the slq ag listener

codykonior commented 4 years ago

Just get all the replicas up and added to the AG before you create the listener. See https://github.com/codykonior/OftenOn/blob/master/OftenOn/Configuration/OftenOn.ps1 for one way to do it.

mgates86 commented 1 year ago

@mgates86

I used xWaitForCluster resource on secondary node after that it resolved my problem.

On Primary and Secondary I assigned one additional IP (Secondary IP)

1st I created cluster on Primary, after that secondary was able to add to cluster using xWaitForCluster resource.

Thanks

@mohamednazar any chance you can post a sample. im still struggling with this.

segoja7 commented 9 months ago

I have the same problem, I am using 1-Cluster_CreateFirstNodeOfAFailoverClusterConfig.ps1 for creating my cluster in node1, and later I am using 2-Cluster_JoinAdditionalNodeToFailoverClusterConfig.ps1 for join in the cluster my second node. But this not work.

Reviewing the comments @mohamednazar I am trying again in my second node with: 1-WaitForCluster_WaitForFailoverClusterToBePresentConfig.ps1 but the error is the same.

Check the spelling of the cluster name. Otherwise, there might be a problem with your network. Make sure the cluster nodes are turned on and connected to the network or contact your network administrator.
    + CategoryInfo          : ConnectionError: (:) [], CimException
    + FullyQualifiedErrorId : ClusterRpcConnection,Microsoft.FailoverClusters.PowerShell.AddClusterNodeCommand
    + PSComputerName        : sqlfsx2

VERBOSE: [SQLFSX2]:                            [[Cluster]JoinSecondNodeToCluster] Added node SQLFSX2 to cluster cluster.
VERBOSE: [SQLFSX2]: LCM:  [ End    Set      ]  [[Cluster]JoinSecondNodeToCluster]  in 31.1600 seconds.
The PowerShell DSC resource '[Cluster]JoinSecondNodeToCluster' with SourceInfo 'C:\Users\Administrator\Documents\WaitForCluster_WaitForFailoverClusterToBePresentConfig.ps1::87::9::Cluster' threw one or more non-terminating errors while running the 
Set-TargetResource functionality. These errors are logged to the ETW channel called Microsoft-Windows-DSC/Operational. Refer to this channel for more details.
    + CategoryInfo          : InvalidOperation: (:) [], CimException
    + FullyQualifiedErrorId : NonTerminatingErrorFromProvider
    + PSComputerName        : sqlfsx2

VERBOSE: [SQLFSX2]: LCM:  [ End    Set      ]
The SendConfigurationApply function did not succeed.
    + CategoryInfo          : NotSpecified: (root/Microsoft/...gurationManager:String) [], CimException
    + FullyQualifiedErrorId : MI RESULT 1
    + PSComputerName        : sqlfsx2

My scenario is the following in AWS:

AD: corp.local Node1: in the domain corp.local with the following IPS: Primary IP x.x.40.104, secondary IPS. x.x.40.105, x.x.40.106 Node2: in the domain corp.local with the following IPS: Primary IP x.x.44.104, secondary IPS. x.x.44.105, x.x.44.106.

Any comment is good!