xCluster : The RPC server is unavailable

michaelsync commented 6 years ago

Details of the scenario you tried and the problem that is occurring

I have 3 services (SQL1, SQL2, SQL3). I want to create a cluster on SQL1 and I want to join SQL 2 and SQL3 to the cluster that I created on SQL1. I have turned off the firewall of my VMs.

Please note that I am able to all those steps manually on those servers. So I think it shouldn't be firewall or config issue.

I am using this code.

xCluster FailOverCluster
        {
            Name                = $Node.ClusterName
              DomainAdministratorCredential = Get-AutomationPSCredential $Node.DSCCredentialName
            DependsOn = "[SqlServerEndpoint]SQLConfigureEndpoint-Instance"
        }

Verbose logs showing the problem

"Message": "The PowerShell DSC resource \u0027[xCluster]FOCluster\u0027 with SourceInfo \u0027::2150::9::xCluster\u0027 threw one or more non-terminating errors while running the Test-TargetResource functionality. These errors are logged to the ETW channel called Microsoft-Windows-DSC/Operational. Refer to this channel for more details.",

"Data": { }, "InnerException": { "ErrorCode": -2147023174, "Message": "Check the spelling of the cluster name. Otherwise, there might be a problem with your network. Make sure the cluster nodes are turned on and connected to the network or contact your network administrator.", "Data": "System.Collections.ListDictionaryInternal", "InnerException": "Microsoft.FailoverClusters.PowerShell.ClusterCmdletException: The RPC server is unavailable",

The DSC configuration that is used to reproduce the issue (as detailed as possible)

xCluster FailOverCluster
        {
            Name                = $Node.ClusterName
              DomainAdministratorCredential = Get-AutomationPSCredential $Node.DSCCredentialName
            DependsOn = "[SqlServerEndpoint]SQLConfigureEndpoint-Instance"
        }

The operating system the target node is running

OsName : Microsoft Windows Server 2016 Datacenter OsOperatingSystemSKU : DatacenterServerEdition OsArchitecture : 64-bit WindowsBuildLabEx : 14393.2248.amd64fre.rs1_release.180427-1804 OsLanguage : en-US OsMuiLanguages : {en-US}

Version and build of PowerShell the target node is running

Name Value

PSVersion 5.1.14393.2248 PSEdition Desktop PSCompatibleVersions {1.0, 2.0, 3.0, 4.0...} BuildVersion 10.0.14393.2248 CLRVersion 4.0.30319.42000 WSManStackVersion 3.0 PSRemotingProtocolVersion 2.3 SerializationVersion 1.1.0.1

Version of the DSC module that was used ('dev' if using current dev branch)

johlju commented 6 years ago

It looks like you doing this in Azure VM's? There are currently a problem using these resources in Azure. It would be awesome if we could get it solved. If you search for Azure among the issues you see others having similar problems.

Could you list the steps you do manually to get cluster working between the three nodes? Maybe that can help making the resource work.

michaelsync commented 6 years ago

Thanks.

Yes. I am on Azure VM.

Here is what I did manually to get the cluster working between the three notes. Please note that I have run the list of DSC to create the login, endpoints, give permission and etc. before I create the cluster.

RDP to SQL1 VM
Give the permission to this folder C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys (Because I was getting keyset-does-not-exist. (link) )
Create a new cluster using UI Wizard via RDP
Add three nodes (SQL1, SQL2, SQL3)
Give the cluster name
did some validation or testing from the UI (still following the wizard )
And then finished.

That's how I did manually. It was working fine. I can enable the 'AlwaysOn' after that.

If you want to see it via skype or Teams, I am happy to show you too. Let me know if you need more information. Thanks.

johlju commented 6 years ago

@michaelsync That is interesting, and you getting validation tests to pass too. How is the VM's configured with networking? There are issues that describes that they need to use IPv6 link-lock addresses to talk between nodes. How is your network (vnet) configured between the nodes? I would love to be able to set up a working three-way cluster too, so I can debug this. So curious how you have configured everything. How did you deploy the environment the nodes, vnet etc? Is it something you can share (with any sensitive information obfuscated)?

If you want to see it via skype or Teams, I am happy to show you too.

Let see if it where we get with sharing the information above first. Please also note that I don't have so much time to put into this, since I maintain all of DSC Resource Kit, but will do what I can! Really love to have this resolved (cluster in Azure VM's), so might be able to put in some off-work hours on this.

michaelsync commented 6 years ago

@johlju Thanks so much for this.

I don't have any special configuration for vnet. We have a vnet with three AD servers. I created three SQL VMs and joined the vnet so there is no public access to those VM. We have to be in our VPN or via our rdp gateway or in same subnet to connect to those VMs.

Honestly, I didn't create that vnet. If you want to know any specific setting, I can figure out and let you know.

IPv6 link-lock addresses

I am not sure but I can access those SQL servers each other by using either name or v4 IP address from SQL management studio.

johlju commented 6 years ago

Arm templates for setting up a similar environment would have been awesome. But won't expect you to be able to provide those (even with sensitive information obfuscated). 😄

Have the cluster any IPv6 network bound, or is it only IPv4 networks and addresses? Please obfuscate any sensitive information.

Get-ClusterNetwork | fl *
Get-ClusterNetworkInterface | fl *
Get-NetIPInterface | ? ConnectionState -eq 'Connected'
Get-NetIPConfiguration

Also, do you have configured a Azure Load Balancer for any of the IP's used by the cluster?

stale[bot] commented 6 years ago

This issue has been automatically marked as stale because it has not had activity from the community in the last 30 days. It will be closed if no further activity occurs within 10 days. If the issue is labelled with any of the work labels (e.g bug, enhancement, documentation, or tests) then the issue will not auto-close.

stale[bot] commented 6 years ago

This issue has been automatically closed because it is has not had activity from the community in the last 40 days.

dsccommunity / FailoverClusterDsc