Multi-node cluster setup failing: wait for system database timeout

[jefflill]

Setup fails when building the neonCLOUD/Devops/jefflill/home/small-hyperv.cluster.yaml cluster:

This cluster consists of 1 master and three workers and does not deploy any cStor devices. I tried restarting setup and that failed again the same way. There's a decent chance that this will impact @granth7.

Here are the cluster setup logs:

logs.zip

from cluster.log:

STATUS: *** ERROR: [TimeoutException]: Timeout waiting for the predicate to return TRUE.

*** ERROR: [TimeoutException]: Timeout waiting for the predicate to return TRUE.
*** STACK:
   at Neon.Common.NeonHelper.WaitForAsync(Func`1 predicate, TimeSpan timeout, Nullable`1 pollInterval, String timeoutMessage) in C:\src\neonKUBE\Lib\Neon.Common\Helper\NeonHelper.Misc.cs:line 501
   at Neon.Kube.KubeSetup.WaitForStatefulSetAsync(ISetupController controller, String namespace, String name, String labelSelector, String fieldSelector) in C:\src\neonKUBE\Lib\Neon.Kube.Setup\KubeSetup.Operations.cs:line 3445
   at Neon.Common.NeonHelper.WaitAllAsync(IEnumerable`1 tasks) in C:\src\neonKUBE\Lib\Neon.Common\Helper\NeonHelper.Misc.cs:line 620
   at Neon.Kube.KubeSetup.<>c__DisplayClass99_0.<<InstallSystemDbAsync>b__3>d.MoveNext() in C:\src\neonKUBE\Lib\Neon.Kube.Setup\KubeSetup.Operations.cs:line 3345
--- End of stack trace from previous location ---
   at Neon.Kube.NodeSshProxy`1.InvokeIdempotentAsync(String actionId, Func`1 action) in C:\src\neonKUBE\Lib\Neon.Kube\Proxies\NodeSshProxy.cs:line 528
   at Neon.Kube.KubeSetup.InstallSystemDbAsync(ISetupController controller, NodeSshProxy`1 master) in C:\src\neonKUBE\Lib\Neon.Kube.Setup\KubeSetup.Operations.cs:line 3340
   at Neon.Kube.KubeSetup.SetupClusterAsync(ISetupController controller, Int32 maxParallel) in C:\src\neonKUBE\Lib\Neon.Kube.Setup\KubeSetup.Operations.cs:line 290
   at Neon.Kube.SetupController`1.<>c__DisplayClass30_0.<<ExecuteStep>b__6>d.MoveNext() in C:\src\neonKUBE\Lib\Neon.Kube\SetupController\SetupController.cs:line 888

• [ ] JEFF PS: This exception was actually thrown from the master node but didn't appear in the master.log file. I'm going to fix this so the exception will be included in both the individual node log as well as the cluster.log and I'll add information to the cluster log indicating which node the error originated from.

[jefflill]

Here are the deployments:

...and here are the pods:

• [ ] The interesting thing here is that it looks like we're deploying cStor even though it wasn't explicitly specified in the cluster definition:

  name: small - hyperv
  datacenter: $<<< profile:home.datacenter >>>
   environment: development
   timeSources:
  -pool.ntp.org
  kubernetes:
  allowPodsOnMasters: true
  hosting:
  environment: hyperv - local
    vm:
  cores: 4
      memory: 8 GiB
      diskLocation: $<<< profile:home.hyperv.diskfolder >>>
   network:
    premiseSubnet: $<<< profile:home.lan.subnet >>>
     gateway: $<<< profile:home.lan.gateway >>>
      nameservers:
      - $<<< profile:home.lan.dns0 >>>
       - $<<< profile:home.lan.dns1 >>>
    openEbs:
    engine: cstor
  nodes:
    master - 0:
      role: master
      address: $<<< profile:home.cluster.hyperv.node0.ip >>>
     worker - 0:
      role: worker
      address: $<<< profile:home.cluster.hyperv.node1.ip >>>
       openEbsStorage: true
    worker - 1:
      role: worker
      address: $<<< profile:home.cluster.hyperv.node2.ip >>>
       openEbsStorage: true
    worker - 2:
      role: worker
      address: $<<< profile:home.cluster.hyperv.node3.ip >>>
       openEbsStorage: true

• [ ] I checked and we currently default to the Jiva storage engine when nothing is specified in the definition and we haven't completed support for Jiva yet, right?

• [ ] It's interesting that single node clusters seem to deploy OK without setting this. You must still have code somewhere that's special-casing single node clusters. I'm going to modify the if statements that check the storage engine with switch statements and throw NotImplementedException for Jiva and Mayastor (which I'm going to add).

• [ ] I'm going to add a temporary HostPath engine type which we can use rather than checking for a single node cluster as well as a Default engine type and then modify OpenEbsOptions.Validate() to change Default to HostPath for single node clusters an cStor for multi-node clusters as well as throw NotImplementedException when Jiva or Mayastor are specified.

• [ ] Also, what's up with the ImagePullBackoff? Does this mean that the image required for the **cspc-stripe-*** pods hasn't been preloaded into CRI-O? I assume that this image can't be retrieved from origin because Harbor can't start without the systemdb which can't start without OpenEBS.

[jefflill]

I created the neonKUBE/storage branch with my changes. Here's the commit:

https://github.com/nforgeio/neonKUBE/commit/a15755225276cb4ce0c8853284eb2880685c6acd

I don't think we need to create a neonCLOUD branch for this. We can probably merge back to master and publish the nugets when this works.