Deploy Yugabyte to cluster that has domain name

[hongbo-miao]

Issue

Currently if the cluster has domain name, Yugabyte will failed to install / deploy.

How to Reproduce

Step 1 - create a cluster with a domain name

Assume cloned the project at /Users/xxx/Downloads/yugabyte-operator, create an empty data folder at /Users/xxx/Downloads/yugabyte-operator/data.

Create a /Users/xxx/Downloads/yugabyte-operator/dev-cluster-config.yaml file:

apiVersion: k3d.io/v1alpha2
kind: Simple
kubeAPI:
  host: "k8s.my-domain.com"
  hostIP: "127.0.0.1"
  hostPort: "6442"
network: hm-network
volumes:
  - volume: "/Users/xxx/Downloads/yugabyte-operator/data:/data"
ports:
  - port: 40002:80
    nodeFilters:
      - loadbalancer
options:
  k3s:
    extraServerArgs:
      - --no-deploy=traefik
      # without below line or change to "--cluster-domain=cluster.local", Yugabyte can be installed successfully
      - --cluster-domain=dev.k8s.my-domain.com

Then create a dev cluster by k3d if you are using macOS by

k3d cluster create dev --config=dev-cluster-config.yaml

Step 2 - deploy Yugabyte

Create a /Users/xxx/Downloads/yugabyte-operator/yugabyte-pv.yaml file with content

Click to see yugabyte-pv.yaml content!

```yml apiVersion: v1 kind: PersistentVolume metadata: name: yugabyte-master-0-pv spec: capacity: storage: 1Gi volumeMode: Filesystem storageClassName: standard accessModes: - ReadWriteOnce hostPath: path: /data/master-0 type: Directory --- apiVersion: v1 kind: PersistentVolume metadata: name: yugabyte-master-1-pv spec: capacity: storage: 1Gi volumeMode: Filesystem storageClassName: standard accessModes: - ReadWriteOnce hostPath: path: /data/master-1 type: Directory --- apiVersion: v1 kind: PersistentVolume metadata: name: yugabyte-master-2-pv spec: capacity: storage: 1Gi volumeMode: Filesystem storageClassName: standard accessModes: - ReadWriteOnce hostPath: path: /data/master-2 type: Directory --- apiVersion: v1 kind: PersistentVolume metadata: name: yugabyte-tserver-0-pv spec: capacity: storage: 1Gi volumeMode: Filesystem storageClassName: standard accessModes: - ReadWriteOnce hostPath: path: /data/tserver-0 type: Directory --- apiVersion: v1 kind: PersistentVolume metadata: name: yugabyte-tserver-1-pv spec: capacity: storage: 1Gi volumeMode: Filesystem storageClassName: standard accessModes: - ReadWriteOnce hostPath: path: /data/tserver-1 type: Directory --- apiVersion: v1 kind: PersistentVolume metadata: name: yugabyte-tserver-2-pv spec: capacity: storage: 1Gi volumeMode: Filesystem storageClassName: standard accessModes: - ReadWriteOnce hostPath: path: /data/tserver-2 type: Directory ```

Then run

rm -rf data/master-0 data/master-1 data/master-2
rm -rf data/tserver-0 data/tserver-1 data/tserver-2
mkdir data/master-0 data/master-1 data/master-2
mkdir data/tserver-0 data/tserver-1 data/tserver-2

kubectl create -f deploy/crds/yugabyte.com_ybclusters_crd.yaml
kubectl create -f deploy/operator.yaml
kubectl create -f yugabyte-pv.yaml # the file is at above 
kubectl create -f deploy/crds/yugabyte.com_v1alpha1_ybcluster_cr.yaml

All Yugabyte pods failed with similar error. Here is the yb-master-0 log:

    @     0x7fc192d59825  __libc_start_main
    @           0x408829  _start
    @              (nil)  (unknown)

*** Check failure stack trace: ***
    @     0x7fc197f215e1  yb::(anonymous namespace)::DumpStackTraceAndExit()
    @     0x7fc197100c8d  google::LogMessage::Fail()
    @     0x7fc197102dfd  google::LogMessage::SendToLog()
    @     0x7fc1971007da  google::LogMessage::Flush()
    @     0x7fc1971038a9  google::LogMessageFatal::~LogMessageFatal()
    @           0x409476  yb::master::MasterMain()
    @     0x7fc192d59825  __libc_start_main
    @           0x408829  _start
    @              (nil)  (unknown)
*** Aborted at 1631289281 (unix time) try "date -d @1631289281" if you are using GNU date ***
PC: @     0x7fc192d6d5a6 __GI_abort
*** SIGSEGV (@0x0) received by PID 1 (TID 0x7fc1a2fa4b80) from PID 0; stack trace: ***
    @     0x7fc1936eaba0 (unknown)
    @     0x7fc192d6d5a6 __GI_abort
    @     0x7fc197f21634  yb::(anonymous namespace)::DumpStackTraceAndExit()
    @     0x7fc197100c8c  google::LogMessage::Fail()
    @     0x7fc197102dfc  google::LogMessage::SendToLog()
    @     0x7fc1971007d9  google::LogMessage::Flush()
    @     0x7fc1971038a8  google::LogMessageFatal::~LogMessageFatal()
    @           0x409475  yb::master::MasterMain()
    @     0x7fc192d59825 __libc_start_main
    @           0x408829 _start
    @                0x0 (unknown)

If I remove --cluster-domain=dev.k8s.my-domain.com or change it to --cluster-domain=cluster.local in the dev-cluster-config.yaml file when I create the cluster by k3d, Yugabyte can be installed without any issue.

Potential Solution

One potential solution is similar to a Dgraph issue I met before https://github.com/dgraph-io/dgraph/pull/7976 by removing hardset cluster.local or allow us pass own domain name. (Dgraph's issue is easy to fix as they are using yaml files directly, for Yugabyte, as this is using operator, I am not confident to update it. Plus there are many other ways to install, worth updating too in future)

It would be great to support the cluster with domain name. Thanks! 😃

[ajcaldera1]

Closed by PR #42

[hongbo-miao]

Thanks @ajcaldera1 !

Now it works by adding

apiVersion: yugabyte.com/v1alpha1
kind: YBCluster
spec:
  domain: k8s.my-domain.com # <- add this line

😃